[tor-bugs] #8662 [Torperf]: Make Torperf log circuit failures

[Tor Bug Tracker & Wiki]

#8662: Make Torperf log circuit failures
-------------------------+--------------------------------------------------
 Reporter:  karsten      |          Owner:                   
     Type:  enhancement  |         Status:  needs_information
 Priority:  normal       |      Milestone:                   
Component:  Torperf      |        Version:                   
 Keywords:               |         Parent:                   
   Points:               |   Actualpoints:                   
-------------------------+--------------------------------------------------

Comment(by karsten):

 Replying to [comment:4 mikeperry]:
 > JSON sounds like a good choice here for maximum extensibility and
 machine-readability.

 Great.

 > Later down the road, we may want some info on stream reliability and
 timeout data, too. JSON will make it easier to add those, too.

 What info on stream reliability and timeout data would that be?  The
 better we know what data we plan to gather, the better we can design the
 file format.

 > However, one area where you'll need to take care is how to handle
 grouping multiple failures. Based on the above example, it looks like you
 intend all circuits related to a fetch to be encoded in the same top-level
 JSON object for the fetch? Or do you intend to have separate objects for
 each failure?

 My original idea was to have separate objects for a) each file download
 including stream details, b) circuit (summary of CIRC events with same
 CircuitID), and c) BUILDTIMEOUT_SET event.  These objects would all be on
 the top level and contain references to each other.  In the example above,
 there's a field "circ_id": "1365473682.13_501" in the "file_download"
 event that refers to the "circuit" event.

 But you're right, it's quite possible that we use more than one circuit in
 a single fetch.  So far, whenever we detached a stream from a circuit and
 re-attached it to another circuit, we simply discarded the old circuit
 information.  We shouldn't do that.  Instead, we should keep a list of
 circuits to which a stream was attached, together with REASON and
 REMOTE_REASON information.

 We'll want to have a separate event type for each stream that contains a
 summary of STREAM events with the same StreamID.  This wasn't necessary so
 far, because we didn't really store anything about the stream we used for
 the measurement, but that is different now.

 Here's an updated example.  There could be several "circuit" events in
 this list, and the "stream" event could contain more than just one of them
 in its "circuits" field.

 {{{
 [
   {
     "type": "buildtimeout_set",
     "id": "1365473682.03",
     "timeout": 4833,
     "quantile": 0.800000
   },
   {
     "type": "circuit",
     "id": "1365473682.13_501",
     "launch": 1365473682.13,
     "path": [ "$62680CF0743460E5F836F949E37A6DEC22622F9E",
               "$11064D066F892DC38AAEFDA5EDAE1A227D07D182",
               "$35F51DCE73B988CBAE06B06312B4D1271979FE3B" ],
     "buildtimes": [ 0.53, 1.10, 1.32 ],
     "circ_id": 501,
     "buildtimeout_used": "1365473682.03"
   },
   { "type": "stream",
     "id": "1365473922.76_884",
     "circuits": [ { "circ_id": "1365473682.13_501",
                     "reason": "END",
                     "remote_reason": "DONE" } ]
   },
   {
     "type": "file_download",
     "id": "ec2_51200_1365473921.75",
     "source": "ec2",
     "filesize": "51200",
     "start": 1365473921.75,
     "socket": 1365473921.75,
     "connect": 1365473921.75,
     "negotiate": 1365473921.75,
     "request": 1365473921.75,
     "response": 1365473922.07,
     "datarequest": 1365473922.07,
     "dataresponse": 1365473922.40,
     "datacomplete": 1365473922.76,
     "writebytes": 82,
     "readbytes": 51323,
     "didtimeout": 0,
     "dataperc": { "10": 1365473922.56,
                   "20": 1365473922.64,
                   "30": 1365473922.64,
                   "40": 1365473922.68,
                   "50": 1365473922.68,
                   "60": 1365473922.72,
                   "70": 1365473922.72,
                   "80": 1365473922.75,
                   "90": 1365473922.75 },
     "stream": "1365473922.76_884"
   }
 ]
 }}}

 > Related: How does Torperf currently handle streams that detach from a
 circuit and get retried on another circuit? This can happen for example if
 you get an EXITPOLICY or HOSTUNREACH relay reason back before
 establishment, or if you hit the stream timeout. Probably some other cases
 too.

 (This should be handled by the new data format.)

 Thanks for your feedback so far!  This is very helpful.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/8662#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online