[tor-bugs] #2672 [Torperf]: Fix bugs/issues with consolidate_stats

[Tor Bug Tracker & Wiki]

#2672: Fix bugs/issues with consolidate_stats
----------------------------------------+-----------------------------------
 Reporter:  mikeperry                   |          Owner:  mikeperry
     Type:  defect                      |         Status:  assigned 
 Priority:  major                       |      Milestone:           
Component:  Torperf                     |        Version:           
 Keywords:  MikePerryIteration20110320  |         Parent:           
   Points:  2                           |   Actualpoints:           
----------------------------------------+-----------------------------------

Comment(by karsten):

 Replying to [comment:9 mikeperry]:
 > Ok, I think I fixed this. Now we store unmatched lines from either .data
 or .extradata, and should also be better about skipping missed lines from
 either. The result is a file that contains everything from .data and
 .extradata, even if they do not match.

 Yes, I tried the new `consolidate_stats.py` script, and it seems to work
 fine.

 > I also added the didtimeout field to the mergedata.

 Looks good.

 > All of this + the earlier changes should be in mikeperry/ticket2672.

 Some comments on that branch:

  - Can the note `"This script will strip all failure information from the
 output. [...]"` in `consolidate_stats.py` go away now?

  - I noticed that 158 of the 2428 lines in my sample `.data` file don't
 have a matching `.extradata` line with a `USED_AT` entry, but only 22 of
 these 158 runs actually timed out.  What could be the reason for
 `extra_stats.py` not setting `USED_AT` even though the circuit was
 actually used for a successful Torperf run?  I attached my data to this
 ticket; see `50kb.data-not-used` for the runs for which `extra_stats.py`
 finds no matching circuit.

  - You have a comment `"Add purpose to the .extradata?"` in your code.
 Should we just add it?  Or add a ticket for this?

  - You have another comment `"Torperf only uses 3 hop paths."`.  This
 seems to be correct, or at least I didn't find a case when we used a
 circuit with fewer or more than 3 hops in my sample data.  But why is your
 comment prefixed with `XXX`?

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/2672#comment:11>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online