[tor-bugs] #3036 [Torperf]: Tweak Torperf's .mergedata format and make it the new default

[Tor Bug Tracker & Wiki]

#3036: Tweak Torperf's .mergedata format and make it the new default
-------------------------+--------------------------------------------------
 Reporter:  karsten      |          Owner:  karsten
     Type:  enhancement  |         Status:  new    
 Priority:  normal       |      Milestone:         
Component:  Torperf      |        Version:         
 Keywords:               |         Parent:         
   Points:               |   Actualpoints:         
-------------------------+--------------------------------------------------

Comment(by karsten):

 The archiving issues are now solved in metrics-db with Torperf's old .data
 and .extradata formats.  We should switch to the new format anyway.

 I attached the Java class that I used to produce siv's 5 MiB Torperf data
 file in the new format (which is also attached).  This class assumes that
 .data and .extradata files exist in the local working directory.  It
 produces a single .tpf file for every pair of .data and .extradata files.

 Next steps are:

  - Fix the TODOs in Convert.java and review that class in detail to be
 sure that all relevant parts of .data and .extradata files are contained
 in .tpf files.

  - Implement a mechanism to keep a state between executions to only read
 newly appended lines to .data and .extradata files.  Watch out for edge
 cases, e.g., when we cannot merge a line in the current execution but
 maybe in the next execution.

  - Merge results into daily .tpf files instead of a huge single .tpf file.
 Don't require old output files to be around forever, because we'll want to
 archive them at least monthly.

  - Integrate the updated Convert.java into metrics-db's
 TorperfDownloader.java which downloads possibly truncated Torperf .data
 and .extradata files.  That class should then output daily .tpf files.

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/3036#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online