[metrics-bugs] #33076 [Metrics/Analysis]: Graph onionperf and consensus information from Rob's experiments

[Tor Bug Tracker & Wiki]

#33076: Graph onionperf and consensus information from Rob's experiments
-------------------------------------------------+-------------------------
 Reporter:  mikeperry                            |          Owner:
                                                 |  metrics-team
     Type:  task                                 |         Status:
                                                 |  needs_review
 Priority:  Medium                               |      Milestone:
Component:  Metrics/Analysis                     |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  metrics-team-roadmap-2020Q1, sbws-   |  Actual Points:  3
  roadmap                                        |
Parent ID:  #33121                               |         Points:  6
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by dennis.jackson):

 Using Karsten's code and dataset, I took another look at the onionperf
 measurements from over the course of Rob's experiment.

 == Time to First Byte

 This graph shows TTFB measurements over non-onion circuits from op-ab. The
 period in which the experiment was active is highlighted in red and the
 period in which the bandwidth values remained in the consensus is in
 orange. The lines show the 5th percentile, 50th percentile (median), the
 mean and the 95th percentile.

 [[Image(https://dennisjj.co.uk/tor-bandwidth-experiment/ttfb_public_op-
 ab.png,680px)]]

 The increase in congestion over the experimental window appears to be
 substantial, with the 95 percentile ttfb rising from under 2 seconds to
 over 10 seconds. The trend is much flatter for onion circuits.

 If we look at the raw data:

 [[Image(https://dennisjj.co.uk/tor-bandwidth-experiment/raw/exit_op-
 ab.png,680px)]]

 We see a band at the 10 second line. Karsten has already noted this in
 #31521 and his investigation suggests it happens when streams timeout from
 one circuit and get reattached to another. I wonder if some exit nodes
 could not keep up with the additional bandwidth leading to timeouts?

 == Bandwidth Measurements

 I have concerns about the reliability of the current dataset for the DL
 graphs. It contains only 2000 measurements over the month and none of the
 DL measurements appear to have finished successfully (have a non-null
 value for `getDataPercentiles().get(100)`). However, I might be
 misunderstanding onionperfs output format.

 == Further Thoughts

 * op-nl and op-usa reported no measurements between the 14th and the 16th
 of August, which makes it hard to use their data for this experiment.

 * There are a total of 10k ttfb measurements for the month of August. When
 you consider how many guard/middle/exit combinations there are, this
 really isn't that much data to work with. For example, it would have been
 interesting to drill down to how TTFB changed on the exit nodes whose
 weighting changed the most during this experiment, but there just aren't
 enough samples.

 * A full set of graphs, Jupyter Notebook with Python code and csv of the
 raw data:
 https://drive.google.com/open?id=1q1JRP5RdPEhQcDddh7KEST1aADMoiyt_

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33076#comment:23>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online