[metrics-bugs] #33076 [Metrics/Analysis]: Graph consensus and vote information from Rob's experiments (was: Graph onionperf and consensus information from Rob's experiments)

Tor Bug Tracker & Wiki blackhole at torproject.org
Wed Jun 10 14:53:25 UTC 2020


#33076: Graph consensus and vote information from Rob's experiments
-------------------------------------------------+-------------------------
 Reporter:  mikeperry                            |          Owner:
                                                 |  metrics-team
     Type:  task                                 |         Status:
                                                 |  needs_review
 Priority:  Medium                               |      Milestone:
Component:  Metrics/Analysis                     |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:  metrics-team-roadmap-2020, sbws-     |  Actual Points:  3
  roadmap                                        |
Parent ID:  #33327                               |         Points:  6
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Changes (by karsten):

 * sponsor:  Sponsor59-must =>


Old description:

> This is a ticket for the work to graph the historical onionperf data from
> Rob's relay flooding experiment.
>
> Some discussion threads:
> https://lists.torproject.org/pipermail/tor-
> scaling/2019-December/000077.html
> https://lists.torproject.org/pipermail/tor-
> scaling/2020-January/000081.html
>
> Basically, we want to have a standard way to graph results from key
> metrics from before, during, and after the experiment.
>
> In this case, we want CDF-TTFB, CDF-DL from onionperf results.
>
> We also want CDF-Relay-Stream-Capacity and CDF-Relay-Utilization for the
> consensus, as well as from the votes, to see if the votes from TorFlow
> drastically differ from sbws during the experiment.
>
> https://trac.torproject.org/projects/tor/wiki/org/roadmaps/CoreTor/PerformanceMetrics

New description:

 This is a ticket for the work to graph the historical onionperf data from
 Rob's relay flooding experiment.

 Some discussion threads:
 https://lists.torproject.org/pipermail/tor-
 scaling/2019-December/000077.html
 https://lists.torproject.org/pipermail/tor-
 scaling/2020-January/000081.html

 Basically, we want to have a standard way to graph results from key
 metrics from before, during, and after the experiment.

 In this case, we want CDF-TTFB, CDF-DL from onionperf results.

 We also want CDF-Relay-Stream-Capacity and CDF-Relay-Utilization for the
 consensus, as well as from the votes, to see if the votes from TorFlow
 drastically differ from sbws during the experiment.

 https://trac.torproject.org/projects/tor/wiki/org/roadmaps/CoreTor/PerformanceMetrics

 '''Update from June 10, 2020: We finished the CDF-TTFB and CDF-DL portions
 by adding these graphs to OnionPerf's visualize mode. The remaining parts
 are the CDF-Relay-* graphs that are based on consensuses and votes. Keep
 this in mind when reading comments up to June 10, 2020.'''

--

Comment:

 This is quite the ticket so far, with lots of attachments and comments.
 Time to join the various threads and summarize what's left to do:

 The first part is CDF-TTFB and CDF-DL graphs:
  - OnionPerf's visualize mode are very soon going to support CDF-TTFB and
 CDF-DL graphs. The only remaining piece is the #33257 review, but I don't
 expect major changes there except for maybe bug fixes. I attached the
 output of op-ab's measurements before/during/after the experiment
 discussed in this ticket.
  - Pages 1 and 2 show CDF-TTFB for the public/onion service cases. Note
 that, in theory, I could have added 6 more lines to these graphs by adding
 6 more data sets to the visualize command. It's unclear how readable the
 graph would have been, so I decided against it.
  - Pages 32 and 33 show CDF-DL for the public/onion service cases.
  - We briefly discussed Dennis' question whether failed measurements are
 included in the CDF-TTFB graph or not. They are not, and even though it
 would be possible to include them in the CDF-TTFB graphs as `Inf` values
 and in the CDF-DL graphs as `-Inf` values, I'm not convinced that it's a
 good idea. If failure rates differ a lot between the data sets we'll see
 that in the error graphs. If there's disagreement about this case, I'd
 like us to create a new ticket and discuss this topic there.
  - Dennis added a cool graph showing op-ab's TTFB over time as rolling 24
 hour values. It showed quite well how the 95th percentile grows to over 10
 seconds during the experiment and drops to much smaller values after the
 experiment. However, the same thing can be seen in the scatter plot on
 page 3 of the PDF I just attached. It wouldn't be hard to add another
 graph like Dennis', also because we're now using pandas just like Dennis
 did for this graph. I'm just not sure whether it's worth the additional
 effort. I'd say if somebody wants to have this graph and ideally provide a
 patch, let's open a new ticket for that enhancement.
  - This concludes the work on the CDF-TTFB and CDF-DL graphs. If we need
 more graphs containing OnionPerf measurement data, let's open new tickets
 for them.

 The other remaining part is CDF-Relay-* graphs:
  - We don't have these graphs in OnionPerf, because we'll need Tor
 directory data in order to make them, and that's not available in
 OnionPerf yet. We also said that these graphs are out of scope for Sponsor
 59.
  - Mike said that he's still puzzling out the differences when we used
 peak_observed (the 02-04 graphs) vs observed (01-29b graphs). He said he
 needs to think on that a bit more and may have another graph request.
  - I'm changing this ticket to discuss these remaining graphs. This
 includes updating the summary and description and removing the sponsor
 tag. I'll leave it in needs_review for mikeperry to comment on the
 difference mentioned in the previous bullet point. I could as well have
 opened a new ticket, but that would have meant carrying over a lot of
 context from this ticket, and that seemed like a lot of work.

 If anything else remains to be done, please comment here or open a new
 ticket for that. Thanks, everyone!

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33076#comment:32>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online


More information about the metrics-bugs mailing list