[tor-scaling] notes & next meeting

[Gaba]

Hi!

next meeting will be on
      - Friday 31st in May at 1600 UTC
      - Location: https://meet.jit.si/TorScalingMeeting

Link to the notes: https://nc.riseup.net/s/AEnQ4CRH2kH3fLe

Agenda (45 min)
   1. Recap plan (experiments; test network eval; standardization of
research)
   2. Questions to discuss (below)
   3. Open questions/discussion
   4. Next steps

Questions to discuss:
    1. Do we like these metrics?
    2. Can we reduce any of these metrics to numbers (CDF->N(u, stddev),
etc)
    3. Do we like these experiments? Some are a little complicated.. Are
any flawed? How about the control methodology?
    4. Do we like the simulation/test network evaluation idea?
    5. Do we like the kanban idea?
    6. How do we classify things in fuzzy stages (eg: walking onions -
has tor proposal but needs crypto peer review)

Experiments:
https://trac.torproject.org/projects/tor/wiki/org/roadmaps/CoreTor/PerformanceExperiments
R&D pipeline ‘kanban’:
https://storm.torproject.org/shared/k0SG5SfFTGGCsR0poyekmpSb-hLfLynJAUERHyMbGc4

Notes (thanks for taking notes)

started: 1605UTC

Mike recaps scaling mail and plans in the wiki page.

Mozilla question: is there something we can ship short term, that
doesn’t need to scale, but still provides good performance?

Answer: the performance metrics we’ve picked out are intended to help us
answer that kind of question: this plan will help us identify
low-hanging performance bottlenecks and fix them.

karsten points out that remembering tor versions will help is analysis:
often old relays are the reason for the surprising behavior.

metrics works best for things where the x axis is time. (that’s what
people are wanting to see.)

Roger points out four things to watch out for during these experiments:

Nick: we should do PIAs (Privacy Impact Assessments) on each experiment.
And communicate to users about the experiment. Consider doing A/B
testing on each of these, to still get good results. Figure out what our
fast-stop criterion is: in what cases will we stop early? We MUST tell
users about these plans before we start. [Roger thinks: how shall we
tell users best?] Anonymity-impacting changes requires much more
analysis than other experiments.  We should make sure that we are only
doing experiments that we believe will help, and won’t have anonymity
dangers. Mike: The field for this is “Anonymity Effects” on the
experiment list

Roger notes that there are many other pieces to the performance map:
changes tor browser should make for its users; better bandwidth
scanners; the exit map stuff Arthur talked about. Where should we
collect these ideas?
Answer: let’s get them in the Kanban. Look at the Kanban and send Mike
things that are missing.

New Experiment fields:
Fast Stop Criteria (stop experiment if..)
Debugging/Instrumentation Information
Evaluation of Need to Inform Users (aka user impact?)
Nick claims that we ALWAYS need to inform users, and that we should NOT
do experiments that we are worried will harm anonymity/security/privacy.

Karsten suggests that we try in simulation first, to make sure we’re not
completely wrong. Mike answers that the chosen experiments have been
simulated many times which is why they were chosen, and that this is
something we should also tell users when informing them. Moreover, we’re
doing this to determine which simulators are an accurate reflection of
the Tor network in the first place — this has never been evaluated by us.

Q: how should we allocate our efforts, between improving the simulation
models vs thinking through better experiments (better things to toggle)?
A: it’s too early to tell really. We don’t know how good the simulators
are. So step one is to get a handle on how well the simulators match the
live network, so we can at least know what to expect (which features
should match, which features we know won’t match, etc).


Next steps:
- Convene with everybody who’s attending All Hands. Loop Rob into this
planning.
- Pick one experiment that we’re comfortable toggling on the live Tor
network?
- Remember that our goal here is to produce a plan that we would do once
we have funding for all this.
- More conversations all around.
- Get Roger’s list of “other performance areas we should consider” onto
the Kanban.
- Instrument clients and relays: design and make changes so that we can
measure whether things are working as expected in each experiment. Roll
- them out now/soon so they’re available when we want to run the
experiments.
- Metrics will add three new graphs, to better present performance data
to - users. tgen has more parameters we can explore. tor itself exports
error codes. #29787
- schedule a next scaling meeting before moz all hands.
- brainstorm if there are other people we should include in this process.



-- 
Project Manager: Network, Anti-Censorship and Metrics teams and OONI support
gaba at torproject.org
she/her are my pronouns
GPG Fingerprint EE3F DF5C AD91 643C 21BE  8370 180D B06C 59CA BD19