[tor-bugs] #27135 [Core Tor/sbws]: Write descriptor bandwidths average in raw results

[Tor Bug Tracker & Wiki]

#27135: Write descriptor bandwidths average in raw results
---------------------------+-------------------------------------
 Reporter:  juga           |          Owner:  juga
     Type:  defect         |         Status:  assigned
 Priority:  Medium         |      Milestone:  sbws 1.0 (MVP must)
Component:  Core Tor/sbws  |        Version:
 Severity:  Normal         |     Resolution:
 Keywords:                 |  Actual Points:
Parent ID:  #27108         |         Points:
 Reviewer:                 |        Sponsor:
---------------------------+-------------------------------------

Comment (by teor):

 Replying to [comment:14 juga]:
 > With a separate python script, i've collected descriptors' bandwidth
 during 3 days, every hour.
 > I've tried the method you proposed, but it still don't smooth out
 spikes. I'm quite sure it's because the descriptors' bandwidth don't
 change (much).

 Please retry with observed bandwidth,

 > I assumed Torflow's `desc_bw` and `avg_desc_bw` are the descriptor
 `average bandwidth`, but looking TorCtl.py [0] seems to be `observed
 bandwidth`, so i'm going to collect those instead and see how that change
 the curve.

 Yes, we want the observed bandwidth. The bandwidth rate and bandwidth
 burst come from the torrc, so they don't change much.

 > In any case, if Torflow is applying a decaying average, do you know
 where is that in the code?.

 Torflow uses a complicated calculation when pid control is active.
 This calculation attempts to apply a decaying weight to previous results,
 so that the feedback loop converges.
 (But the feedback loop does not converge, because this part of the torflow
 design is broken.)
 https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/aggregate.py#n117

 sbws needs to be able to operate in two different modes:
 1. Ignore self-reported bandwidth, so sbws can be more secure
 2. Use observed bandwidth to scale measurements, so sbws can be more like
 torflow

 Then, if sbws totals are different to torflow totals, we can apply a final
 linear scale step.

 Right now, we need to focus on getting the code working, and comparing
 sbws with torflow.

 Please use whatever observed bandwidth you have right now.

 Any of the following methods are ok:
 * the latest observed bandwidth
 * an average of recent observed bandwidths
 * a decaying average of recent observed bandwidths

 We'll also need a short explanation of how sbws works, so we can explain
 sbws to authority and relay operators.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/27135#comment:15>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online