[tor-relays] Consensus Weight calculation

[Vort]

> Thanks for writing to us.

Thanks for the answers.


> This is a question that gets asked a lot:
> "Many people set up new fast relays and then wonder why their bandwidth
> is not fully loaded instantly…"
> https://blog.torproject.org/blog/lifecycle-of-a-new-relay

Maybe this is correct in many cases.
But definitely not in all of them.

For example, this line:
"once the bwauths have measured you and the directory authorities lift
  the 20KB cap, you'll attract more and more traffic"
Events can go other way: bwauths will assign lower weight, and relay
  will be getting less and less traffic.


> It can take a week or two for the bandwidth authorities to measure a
> relay.

Relay, which hit the problem, can be in underpowered state for months.


> I'm not sure if this is a problem. And I'm not sure how many relays it
> impacts.

Hundreds, I guess.

Here is some examples:

https://atlas.torproject.org/#details/9FC2673BB2704C2AAB851F8334938565DF1D0819
Now used bandwidth:     1    KiB/s
Advertised Bandwidth: 131.38 KiB/s
Top used bandwidth:   250    KiB/s
Bandwidth rate:      4000    KiB/s

https://atlas.torproject.org/#details/B918EB3FA4D03A4F9F632AA17F217A6C04044EF7
Now used bandwidth:     1    KiB/s
Advertised Bandwidth:  82.65 KiB/s
Top used bandwidth:   245    KiB/s
Bandwidth rate:       800    KiB/s

https://atlas.torproject.org/#details/DF1C6C645C5854780778A3E81D12F2A8FF65744B
Now used bandwidth:     1    KiB/s
Advertised Bandwidth:  62.29 KiB/s
Top used bandwidth:     7    KiB/s
Bandwidth rate:      3000    KiB/s

https://atlas.torproject.org/#details/E2AF5879F39FF40DF8994E9B8FAEAB2518AEEBA4
Now used bandwidth:     1    KiB/s
Advertised Bandwidth:  70.94 KiB/s
Top used bandwidth:   916    KiB/s
Bandwidth rate:      1000    KiB/s


As you can see, most of them can handle a lot more traffic: 50x-4000x.
Also don't see why they can have high latency.
Good relays, on my opinion.


> But we know there is a bias in Tor's measurements towards North America
> and Europe, because that's where most of the measurements are made from:

No, this have no impact in this case.

I have launched my own instance of BwAuthority and I see, that measured
  "filt_bw" values are pretty close to "Advertised Bandwidth":

node_id=$9FC2673BB2704C2AAB851F8334938565DF1D0819 nick=qq strm_bw=52732 filt_bw=77967 circ_fail_rate=0.0 desc_bw=134537 ns_bw=13000
node_id=$9FC2673BB2704C2AAB851F8334938565DF1D0819 nick=qq strm_bw=61278 filt_bw=70430 circ_fail_rate=0.0 desc_bw=85495 ns_bw=13000
node_id=$B918EB3FA4D03A4F9F632AA17F217A6C04044EF7 nick=TranTor strm_bw=40485 filt_bw=47052 circ_fail_rate=0.0 desc_bw=84635 ns_bw=12000

The problem is on the next step, I think.


>> The result has revealed some anomalies:
>>  https://s8.hostingkartinok.com/uploads/images/2017/06/fed1cf8b57fc027223c8eaf3deb0d28a.png
>> First, and most important, - a lot of relays have bandwidth estimate
>>  in range 0-50: 1082 of them.

> I don't know what each axis is on this graph.

x is KiB/s, y is count
(yellow bars are for "Advertised Bandwidth", blue - for
 "Consensus Weight", grey mean both values)

> 20 is the default, 50 is the maximum for a relay's self-test.
> If a relay isn't measured, or measures very low, it usually gets a
> figure in this range.

I have excluded non-measured relays from this histogram.

>> Second - there are incorrect estimates
>>  for popular bandwidths of 5, 10 and 20 MBits.

> I don't understand what you mean here. The advertised bandwidth is in
> kilobytes per second, and the consensus weight is dimensionless (but
> scaled from kilobytes per second).

> Can you point out the lines you mean?

Look at the yellow spike at x = ~1200.
Low blue bars at the same point means that "Consensus Weight" model
  did not take into account that there are many 1200 KiB/s nodes on
  the network, which will result in theirs underload.


-- Vort