[tor-relays] excessive bandwidth assigned bandwidth-limited exit relay

[Dhalgren Tor]

 Was going to wait a few days before reporting back, but early results
are decisive.

The overload situation continued to worsen over a two-day period, with
consensus weight continuing to rise despite the relay often running in
a state of extreme overload and performing its exit function quite
terribly.

Applied the following to the  server, which has a single network interface:

tc qdisc add dev eth0 handle ffff: ingress
tc filter add dev eth0 parent ffff: protocol ip prio 10 u32 match ip
protocol 6 0xff police rate 163600kbit burst 6mb drop flowid :1

The above two lines put in effect a "traffic control" ingress policing
filter that drops TCP protocol packets in excess of 163.6 MBPS.  This
is 13.6% above the original rate-limit target of 18Mbyte/sec of
payload data.  UDP traffic is not restricted so the rule will not
impact most 'unbound' resolver responses nor 'ntpd' replies.

Statistics for the filter are viewed with the command

tc -s filter show dev eth0 root

With the initial settings the bandwidth filter discarded 0.25% of
incoming TCP packets--in line with what one sees in 'netstat -s'
statistics for a not-overloaded relay.  However the 'tor' daemon went
straight back into the throttling state and, while improving slightly,
continued to operate unacceptably.

Next BandwidthRate was set to parity with the 'tc' filter or
20Mbyte/sec.  Result was a dramatic increase in dropped packets to
1.5% and a dramatic improvement of the responsiveness of the relay.
If fact the exit now runs great.

The bandwidth measurement system now looks like a secondary issue.
The big problem is that Tor daemon bandwidth throttling sucks and
should be avoided in favor of Linux kernel rate-limiting where actual
bandwidth exceeds bandwidth allocated to Tor, whatever the motivation.
TCP is much better at dealing with congestion than the relay internal
limit-rate logic.

The above 'tc' filter is simplest possible manifestation and works
well for an interface dedicated to Tor.  More nuanced rules can be
crafted for situations where Tor traffic coexists with other types
such that a limit will apply only to the Tor traffic.

-----

Note the applied 163.6mbps rate-limit is above the 150.0mbps rate that
works out to exactly 100TB per month of data.  The intent is to use
approximately 96% of the 100TB quota and the 'tc' rate will be
gradually fine-tuned to effect this outcome.  BandwidthRate and
BandwidthBurst are are now reset to the 1Gbyte no-effect default
value.

The 6mb buffer/burst chosen is an educated guess (very little
searchable info on this) and equates to 50ms at 1gbps or 300ms at
163mbps.  The number also corresponds with a bit more than what one
might reasonably expect as the per-port egress buffer capacity of a
enterprise-grade switch.