Hi All
We (myself and my student, cc'ed here) have some questions regarding Tor relay bandwidth. We created a small testbed (private Tor network) involving relays, clients and servers over virtual machines. The client uses the relays to create two hop circtuits to communicate to the server. The link bandwidths between the virtual machines is otherwise ~ 10 Gbps (tested via end-to-end measurement tools like iperf and via downloading large files of sizes ~2 GB through https downloads).
When not using Tor, the client achieve a download rate of approximately 8 Gbit/s while downloading a file of approximately 2 Gbytes. The same process, when repeated over Tor circuit (through relays of the private Tor network, that used virtual machines), p*rovides a maxmium download rate **of about 700 Mbit/s for the client (while the CPU utilization was over 80%).* This is much less compared to the expected throughput of 8Gbps achieved when client communicates directly to the server (without using Tor circuits).
We tinkered around with the Tor config and discovered that when changed the config fields of BandwidthBurst and BandwidthRate to 100 MBytes, we achieved end-to-end throughput of about 100 Mbit/s (while the CPU Utilization was less than 20%) (while the underlying network link capacity is 1 Gbit/s). The moment the BandwidthBurst and BandwidthRate values are increased to 1 GB (which should effectively be 8 Gbit/s), the achieved throughput is about 650-700 Mbps (which is still 8 times slower than expected).
Further, removing all bandwidth restrictions, the inter VM bandwidth (for the relays in the private Tor network) is around 8 Gbit/s (measured both via Iperf and through downloading a file > 2 GBytes using encrypted HTTP traffic). While this is as per the expected behavior of TCP, to utilize all the available bandwidth, the same does not happen for Tor which continues to ramp upto a maximum of 700 Mbit/s
We are confused as to why Tor's link bandwidth utilization is not higher (Even on a private Tor network), even when using a high capacity VMs in the private Tor network -- each one having about 4 cores and 8 GBytes of RAM. I understand after speaking from some Tor developers and maintainers that Tor does not utilize all the available CPU cores. Our VMs however do not have any other process running either. We run the Tor processes over Debian servers with little no other programs. More strangely, while download via HTTPS achieves a very large fraction of the link capacity (>80%), Tor traffic happens to achieve a throughput of around 700 Mbit/s (which is around 90% lower than the max link bandwidth).
We tried the same with 2-hop Tor circuits, hoping to see some improvement. But to no avail. We continue to observe the same poor bandwidth utilization.
It would be great if you shed some light as to where we may be going wrong.
Regards Sambuddho
On 3/15/18 11:43, Sambuddho Chakravarty wrote:
When not using Tor, the client achieve a download rate of approximately 8 Gbit/s while downloading a file of approximately 2 Gbytes. The same process, when repeated over Tor circuit (through relays of the private Tor network, that used virtual machines), p*rovides a maxmium download rate **of about 700 Mbit/s for the client (while the CPU utilization was over 80%).* This is much less compared to the expected throughput of 8Gbps achieved when client communicates directly to the server (without using Tor circuits).
We tinkered around with the Tor config and discovered that when changed the config fields of BandwidthBurst and BandwidthRate to 100 MBytes, we achieved end-to-end throughput of about 100 Mbit/s (while the CPU Utilization was less than 20%) (while the underlying network link capacity is 1 Gbit/s). The moment the BandwidthBurst and BandwidthRate values are increased to 1 GB (which should effectively be 8 Gbit/s), the achieved throughput is about 650-700 Mbps (which is still 8 times slower than expected).
So you have two data points when using Tor.
1. Limit Tor to 100 MBytes --> only get 100 Mbps 2. Limit Tor to 1 GByte --> only get 700 Mbps
(1) sounds like it could be a real bug.
Both are approximately 8x slow downs, but I think that is coincidental. I would guess that if you limited Tor to 7 Gbits, 5 Gbits, etc. you would continue to see 700 Mbps. It would be interesting to see the result of this.
As I hypothesized to either you or your student on IRC, I believe your Tor relays as limited by CPU capacity (and its inability to use many cores effectively); therefore, as Tor currently exists, is unable to achieve a throughput of higher than 700 Mbps. Tor does a lot of crypto. Perhaps it is way less efficient at doing its crypto than the TLS library you were using for HTTPS downloads is at doing its crypto.
As another piece of evidence supporting my "700 Mbps is the max Tor can do" theory, see the list of the fastest Tor relays[0] in the real network. IPredator has likely done some heavy kernel tweaking, but the remaining relays are all rather close to 700 Mbps.
Thank you for your interest in Tor.
Matt
-
Matt Traudt -
Sambuddho Chakravarty