[tor-relays] Cloudflare Onions Beta and Network Stability

DaKnOb daknob at daknob.net
Mon Aug 20 21:39:00 UTC 2018


Cloudflare had a post yesterday[1] on their blog[2] that said they have about 10,000,000 domain names using their service. So that’s a rough number of the maximum number of websites that will be made available over Tor. Now in reality I expect all their large customers to opt-out, unless it’s an opt-in, in which case I don’t expect large websites, only medium to small to join, let’s say <100,000. That said, depending on whether or not this is an opt-in, or opt-out, the number of websites can be from “hundreds of thousands” to “few millions”. 

In terms of traffic, I’d estimate Cloudflare to be in the “hundreds of Gb/s” to “a few Tb/s”, but their PeeringDB entry[3] doesn’t really say. In general they keep these numbers quite secret (never post chart Y axis, or post relative values). However, they have some info on their IX port capacities, which I believe is their largest bandwidth usage, but not all of it (also rely on private connections / transit): 5,350 Gb/s. This is almost 5.5 Tb/s, and it takes 2-3 minutes to add up all their ports manually from PeeringDB. Of course, this is their *capacity*, not their current bandwidth. All their ports are 10 Gb/s+, so even if they had 100 Mb/s they would go with 10G. Adding up their ports that are 20G+, we get 2,930 Gb/s, which is ~ 3 Tb/s. Even if we only count their 100G ports, we get 2.6 Tb/s.

Let’s assume now that Cloudflare has a daily peak of 2.6 Tb/s, so they run at 50% capacity during peak hours. Now let’s also assume that an 80-20 rule is in place. That said, 80% of their traffic is served for the top 20% of their customers. 

Now let’s make the worst case assumption that this service is opt-in, and their bottom 80% joins, while the top 20% does not.. That means that during peak time, this traffic is 520 Gb/s. 

So if Cloudflare has a peak of 2.6 Tb/s, and only 20% of its bandwidth-users participate, then that’s 520 Gb/s peak. On top of that, it’s also ~8M hidden services. 

Now I know that some of these numbers may not seem exactly right, and some are worst case scenarios, but here’s what it means for the Tor network.

Unlike Cloudflare, Tor actually does publish their numbers[4]!

Currently Tor has ~100,000 Onion Addresses[5]. If Cloudflare adds 8M more, then that’s a pretty huge increase, putting significant load on the directory servers. 

The current Tor Onion Service traffic is 1.25 Gb/s[6]. Adding 520 Gb/s to it is significant as well. 

The current Relay Bandwidth is currently ~270 Gb/s[7]. If we divide this by 3, we get ~ 90 Gb/s. So again, 520 Gb/s, is significant. 

HOWEVER, in terms of traffic, the above assumes that 100% of the visitors will be coming from Tor. This is far from the truth. Now let’s make a really generous estimation and say that 1% of the traffic to these websites (bandwidth-wise) will be from Tor (this is likely orders of magnitude smaller).

This means that the total Onion Service traffic will jump from 1.25 Gb/s to ~ 6.5 Gb/s, a significant increase. 

As far as the total relay bandwidth, the 90 Gb/s that are currently available will be enough, as current utilization is at ~43 Gb/s, so this will become ~75 Gb/s (actual traffic * 6).

So as you can see, in terms of traffic, Cloudflare won’t have a huge impact, since the current Tor network is more than 2 times above peak usage. 

HOWEVER, Cloudflare doesn’t need to hide their location. Everyone knows their servers. So they can use single hop Onion Services, and not the traditional three hop ones.

That means that in terms of total traffic, they will use 43 + (3*5) = ~ 60 Gb/s, out of the ~ 90 Gb/s available. 

As you can see, these are all estimations, and actual traffic will greatly vary. Excuse any mistakes since I’m writing this from my phone, without putting much thought into it. I also didn’t check any beta announcements they may have to see how they will implement this, I just imagined the most straight forward way to do so. 

From the numbers above I see the main concert is the amount of Onion Services, and not traffic, so to answer your question I guess it will be to work on that front, and not so much on the traffic side of things. 

In any case, I really hope that Cloudflare moves slowly with this, and in small batches, as it could cause trouble, especially after the 2-3M Onion Service mark (which is untested)..

Antonis


1: https://blog.cloudflare.com/african-traffic-growth-and-predictions-for-the-future/
2: https://blog.cloudflare.com/
3: https://www.peeringdb.com/net/4224
4: https://metrics.torproject.org/
5: https://metrics.torproject.org/hidserv-dir-onions-seen.html
6: https://metrics.torproject.org/hidserv-rend-relayed-cells.html
7: https://metrics.torproject.org/bandwidth.html

> On 20 Aug 2018, at 23:44, Robert Keizer <robert at keizer.ca> wrote:
> 
> Is there any kind of information about what kind of bandwidth /
> connections they are expecting to route?
> 
> Having a sense of scale in a ratio of current numbers I think would let
> everyone plan for what they're currently seeing multiply by X as a baseline.
> 
> 
>> On 2018-08-20 11:23 AM, Nathaniel Suchy wrote:
>> As some of you may have heard, Cloudflare is beta testing
>> opportunistic onions. This of course is going to create more Tor
>> traffic. Cloudflare has several concerns about running their own
>> relays and says they won’t at this time. That said if every Cloudflare
>> website becomes an Onion Service overnight how would that affect
>> network stability and what can we as relay operators do to prepare for it?
>> 
>> 
>> _______________________________________________
>> tor-relays mailing list
>> tor-relays at lists.torproject.org
>> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
> 
> 
> _______________________________________________
> tor-relays mailing list
> tor-relays at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20180821/6644370a/attachment-0001.html>


More information about the tor-relays mailing list