[anti-censorship-team] How much traffic is Snowflake sending to stunprotocol.org

[Cecylia Bocovich]

On 12/26/22 00:55, John Selbie wrote:
> Greetings,
>
> I hope this is an appropriate mailing list to discuss a technical 
> issue with Tor's Snowflake project.  Please redirect me to the right 
> place if not.
>
> I am the original author and maintainer of the open source project, 
> Stuntman.  Stuntman is an implementation of the STUN protocol, which 
> includes the STUN server. More details at www.stunprotocol.org 
> <http://www.stunprotocol.org>.  In short, a STUN server helps 
> bootstream direct "p2p" connections such as WebRTC sessions or similar 
> VOIP scenarios by allowing internet devices to self-discover their own 
> public IP address and obtain a (UDP) port for communicating with 
> another node.
>
> I also run a public instance of a STUN server with the code at 
> stun.stunprotocol.org <http://stun.stunprotocol.org>.  It's been up 
> and running for about 10 years now. It's hosted on AWS.  In recent 
> years, the hosting bills for this server have started to get on the 
> high side, even with reserved instances.  The number of STUN queries 
> it processes per day is now on the order of hundreds of millions. The 
> stunprotocol.org <http://stunprotocol.org> domain receives nearly a 
> million DNS queries on Route 53 daily. What used to cost a trivial 
> number of dollars to run is now starting to reach $1000 in annual 
> service costs. This isn't paid for by a corporation or well funded 
> internet organization. I pay this out of my personal pocket.
>
> It's been a mystery what has been driving the increasing traffic to 
> the server - especially redundant requests from the same IPs.  I was 
> inspecting the DNS logs the other day and started to investigate the 
> nodes sending out redundant DNS requests repetitively.  Trying to 
> understand why these nodes wouldn't leverage DNS caching.  And to my 
> surprise, one of the IPs was running a web server that presented a TOR 
> landing page.  That led me to discover this discussion online:
>
> https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/30579
>
> And a quick inspection of the Snowflake code leads me to find that 
> stun.stunprotocol.org <http://stun.stunprotocol.org> is the default 
> STUN server for Snowflake proxy and listed throughout the 
> documentation as well.
>
> While the Snowflake project has good intentions, it doesn't appear to 
> take my hosting costs into consideration. I'm hoping we can have a 
> good discussion on the following:
>
> 1) How many snowflake clients and proxies are active and how many STUN 
> requests are each generating towards stunprotocol.org 
> <http://stunprotocol.org>? Do we think the entire worldwide usage of 
> Snowflake could be responsible for millions of STUN queries to 
> stunprotocol.org <http://stunprotocol.org> per day?
>
> 2) Expected number of DNS queries (it's a 3-day TTL on these DNS 
> entries, so it blows my mind that there are so many redundant 
> requests). Does Pion or any other part of the Snowflake code tend to 
> go direct to the namespace server itself?
>
> 3) Removing stun.stunprotocol.org <http://stun.stunprotocol.org> as 
> the default STUN server.
>
> OR...
>
> 4) Alternatively, I'm always open to accepting donations to help run 
> the service costs of stunprotocol.org <http://stunprotocol.org>. I'm 
> definitely not getting rich running this thing.
>
> Thanks,
> John Selbie


Hi John,

Thank you for reaching out. This is exactly the right place to discuss 
this. This was an oversight on my part not to reach out to you as the 
operator of our configured default STUN server and I'm very sorry for 
the unexpected increased costs. We can absolutely remove 
stunprotocol.org as the default.

First, to answer some of your questions:

1) I would definitely believe the amount of snowflake traffic to 
stunprotocol.org to be this high. We have over 100,000 proxies. 
According to recent metrics[0], there are around 8 million matches a day 
and therefore that many WebRTC ICE gathering requests coming from just 
the proxies. The clients use a randomized subset of configured STUN 
servers, so the number is slightly different but it's safe to say they 
are also generating a few million STUN queries to your server.

2) I'm not sure about the DNS queries. It also surprises me that there 
are this many, I'll open an issue to investigate why.

Now for immediate next steps. I've sent an email to people internally to 
start the process of looking into sending some funds your way for the 
costs. We might not get an answer until after everyone is back at work 
in January. In the meantime:

- I'd like to remove stunprotocol.org as the default STUN server for the 
proxies. The reason we added to our list in the first place was because 
it implements RFC 5780 which we are using on the client side to 
determine NAT matching and filtering types (thank you for this 
implementation BTW!). The proxies no longer use this however, so there's 
no reason we can't have them use any number of other public STUN servers.

- We can remove stunprotocol.org from the list of *default* client STUN 
servers and reserve it for a small subset of users instead. This would 
really cut down on the traffic, but keep it open as an option for 
clients in places that have blocked other industry STUN servers.

How does this sound to you?

Thanks again for all the work you've done on Stuntman and in running a 
public server. I'm glad you reached about this.

- Cecylia

[0] https://snowflake-broker.bamsoftware.com/metrics