[anti-censorship-team] Spike in client polls from Snowflake broker metrics

[Cecylia Bocovich]

On 2023-08-30 14:31, Cecylia Bocovich wrote:
> On 2023-08-30 14:28, Cecylia Bocovich wrote:
>> I was alerted by trinity-1686a on irc that Snowflake standalone proxy 
>> operators were reporting on #tor-relays about increased OOM errors 
>> from increased load as of 2023-08-28.
>>
>> After looking at the Snowfake broker metrics[0], there's a huge jump 
>> in client polls (seen by summing the client-denied-count and 
>> client-snowflake-match-count).
>>
>> I've attached a graph of the collected prometheus metrics that shows 
>> this spike happening at exactly 17:40 UTC on 2023-08-27. It looks like 
>> way too sharp an increase to me to be a censorship event, perhaps it 
>> is a DoS?
>>
>> It's still too early to see the bridge metrics from the metrics page, 
>> but we should start to see the effects there tomorrow.
>>
>> [0] https://metrics.torproject.org/collector.html#snowflake-stats
> 
> I should add that it looks to have stopped around 12:25 UTC earlier 
> today (2023-08-30).

The bridge metrics show a small increase in usage on August 30th (the 
day it stopped), but not for August 28th or 29th, and definitely not 
proportional to the increase in client polls. This could indicate a DoS 
attack that only affected the broker, where connections are severed or 
the SDP information didn't contain enough information to establish a 
datachannel between the client and matched proxy. Or it could be that 
most of these polls were coming from a small set of IP addresses.

Whatever it was, the biggest consequence was a shortage of proxies with 
unrestricted NAT types, and that standalone proxies with the default 
configuration of no limit to the number of clients were OOM killed[0]

I haven't noticed any unusual activity again since August 30th.

[0] 
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40101
-------------- next part --------------
A non-text attachment was scrubbed...
Name: userstats-bridge-transport-2023-08-20-2023-09-02-snowflake.png
Type: image/png
Size: 21568 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20230902/52b4816b/attachment-0001.png>