On 2023-08-30 14:31, Cecylia Bocovich wrote:
On 2023-08-30 14:28, Cecylia Bocovich wrote:
I was alerted by trinity-1686a on irc that Snowflake standalone proxy operators were reporting on #tor-relays about increased OOM errors from increased load as of 2023-08-28.
After looking at the Snowfake broker metrics[0], there's a huge jump in client polls (seen by summing the client-denied-count and client-snowflake-match-count).
I've attached a graph of the collected prometheus metrics that shows this spike happening at exactly 17:40 UTC on 2023-08-27. It looks like way too sharp an increase to me to be a censorship event, perhaps it is a DoS?
It's still too early to see the bridge metrics from the metrics page, but we should start to see the effects there tomorrow.
[0] https://metrics.torproject.org/collector.html#snowflake-stats
I should add that it looks to have stopped around 12:25 UTC earlier today (2023-08-30).
The bridge metrics show a small increase in usage on August 30th (the day it stopped), but not for August 28th or 29th, and definitely not proportional to the increase in client polls. This could indicate a DoS attack that only affected the broker, where connections are severed or the SDP information didn't contain enough information to establish a datachannel between the client and matched proxy. Or it could be that most of these polls were coming from a small set of IP addresses.
Whatever it was, the biggest consequence was a shortage of proxies with unrestricted NAT types, and that standalone proxies with the default configuration of no limit to the number of clients were OOM killed[0]
I haven't noticed any unusual activity again since August 30th.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
