[anti-censorship-team] Crash in Snowflake users since 2021-01-17, recovery since 2021-01-21

[David Fifield]

On Mon, Jan 18, 2021 at 09:30:53PM -0700, David Fifield wrote:
> I happened to look at the Snowflake bridge's bandwidth and user history
> page, and the counts are close to zero since yesterday. I haven't
> investigated it at all.
> 
> https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F

The measured decrease in users occurred on 2021-01-17 and lasted 3–4
days. Since 2021-01-21 there has been a recovery almost to previous
levels.

I had mentally attributed the drop to the tor bug that made default
bridges not work in Tor Browser 10.5a7.
https://gitlab.torproject.org/tpo/applications/tor-browser/-/issues/40282

But the timing doesn't line up. The measured decrease happened *before*
the release of 10.5a7 on 2021-01-20. Tor Browser 10.5a8 was just
released with a fix for tor-browser!40282, but the recovery in Snowflake
user numbers happened before that.
10.5a7	2021-01-20	https://blog.torproject.org/new-release-tor-browser-105a7
10.5a8	2021-01-26	https://blog.torproject.org/new-release-tor-browser-105a8

At the team meeting today, cohosh suggested that the cause may not
actually be tor-browser!40282, but rather an unexplained drop in the
number of NAT-unrestricted proxies starting 2021-01-20:
https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40030#note_2722715
http://meetbot.debian.net/tor-meeting/2021/tor-meeting.2021-01-28-15.58.log.html#l-19
> 16:03:24 <dcf1> I'm planning to start an email thread but I'll just mention here, Snowflake user numbers recovered even before the 10.5a8 release
> 16:03:31 <dcf1> https://metrics.torproject.org/rs.html#details/5481936581E23D2D178105D44DB6915AB06BFB7F
> 16:03:45 <cohosh> dcf1: yeah i noticed that too
> 16:04:03 <cohosh> i wonder if it had something to do with the weird broker metrics we were getting
> 16:04:04 <dcf1> so I don't know if people found their own workarounds, or if the majority of our users are not Tor Browser users, or what
> 16:04:23 <cohosh> whereall of our proxies suddenly didn't know what kind of NAT they had
> 16:05:34 <dcf1> that might be a better explanation, since the non-working 10.5a7 was released 2021-01-20 and the decrease in users happened *before* that
> 16:06:14 <cohosh> i rolled out a fix that should prevent that from happening again
> 16:06:56 <cohosh> but it's weird that it dropped to zero
> 16:07:05 <cohosh> the number of clients i mean
-------------- next part --------------
A non-text attachment was scrubbed...
Name: flakey-onionoo-20210128.png
Type: image/png
Size: 40564 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/anti-censorship-team/attachments/20210128/3e9ef3d7/attachment.png>