Here are some graphs of the bandwidth at the 2 Snowflake bridges since
the end of domain fronting on Fastly on 2024-03-01
(https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/135)
You can compare this to similar analysis made in October 2023, after a
similar situation when the default front domain changed from Fastly to
Cloudflare:
https://lists.torproject.org/pipermail/anti-censorship-team/2023-October/00…
1. As before, snowflake-02 was relatively harder hit than snowflake-01.
But this time it didn't get nearly as close to zero. However, there's
an evident secondary step down on 2024-03-07, which may or may not be
related.
2. Also as before, snowflake-02 suddenly (within 10 minutes) recovered a
large amount of bandwidth about 14 days later, at 2024-03-15 10:45. A
similar sudden increase is visible with snowflake-01 at the same
time. Last October, it (if it is indeed the same effect) happened at
about 11:45, one hour later UTC.
Hi all,
While updating Briar's bridge config I noticed that Moat is still
returning a Fastly URL among its snowflake bridge lines for some
countries. Are these lines still working (temporarily?) or should I
substitute the CDN77 front?
Thanks,
Michael
On Mon, Jan 22, 2024 at 04:33:53PM +0100, ERIK NORDBERG wrote:
> This is a great report.
> I would like to post it as it is as ’2023 year in review' on https://
> opencollective.com/censorship-circumvention/projects/snowflake-daily-operat…
> /updates.
> No need to strip it to cover only snowflake-01, as I see it.
> OK?
Yes, that's OK. Here's the slightly edited text I posted.
---
This is a summary of usage of the [Snowflake](https://snowflake.torproject.org/) pluggable transport in 2023. For the previous year's summary, see the [Snowflake 2022 year in review](https://opencollective.com/censorship-circumvention/projects/snowfl….
The primary Snowflake bridge, called snowflake-01, is supported by donations. In 2023, the [Snowflake Daily Operations team](https://opencollective.com/censorship-circumvention/projects/snowflak… paid about [€4,820](https://opencollective.com/censorship-circumvention/projects/snowflake-daily-operations/expenses?collectiveType=projects&parentCollectiveSlug=censorship-circumvention&collectiveSlug=snowflake-daily-operations&period=2023-01-01T00%3A00%3A00.000Z%E2%86%922023-12-31T23%3A59%3A59.999Z%7EUTC) in hosting expenses to keep the bridge running—with most of that going for bandwidth. In 2024, in addition to paying for bandwidth, we would like to buy new hardware for the bridge, which will enable it to support more users at the busiest times of day. You can help by [donating to support the bridge](https://opencollective.com/censorship-circumvention/projects/snowfl….
Let's start by looking at the daily users and bandwidth. Tor user graphs show an estimate of the [average number of simultaneously connected users](https://metrics.torproject.org/reproducible-metrics.html#users) (not unique IP addresses). For instance, if the line passes through 40,000 on a particular day, it means there were, on average, 40,000 users connected to Snowflake at any given time during that day. The bandwidth graph shows the total amount of Tor traffic transferred per day. As of January 1, 2024, Snowflake supported an average of about 40,000 simultaneous users and transferred about 35 TB of circumvention traffic per day. While the number of users is lower than it was at the beginning of the year, bandwidth is up by more than 50%.
[snowflake-userstats-bridge-transport-2023.png]
[snowflake-bandwidth-2023.png]
These are some of the events that affected Snowflake usage in 2023:
* Between 2023-01-16 and 2023-01-24, and intermittently thereafter, domain fronting rendezvous was [blocked in parts of Iran](https://bugs.torproject.org/tpo/anti-censorship/team/115). As more Snowflake users are in Iran than in any other country, this resulted in an observable decrease in the number of users.
* On 2023-03-13, the Tor anti-censorship team [fixed a security bug](https://forum.torproject.org/t/security-advisory-cross-user-tls-traffi… that had also negatively affected performance since September 2022. This greatly improved bandwidth for all users.
* The big drop in users on 2023-09-20 was not caused by a censor. Rather, it was an [unexpected change](https://forum.torproject.org/t/problems-with-snowflake-since-2023-0… in the infrastructure used by Snowflake for rendezvous. Mitigating the problem required new software releases.
Before 2023, we had used just one, centralized Snowflake bridge. Last year we started using [a second bridge](https://bugs.torproject.org/tpo/anti-censorship/pluggable-transport… in order to scale to more users. Here are the same user and bandwidth graphs, but with the snowflake-01 and snowflake-02 bridges shown separately. Strangely, the domain front change of 2023-09-20 [affected the two bridges differently](https://lists.torproject.org/pipermail/anti-censorship-team/20….
[snowflake-userstats-bridge-transport-multi-2023.png]
[snowflake-bandwidth-multi-2023.png]
We can also break down users by country. The largest contingent of Snowflake users are in Iran, which has been the case since [the Mahsa Amini protests in 2022](https://ooni.org/post/2022-iran-blocks-social-media-mahsa-amini-prote…. The graph shows also a large number of users apparently from the United States, but we believe that may be partly the result of geolocation errors, and many of them are [actually from Iran](https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/…. After Iran, the countries with the most Snowflake users are Russia and China. On 2023-04-10 the team [fixed a bug](https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/s… in the browser proxy that caused some users to be wrongly attributed to the unknown "??" country code. The cause of the per-country user count estimates becoming less precise between October and December was a [user-counting bug](https://bugs.torproject.org/tpo/core/tor/40871) in Tor that existed from version 0.4.8.4 until 0.4.8.10.
[snowflake-top-countries-2023.png]
Hi,
I started asking questions in #tor-anticensorship about how traffic can
be directed towards certain snowflake bridges. Moving my questions here
to reach more/other people.
--8<---------------cut here---------------start------------->8---
Hey all. In trying to make a 2024 budget for the [Snowflake Operations][]
project operating snowflake-01.tpn I need a better understanding of how
we direct traffic to the running bridges. Both what potential challenges
there are to do it and what the policy for it looks like. The background
is that snowflake-01 is close to going full due to CPU consumption. I
haven't spotted any flat lines yet but have seen momentary CPU
utilisation of 98% a couple of times.
Here are two of the questions I'm looking for an answer to.
1. If we get another server, similar to snowflake-01 wrt performance,
will it be useful to the network? Ie will it offload snowflake-01?
2. If we do **not** get another server and snowflake-01 goes full, will
users have a bad network experience as a result of this? Can traffic be
moved to snowflake-02?
[Snowflake Operations]: https://opencollective.com/censorship-circumvention/projects/snowflake-dail…
--8<---------------cut here---------------end--------------->8---
I've drafted a year-in-review post for Snowflake users in 2023 that
doubles as a request for donations for running the bridge. I'm planning
to post it to the Tor forum.
Is there anything to add?
Erik: is the quoted hosting costs of €4,800 correct?
The post discusses both existing bridges. We could also do one
specifically for snowflake-01 to post to the Open Collective.
---
This is a summary of usage of the
[Snowflake](https://snowflake.torproject.org/) pluggable transport in
2023. For the previous year's summary, see the
[Snowflake 2022 year in review](https://opencollective.com/censorship-circumvention/projects/snowfl….
The primary Snowflake bridge, called snowflake-01, is supported by
donations. In 2023, the
[Snowflake Daily Operations team](https://opencollective.com/censorship-circumvention/projects/snowflak…
spent about
[€4,800](https://opencollective.com/censorship-circumvention/projects/snowflake-daily-operations/expenses?collectiveType=projects&parentCollectiveSlug=censorship-circumvention&collectiveSlug=snowflake-daily-operations&period=2023-01-01T00%3A00%3A00.000Z%E2%86%922023-12-31T23%3A59%3A59.999Z%7EUTC)
to keep the bridge running—with most of that going to pay for bandwidth.
In 2024, we would like to buy new hardware for the bridge, which will
enable it to support more users at the busiest times of day. You can
help by
[donating to support the bridge](https://opencollective.com/censorship-circumvention/projects/snowfl….
[Donate funds for the snowflake-01 bridge](https://opencollective.com/censorship-circumvention/projects/snowfl…
Let's start by looking at the daily users and bandwidth. Tor user graphs
show an estimate of the
[average number of simultaneously connected users](https://metrics.torproject.org/reproducible-metrics.html#users).
For instance, if the line passes through 50,000, it means there were, on
average, 50,000 users using Snowflake at the same time on that day. The
bandwidth graph shows the total amount of Tor traffic transferred per
day.
While there were fewer users at the end than at the beginning of the
year, Snowflake users are using greater total bandwith, now about 35 TB
per day.
[snowflake-userstats-bridge-transport-2023.png]
[snowflake-bandwidth-2023.png]
These are some of the events that affected Snowflake usage in 2023:
* Between 2023-01-16 and 2023-01-24, and intermittently thereafter,
domain fronting rendezvous was
[blocked in parts of Iran](https://bugs.torproject.org/tpo/anti-censorship/team/115).
As more Snowflake users are in Iran than in any other country, this
resulted in an observable decrease in the number of users.
* On 2023-03-13, the Tor anti-censorship team
[fixed a security bug](https://forum.torproject.org/t/security-advisory-cross-user-tls-traffi…
that had also negatively affected performance since September 2022.
This greatly improved bandwidth for all users.
* The big drop in users on 2023-09-20 was not caused by a censor.
Rather, it was an
[unexpected change](https://forum.torproject.org/t/problems-with-snowflake-since-2023-0… in the infrastructure
used by Snowflake for rendezvous. Mitigating the problem required new
software releases.
Before 2023, we had used just one, centralized Snowflake bridge. Last
year we started using
[a second bridge](https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transpo…
in order to scale to more users. Here are the same user and bandwidth
graphs, but with the snowflake-01 and snowflake-02 bridges shown
separately:
[snowflake-userstats-bridge-transport-multi-2023.png]
[snowflake-bandwidth-multi-2023.png]
We can also break down users by country. The largest contingent of
Snowflake users are in Iran, which has been the case since
[the Mahsa Amini protests in 2022](https://ooni.org/post/2022-iran-blocks-social-media-mahsa-amini-prote….
The graph shows also a large number of users apparently from the United
States, but we believe that may be partly the result of geolocation
errors, and many of them are
[actually from Iran](https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/….
After Iran, the countries with the most Snowflake users are Russia and
China. On 2023-04-10 the team
[fixed a bug](https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/s…
in the browser proxy that caused some users to be wrongly attributed to
the unknown `??` country code. The reason for the per-country estimates
becoming less precise from October to December was a
[user-counting bug](https://bugs.torproject.org/tpo/core/tor/40871)
in Tor that existed from version 0.4.8.4 until 0.4.8.10.
[snowflake-top-countries-2023.png]
We received the following email from Fastly today. They have also
included a customer report in .csv format that shows accesses to
snowflake-broker.torproject.net.global.prod.fastly.net and
moat.torproject.org.global.prod.fastly.net with a variety of front
domains including those we currently recommend.
---
Hello The Tor Project,
Fastly is committed to improving the security of our platform for all
our users. One area we are working on is in enforcing the association
between a TLS certificate’s SAN entries and the hostname in the HTTP
request’s host header.
We will be forbidding domain fronting from happening by restricting it
on a shared offset you might depend upon. This change will be applied
during February 27th, 2024 .
Here are a few things to highlight based on our previous conversations
with customers:
*Why block Domain Fronting now?*
We want to block external malicious actors from utilizing domain
fronting for our customers.
*Does Domain Fronting cause immediate impact?*
Existing domain fronting requests will be allowed. Any new domain
fronting requests would be blocked. The exception for the existing
domain fronting requests would be in place until the cert used for the
request(s) expires or is replaced.
*What does this mean?*
The earliest cert expiration is shown in the “fastlycertificatedetail”
column in the domain fronting report.. This means that even if we block
domain fronting today, you will have until the cert expires before
impacts to domains will be seen. However, new domain fronting requests
would be blocked.
*What does the report show?*
The purpose of the report is to provide visibility to you regarding
external requests that are currently defined as domain fronting. These
requests may be external requests that have explicit purpose to perform
domain fronting and some requests may be requests that you currently use
for the operations of your application.
Excluded from this report are services that are service chained or use
shielding which will continue to work.
*What is Fastly's ask?*
Review the report and take action accordingly.
Actions may include but not limited to:
- *Do nothing* and allow new requests to be blocked after the
certificate expires.
- *Change Code* to provide the necessary SNI and hostname in TLS
requests. This needs to be completed before the certificate expires.
- *Update Fastly TLS settings* to ensure that your service domains have
a corresponding Fastly TLS domains.
During last week's meeting we had a conversation on the future of Pluggable
Transports. Our current pt protocol[0] have many limitations, the two main ones
are:
* We have to do hacks to do things like passing arguments, and they come with
many problems[1]
* There are not many SOCKS server implementations and many PTs end up needing
to implement their own (goptlib and proteus have done it)
We have being wondering if making a change from SOCKS to a HTTP based protocol
is the solution we need[2].
For mobile phones we will need PTs to be implemented as libraries, and some
projects like IPtProxy[3] have being created to work around that limitation. It
looks like we will want anyway a future were arti has PT plugins and PTs are
integrated in arti. We might want to extend that idea to implement all the
connect assist workflow there, so clients that use arti don't need to implement
all the logic themselves again and again.
With that future of PTs as plugins. Do we still need a inter-process PT
protocol? and need to make a new version of the pt-spec[0]? Or should we ditch
it and start working on what will that plugin API/ABI look like and how PTs will
be integrated there?
It looks like this conversation doesn't fit well in our irc weekly meetings,
let's have a voice meeting to discuss this. I have created a poll to see what
date next week will work better for all of us:
https://www.systemli.org/poll/#/poll/YRRF8K19Bq/participation?encryptionKey…
[0] https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/pt-spec.txt
[1] https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/104
[2] https://gitlab.torproject.org/tpo/anti-censorship/team/-/issues/130
[3] https://github.com/tladesignz/IPtProxy
--
meskio | https://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
My contact info: https://meskio.net/crypto.txt
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.
I was alerted by trinity-1686a on irc that Snowflake standalone proxy
operators were reporting on #tor-relays about increased OOM errors from
increased load as of 2023-08-28.
After looking at the Snowfake broker metrics[0], there's a huge jump in
client polls (seen by summing the client-denied-count and
client-snowflake-match-count).
I've attached a graph of the collected prometheus metrics that shows
this spike happening at exactly 17:40 UTC on 2023-08-27. It looks like
way too sharp an increase to me to be a censorship event, perhaps it is
a DoS?
It's still too early to see the bridge metrics from the metrics page,
but we should start to see the effects there tomorrow.
[0] https://metrics.torproject.org/collector.html#snowflake-stats