Making some graphs for the August 2024 Snowflake bridge report, there are a couple of weird things to comment on.
On August 22–23, there was a drop in users across both bridges that later almost evened out. (See first attachment.) It affected both bridges.
From August 16–31, the ?? country broke into the top 5 countries by user
count, on the snowflake-01 bridge only. (See third attachment.) Because the country code comes from the proxy, which reports the remote IP address of the client, a country code of ?? means that either the proxy did not report an IP address (no client_ip WebSocket URL parameter), or the IP address was not found in a geolocation lookup. Here's a past case where a bug in the WebExtension was resulting in ?? countries: https://bugs.torproject.org/tpo/anti-censorship/pluggable-transports/snowfla...
The snowflake server logs daily the fraction N/D of connections that had client_ip set. In the case before, N decreased while D remained constant. In this case, it looks like N is remaining constant while D is greatly increasing. See in the listing below how N/D suddenly drops from 99% to 41% on 2024-08-13. In the last couple of days, ⅔ to ¾ of connections have been missing the client_ip parameter. The client_ip-less connections must not represent very many user-hours, otherwise they would completely dominate the user graphs.
Did something happen on 2024-08-13? Is this a new source of proxies with an implementation bug? Some kind of attack? (Against the bridge?)
$ grep 'connections had client_ip$' /var/log/snowflake-server/snowflake-server.log | perl -ane '$F[7] =~ m#^(.*)/(.*)$#; printf("%s %s %8d/%-8d %6.2f%%\n", $F[0], $F[1], $1, $2, 100 * ($2 == 0 ? 0.0 : $1/$2));' 2024/08/06 08:57:22 463234/464597 99.71% 2024/08/07 08:57:22 462398/463581 99.74% 2024/08/08 08:57:22 453003/454165 99.74% 2024/08/09 08:57:22 450344/451707 99.70% 2024/08/10 08:57:22 450057/452228 99.52% 2024/08/11 08:57:22 449782/452296 99.44% 2024/08/12 08:57:22 452387/475006 95.24% 2024/08/13 08:57:22 450993/1089806 41.38% 2024/08/14 08:57:22 455701/828402 55.01% 2024/08/15 08:57:22 449159/886577 50.66% 2024/08/16 08:57:22 430698/1061690 40.57% 2024/08/17 08:57:22 426891/1240861 34.40% 2024/08/18 08:57:22 428010/1139078 37.58% 2024/08/19 08:57:22 425667/1363289 31.22% 2024/08/20 08:57:22 427042/1181316 36.15% 2024/08/21 08:57:22 422347/1407363 30.01% 2024/08/22 08:57:22 418760/1437024 29.14% 2024/08/23 08:57:22 271142/1217897 22.26% 2024/08/24 08:57:22 353388/1291375 27.37% 2024/08/25 08:57:22 366323/1117172 32.79% 2024/08/26 08:57:22 382452/1107561 34.53% 2024/08/27 08:57:22 416759/1451520 28.71% 2024/08/28 08:57:22 413329/1445703 28.59% 2024/08/29 08:57:22 403754/1490629 27.09% 2024/08/30 08:57:22 388147/1490993 26.03% 2024/08/31 08:57:22 395311/1377168 28.70% 2024/09/01 08:57:22 389361/1344726 28.95% 2024/09/02 08:57:22 377702/1169902 32.28% 2024/09/03 08:57:22 369956/1272026 29.08% 2024/09/04 08:57:22 368266/1330355 27.68% 2024/09/05 08:57:22 371429/1260571 29.47%
On Thu, Sep 05, 2024 at 01:05:01PM -0600, David Fifield wrote:
From August 16–31, the ?? country broke into the top 5 countries by user count, on the snowflake-01 bridge only. (See third attachment.) Because the country code comes from the proxy, which reports the remote IP address of the client, a country code of ?? means that either the proxy did not report an IP address (no client_ip WebSocket URL parameter), or the IP address was not found in a geolocation lookup.
Another possibility is that there's some kind of custom client that's connecting to the bridge directly over its WebSocket interface (not using a snowflake proxy), and it's not sending client_ip.
On Thu, Sep 5, 2024, at 3:05 PM, David Fifield wrote:
Did something happen on 2024-08-13? Is this a new source of proxies with an implementation bug? Some kind of attack? (Against the bridge?)
We rolled out a new release of Orbot around then. Are you saying this is an issue with Snowflake proxies, not clients?
I can review more deeply tomorrow.
On Thu, Sep 05, 2024 at 10:15:16PM -0400, Nathan of Guardian wrote:
On Thu, Sep 5, 2024, at 3:05 PM, David Fifield wrote:
Did something happen on 2024-08-13? Is this a new source of proxies with an implementation bug? Some kind of attack? (Against the bridge?)
We rolled out a new release of Orbot around then. Are you saying this is an issue with Snowflake proxies, not clients?
I can review more deeply tomorrow.
It is the proxy that reports the client IP address to the bridge, not the client. The "client IP address" is just the IP address of the proxy's WebRTC peer. The proxy attaches that information in a ?client_ip=X.X.X.X URL query parameter in its initial WebSocket HTTP request to the bridge.
I think it's unlikely that a change in Orbot would manifest this way. If that were the case, in the ratio N/D, the total D would stay the same, but the address-reporting fraction N would go down. (The same old bunch of proxies, except a few are malfunctioning and not reporting the address.) Whereas in this case, we have N staying the same, with D going up. Facially, that means that there are more proxies now than before, and the "new" ones are not reporting addresses.
One possibility, I suppose, is that an Orbot upgrade caused a subset of existing proxies not only to stop reporting client_ip, but also to fail connections, period. (Or fail probabilistically, perhaps.) That might cause the total number of apparent connections (D) to increase, while clients retry until they find a working proxy, keeping the number of client_ip connections (N) the same. I am not sure exactly at what point a "connection" is counted inside snowflake-server for purposes of the N/D ratio.
anti-censorship-team@lists.torproject.org
-
David Fifield -
Nathan of Guardian