Thanks, that's a good lead.

The effect of this apparent bug is that the lower bound of per-country per-transport user count intervals goes to zero. See the first attached graph. The snowflake-01 bridge upgraded to 0.4.8 on 2023-10-03; snowflake-02 upgraded on 2023-09-24. Wheres formerly, the low–high intervals were so narrow as to be indistinguishable from a line, now they extend all the way down to the x-axis.

https://metrics.torproject.org/reproducible-metrics.html#bridge-users The formula for the lower bound is: low = max(0, country_reqs + transport_reqs − total_reqs) On a bridge that hides its ORPort and runs just one transport, we have transport_reqs ≈ total_reqs, and so the above just becomes low = max(0, country_reqs) But with the apparent metrics bug in 0.4.8.6, total_reqs is twice as large as it should be (i.e., 2 × transport_reqs ≈ total_reqs), which means the formula becomes low = max(0, country_reqs − transport_reqs) which, since country_reqs < transport_reqs, always becomes zero.

The upper bound of the interval is unaffected; its formula is high = min(country_reqs, transport_reqs)

The other side effect is that directory requests that ought to be attributed entirely to a pluggable transport are instead being ascribed 50/50 to that transport and the plain OR protocol. "We approximate [directory requests by transport] by multiplying the total number of requests with the fraction of unique IP addresses by transport." See, in the second attached graph, how estimated pluggable transport users have declined and OR protocol users have increased, in an almost inverse relationship, roughly in step with the 0.4.7→0.4.8 upgrade.

https://metrics.torproject.org/userstats-bridge-transport.html?start=2023-07... https://metrics.torproject.org/versions.html?start=2023-07-11&end=2023-1...

Since the release of 0.4.8, we are systematically undercounting pluggable transport users and overcounting vanilla bridge users.

On Tue, Oct 03, 2023 at 09:57:53PM +0200, trinity pointard wrote:

Hi,

I'm highly suspicious of commit 3e18507dc75afcf0c6560e966c9f18942406b0c8 which adds a call to geoip_note_client_seen with a NULL (==no PT) in src/core/or/channeltls.c

It's fine for DoS defense, but it's likely what's messing up stats.

Regards, trinity-1686a

On Tue, 3 Oct 2023 at 21:12, David Fifield david@bamsoftware.com wrote:

...

The Snowflake bridges do not expose their plain ORPort (they have `ORPort 127.0.0.1:auto` in torrc), and consequently they have always reported ≈0 <OR> IP addresses in the bridge-ip-transports line of bridge-extra-info descriptors, with virtually all connecting IP addresses being instead attributed to the snowflake transport. (The <OR> count is low but nonzero, which I have rationalized as tor making some small number of internal connections, or something.)

I upgraded one of the two Snowflake bridges from 0.4.7.13 to 0.4.8.6 on 2023-09-24. Since then, the number of <OR> IP addresses has been roughly equal to the number of snowflake IP addresses. The ORPort is still not exposed; these are not external vanilla bridge users. Did something change between these versions that might cause PT connections to be double-counted, once for the transport and once for <OR>?

Here are excerpted bridge-extra-info descriptors from before and after the version upgrade. Note the bridge-ip-transports lines.

@type bridge-extra-info 1.3
extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0
master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks
published 2023-09-19 10:46:08
transport snowflake
bridge-ip-versions v4=13528,v6=1384
bridge-ip-transports <OR>=8,snowflake=14912

@type bridge-extra-info 1.3
extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0
master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks
published 2023-09-29 17:33:20
transport snowflake
bridge-ip-versions v4=2880,v6=336
bridge-ip-transports <OR>=1632,snowflake=1592

Below is a time series for a whole month. The thing to notice is not the overall decrease in users on 2023-09-20; that's due to https://forum.torproject.org/t/problems-with-snowflake-since-2023-09-20-brok.... Rather, the thing to notice is that before 2023-09-24, <OR> ≈ 0; and after, <OR> ≈ snowflake.

date <OR> snowflake 2023-08-30 4 45240 ← running 0.4.7.13 here 2023-08-31 21 193412 2023-09-01 30 195647 2023-09-02 40 200405 2023-09-03 40 201730 2023-09-04 42 195097 2023-09-05 35 190842 2023-09-06 30 189694 2023-09-07 26 191524 2023-09-08 35 195833 2023-09-09 32 196244 2023-09-10 27 146972 2023-09-11 6 55491 2023-09-12 18 182519 2023-09-13 17 173536 2023-09-14 21 161396 2023-09-15 21 166379 2023-09-16 16 174350 2023-09-17 17 165063 2023-09-18 23 164734 2023-09-19 32 164075 2023-09-20 23 121623 2023-09-21 2 25268 2023-09-22 0 4314 2023-09-23 0 746 2023-09-24 239 606 ← upgrade to 0.4.8.6 on this day 2023-09-25 1919 1900 2023-09-26 3533 3470 2023-09-27 11919 11687 2023-09-28 20242 19890 2023-09-29 26583 26100 2023-09-30 25929 25400