[tor-dev] PT bridge reporting significant numbers of <OR> IP addresses after upgrade to 0.4.8.6

Tue Oct 10 01:51:58 UTC 2023

Thanks, that's a good lead.

The effect of this apparent bug is that the lower bound of per-country
per-transport user count intervals goes to zero. See the first attached
graph. The snowflake-01 bridge upgraded to 0.4.8 on 2023-10-03;
snowflake-02 upgraded on 2023-09-24. Wheres formerly, the low–high
intervals were so narrow as to be indistinguishable from a line, now
they extend all the way down to the x-axis.

https://metrics.torproject.org/reproducible-metrics.html#bridge-users
The formula for the lower bound is:
	low = max(0, country_reqs + transport_reqs − total_reqs)
On a bridge that hides its ORPort and runs just one transport, we have
transport_reqs ≈ total_reqs, and so the above just becomes
	low = max(0, country_reqs)
But with the apparent metrics bug in 0.4.8.6, total_reqs is twice as
large as it should be (i.e., 2 × transport_reqs ≈ total_reqs), which
means the formula becomes
	low = max(0, country_reqs − transport_reqs)
which, since country_reqs < transport_reqs, always becomes zero.

The upper bound of the interval is unaffected; its formula is
	high = min(country_reqs, transport_reqs)

The other side effect is that directory requests that ought to be
attributed entirely to a pluggable transport are instead being ascribed
50/50 to that transport and the plain OR protocol. "We approximate
[directory requests by transport] by multiplying the total number of
requests with the fraction of unique IP addresses by transport." See, in
the second attached graph, how estimated pluggable transport users have
declined and OR protocol users have increased, in an almost inverse
relationship, roughly in step with the 0.4.7→0.4.8 upgrade.

https://metrics.torproject.org/userstats-bridge-transport.html?start=2023-07-11&end=2023-10-09&transport=%21%3COR%3E&transport=%3COR%3E
https://metrics.torproject.org/versions.html?start=2023-07-11&end=2023-10-09

Since the release of 0.4.8, we are systematically undercounting
pluggable transport users and overcounting vanilla bridge users.

On Tue, Oct 03, 2023 at 09:57:53PM +0200, trinity pointard wrote:
> Hi,
> 
> I'm highly suspicious of commit
> 3e18507dc75afcf0c6560e966c9f18942406b0c8 which adds a call to
> geoip_note_client_seen with a NULL (==no PT) in
> src/core/or/channeltls.c
> 
> It's fine for DoS defense, but it's likely what's messing up stats.
> 
> Regards,
> trinity-1686a
> 
> On Tue, 3 Oct 2023 at 21:12, David Fifield <david at bamsoftware.com> wrote:
> >
> > The Snowflake bridges do not expose their plain ORPort (they have
> > `ORPort 127.0.0.1:auto` in torrc), and consequently they have always
> > reported ≈0 <OR> IP addresses in the bridge-ip-transports line of
> > bridge-extra-info descriptors, with virtually all connecting IP
> > addresses being instead attributed to the snowflake transport. (The <OR>
> > count is low but nonzero, which I have rationalized as tor making some
> > small number of internal connections, or something.)
> >
> > I upgraded one of the two Snowflake bridges from 0.4.7.13 to 0.4.8.6 on
> > 2023-09-24. Since then, the number of <OR> IP addresses has been roughly
> > equal to the number of snowflake IP addresses. The ORPort is still not
> > exposed; these are not external vanilla bridge users. Did something
> > change between these versions that might cause PT connections to be
> > double-counted, once for the transport and once for <OR>?
> >
> > Here are excerpted bridge-extra-info descriptors from before and after
> > the version upgrade. Note the bridge-ip-transports lines.
> >
> > ```
> > @type bridge-extra-info 1.3
> > extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0
> > master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks
> > published 2023-09-19 10:46:08
> > transport snowflake
> > bridge-ip-versions v4=13528,v6=1384
> > bridge-ip-transports <OR>=8,snowflake=14912
> > ```
> >
> > ```
> > @type bridge-extra-info 1.3
> > extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0
> > master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks
> > published 2023-09-29 17:33:20
> > transport snowflake
> > bridge-ip-versions v4=2880,v6=336
> > bridge-ip-transports <OR>=1632,snowflake=1592
> > ```
> >
> > Below is a time series for a whole month.
> > The thing to notice is not the overall decrease in users on 2023-09-20;
> > that's due to
> > https://forum.torproject.org/t/problems-with-snowflake-since-2023-09-20-broker-failure-unexpected-error-no-answer/9346.
> > Rather, the thing to notice is that before 2023-09-24, <OR> ≈ 0; and
> > after, <OR> ≈ snowflake.
> >
> > date        <OR> snowflake
> > 2023-08-30     4     45240  ← running 0.4.7.13 here
> > 2023-08-31    21    193412
> > 2023-09-01    30    195647
> > 2023-09-02    40    200405
> > 2023-09-03    40    201730
> > 2023-09-04    42    195097
> > 2023-09-05    35    190842
> > 2023-09-06    30    189694
> > 2023-09-07    26    191524
> > 2023-09-08    35    195833
> > 2023-09-09    32    196244
> > 2023-09-10    27    146972
> > 2023-09-11     6     55491
> > 2023-09-12    18    182519
> > 2023-09-13    17    173536
> > 2023-09-14    21    161396
> > 2023-09-15    21    166379
> > 2023-09-16    16    174350
> > 2023-09-17    17    165063
> > 2023-09-18    23    164734
> > 2023-09-19    32    164075
> > 2023-09-20    23    121623
> > 2023-09-21     2     25268
> > 2023-09-22     0      4314
> > 2023-09-23     0       746
> > 2023-09-24   239       606  ← upgrade to 0.4.8.6 on this day
> > 2023-09-25  1919      1900
> > 2023-09-26  3533      3470
> > 2023-09-27 11919     11687
> > 2023-09-28 20242     19890
> > 2023-09-29 26583     26100
> > 2023-09-30 25929     25400
-------------- next part --------------
A non-text attachment was scrubbed...
Name: snowflake-top-countries-20231010.png
Type: image/png
Size: 174414 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20231009/ac3c1ee5/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: userstats-bridge-transport-2023-07-11-2023-10-09-! OR - OR .png
Type: image/png
Size: 38977 bytes
Desc: not available
URL: <http://lists.torproject.org/pipermail/tor-dev/attachments/20231009/ac3c1ee5/attachment-0003.png>