network-health

network-health@lists.torproject.org

July 2026

2 participants
2 discussions

The format of snowflake-stats descriptors and collector-rs
by David Fifield 08 Jul '26

08 Jul '26

For the past 3 months, collector-rs has been making snowflake-stats tarballs in an incorrect format. The recently committed fix for this problem implies that the incorrect format was the result of the Snowflake broker changing the format of its metrics. But this is not the case: Snowflake broker metrics have always (since 2019) contained multiple days of metrics. This was by design, and Java Collector was always able to parse that format. Since the recent fix, collector-rs treats snowflake-stats as a special case ("skip_seed_for_archive"), but Java Collector didn't treat snowflake-stats as exceptional—it parsed snowflake-stats in the same way as other types that have multiple days of data in one file, like bridgedb-metrics and bridgestrap-stats, using a method called parseOneOrMoreDescriptors. In summary, the problem was with collector-rs not parsing the metrics files in the same way as Java Collector did, not because of any change in the output of the Snowflake broker. snowflake-stats is not really a special case, but the collector-rs code now treats it as a special case, which makes me wonder if it could be handled in a more uniform way. This is the issue for collector-rs serving an incorrect format for snowflake-stats tarballs: "Snowflake stats format is broken for April" https://gitlab.torproject.org/tpo/network-health/metrics/collector-rs/-/wor… This is the merge request that closes #48 along with many other things: "Rework archive/publish/sync pipeline; add bridge import; vote & dirauth fixes" https://gitlab.torproject.org/tpo/network-health/metrics/collector-rs/-/mer… I understand that the fix has two parts: 1. Changing the collector-rs to ignore all but the last "snowflake-stats-end": https://gitlab.torproject.org/tpo/network-health/metrics/collector-rs/-/mer… 2. Running a one-time fix_snowflake script that rewrites past downloaded files so that they contain only the last "snowflake-stats-end": https://gitlab.torproject.org/juga/collector-fixes/-/commit/7b0aa3170313cb4… I'll quote from fix_snowflake.rs, because I think it reveals the misunderstanding on the collector-rs side: //! `fix-snowflake` - trim bloated snowflake-stats files down to a single //! reporting period, both in `out/recent/snowflakes/` and in the monthly //! `out/archive/snowflakes/snowflake-stats-*.tar.xz` tarballs. //! //! ## Background //! //! The snowflake broker's `/metrics` endpoint is meant to return a single //! `snowflake-stats-end <date> <time> (86400 s)` reporting block per fetch. //! Starting around collector-rs's deployment in 2026, the broker began //! returning the **entire cumulative history** since 2024-09-19 on every //! fetch instead: every previous day's block concatenated, with the new //! day's block appended at the end. Every file collector-rs wrote during //! that window inherited that bloat verbatim - both the daily files in //! `out/recent/snowflakes/` and the entries written into the monthly //! archive tarballs. The format emitted by the broker /metrics endpoint is not "bloat" and it is not "incorrect". Nor did the format change at any point. Here's a Wayback capture that shows it worked like that in 2020: https://web.archive.org/web/20200619131042/https://snowflake-broker.torproj… The format was intentionally designed this way. Here is part of a conversation with Karsten when the metrics feature was originally designed: https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/work_i… To answer your earlier question above: having just the last 24 hours of data might be problematic if the metrics host goes down for, say, a weekend. Ideally, there would be at least a week of statistics available. Or maybe just accumulate new statistics forever, given the tiny amount of statistics per day. This is the way it is intended to work and the way it worked before. collector-rs should not treat the format as anomalous or unexpected. The old metrics/library had a test case with 2 descriptors in it, since 2019: https://gitlab.torproject.org/tpo/network-health/metrics/library/-/blob/2b6… This is how the old metrics/library handled snowflake-stats descriptors: https://gitlab.torproject.org/tpo/network-health/metrics/library/-/blob/2b6… } else if (firstLines.startsWith("@type snowflake-stats 1.") || firstLines.startsWith(Key.SNOWFLAKE_STATS_END.keyword + SP) || firstLines.contains(NL + Key.SNOWFLAKE_STATS_END.keyword + SP)) { return this.parseOneOrMoreDescriptors(rawDescriptorBytes, sourceFile, Key.SNOWFLAKE_STATS_END, SnowflakeStatsImpl.class); } It called parseOneOrMoreDescriptors with "snowflake-stats-end" as a delimiter. Then it iterated over parsed descriptor and handled each one individually: https://gitlab.torproject.org/tpo/network-health/metrics/collector/-/blob/0… DescriptorParser descriptorParser = DescriptorSourceFactory.createDescriptorParser(); SortedSet<LocalDateTime> snowflakeStatsEnds = new TreeSet<>(); this.outputPathName = config.getPath(Key.OutputPath).toString(); for (Descriptor descriptor : descriptorParser.parseDescriptors( downloadedBytes, null, null)) { ... The important thing to note is that Snowflake is not at all a special case; the detectTypeAndParseDescriptors method calls parseOneOrMoreDescriptors for many different descriptor types (with different delimiters): https://gitlab.torproject.org/tpo/network-health/metrics/library/-/blob/2b6…

2 2

Mid-month descriptors from collector-rs?
by David Fifield 05 Jul '26

05 Jul '26

It used to be that the "archive" directory of collector.torproject.org would serve unchanging descriptor tarballs from past months, along with a more or less continuously updated tarball for the current month. But now it seems that there is no tarball for the current month. See here: https://collector.torproject.org/archive/bridge-descriptors/extra-infos/ https://web.archive.org/web/20260504132617/https://collector.torproject.org… The date is currently 2026-05-04, but the latest tarball is extra-infos-2026-04.tar.xz 2026-05-02 03:42 218M Is there a way to get, for example, descriptors from May 3, without waiting until the end of the month? There is the "recent" directory which might work, but it only goes back a few days. At this moment, "recent" covers from 2026-04-30 to 2026-05-04, but if I were to check on 2026-05-20, say, I might not be able to get descriptors from 2026-05-10. https://collector.torproject.org/recent/bridge-descriptors/extra-infos/ https://web.archive.org/web/20260504132701/https://collector.torproject.org… The reason I'm asking: When I make monthly graphs of snowflake bridges, I usually wait a few days into the next month and download the next month's descriptor to include it with the data. The reason is that the tarball for May, for example, contains descriptors that cover the end of April. Without including the beginning of the next month's data, I found, the last 1–2 days of the previous month's data can be incomplete. For example, I just used https://gitlab.torproject.org/dcf/snowflake-graphs to compute statistics using tarballs up to extra-infos-2026-04.tar.xz. This is showing just one bridge (which consists of 12 tor instances). Notice the last few days of April 2026, the "coverage" drops from from 12.0/12.0 to 9.95/12.0: date, fingerprint, transport, users, num_instances,coverage 2026-04-24,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14592.87,12.00,12.00 2026-04-25,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14615.31,12.00,12.00 2026-04-26,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14992.47,12.00,12.00 2026-04-27,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,15056.68,12.00,12.00 2026-04-28,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,15250.09,12.00,12.00 2026-04-29,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,15034.07,12.00,11.92 2026-04-30,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,12877.20,12.00,9.95 If I regenerate the statistics, manually including also the "recent" descriptors, the coverage gets "filled in" (and the users count correspondingly goes up): date, fingerprint, transport, users, num_instances,coverage 2026-04-24,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14592.87,12.00,12.00 2026-04-25,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14615.31,12.00,12.00 2026-04-26,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14992.47,12.00,12.00 2026-04-27,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,15056.68,12.00,12.00 2026-04-28,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,15250.09,12.00,12.00 2026-04-29,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,15099.58,12.00,12.00 2026-04-30,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14834.64,12.00,12.00 2026-05-01,5481936581E23D2D178105D44DB6915AB06BFB7F,snowflake,14781.53,12.00,12.00

2 3