I got a notification from AWS that our SQS rendezvous service exceeded
the free-tier usage this month with over 1,000,000 SQS API requests.
This is in some sense exciting news, because it shows that the
rendezvous channel is effective and getting some use.
It does, however mean that we will have to start paying for the service.
The current billing period falls on month boundaries, from January 1st
to January 31st. The budget action fired on January 9th, which is pretty
early in the month. Looking at our broker metrics[0,1], there were
approximately 38,608 client polls using SQS. That's approximately 2.5
requests per poll, which is about what I'd expect.
It's reassuring to know that the budget actions work. I'm going to set
them a little higher, to something I can reasonably afford. I don't yet
know how the cost will scale with the number of polls, and how that will
compare with the cost of domain fronted requests.
[0] https://snowflake-broker.torproject.net/metrics
[1] https://metrics.torproject.org/collector.html#type-snowflake-stats
I got an email on 2024-12-05 saying that that Azure CDN from
Edgio/Verizon is going to shut down sooner than expected. The shutdown
date was supposed to be 2025-11-04, now it is 2024-01-15 (about one
month from now).
This affects (at least) snowflake-broker.azureedge.net, which was first
set up and is still on Azure CDN with Edgio. I think that
snowflake-broker.azureedge.net has not been used by this team since 2021
and https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/merge_re….
But it is still getting some use from somewhere, as evidenced by the
nonzero monthly bills:
https://gitlab.torproject.org/tpo/anti-censorship/team/-/wikis/Snowflake-co…
Whatever is out there that is still using snowflake-broker.azureedge.net
will likely stop working after 2025-01-15.
> Date: 5 Dec 2024 17:03:50 +0000
> From: Microsoft Azure <azure-noreply(a)microsoft.com>
> Subject: Urgent action required: Migrate workloads on Azure CDN from Edgio (formerly Verizon) before 15 January 2025
>
> Migrate to Azure Front Door or other CDN services as soon as possible
> to avoid service disruptions.
>
> Migrate workloads on Azure CDN from Edgio (formerly Verizon) before 15
> January 2025.
>
> You're receiving this notice because you're currently using Azure CDN
> Standard/Premium from Edgio.
>
> All Azure CDN from Edgio customers must migrate their workloads to
> Azure Front Door or other CDN services before 15 January 2025 as Edgio
> has advised their platform is currently scheduled to be shut down by
> that date. Migrate as soon as possible to avoid an imminent service
> shut-down.
>
> On 31 October 2024, we sent you a notice advising that Azure CDN
> Standard/Premium from Edgio (formerly Verizon) will be retired on
> 4 November 2025, and that customers of this service must migrate their
> workload(s) to a comparable service before this date to avoid service
> interruptions. In that email we also advised given that Edgio filed
> for Chapter 11 Bankruptcy on 9 September 2024, Microsoft cannot
> guarantee that Edgio will continue to support this service through
> 4 November 2025 as previously stated.
Online references about Azure CDN with Edgio retirement:
https://learn.microsoft.com/en-us/azure/cdn/edgio-retirement-faqhttps://azure.microsoft.com/updates?id=467688
I did try setting up a Front Door CDN profile (which is the recommended
replacement for Edgio), following the migration instructions:
https://learn.microsoft.com/en-us/azure/frontdoor/migrate-cdn-to-front-door
I got the hostname:
snowflake-broker-hadmaqbnc4dmcffs.z03.azurefd.net
It works as an alias for the broker:
curl -i https://snowflake-broker-hadmaqbnc4dmcffs.z03.azurefd.net/debug
However, it apparently does not work with domain fronting, so is likely
not actually useful. Front Door not working with domain fronting is
consistent with prior announcements by Azure:
https://github.com/net4people/bbs/issues/67
Hi all,
Devices running versions of Android older than 7.1.1 can't verify
certificates signed with Let's Encrypt's ISRG Root X1 root certificate,
so they can't connect to domain fronts that use such certificates. [1]
These devices (released in 2016 or earlier) still make up nearly 5% of
active Android devices. [2]
There was a workaround in place at one point -- cross-signing Let's
Encrypt certificates with a different, expired root certificate and
relying on Android not to check the expiry date -- but I believe the
cross-signature expired in early 2024. [3]
With the loss of Fastly and Azure, the only remaining fronts for Meek
and Snowflake in the default config served by Moat will be cdn77.com and
phpmyadmin.net, both of which use Let's Encrypt certificates that are
signed with ISRG Root X1 and don't appear to be cross-signed.
It looks like there's some work in progress to address this issue in
Lyrebird by adding the relevant certificates, so hopefully Meek and
Snowflake will work in a future Lyrebird release. But what about the
initial connection to Moat?
Orbot has moved from Fastly to CDN77 for its Moat front [4]. Are there
any plans underway to make another front available, or should we move to
CDN77 and plan for Moat being unavailable on older Android devices?
Thanks,
Michael
[1] https://letsencrypt.org/2020/11/06/own-two-feet/
[2] https://apilevels.com/
[3]
https://arstechnica.com/gadgets/2020/12/lets-encrypt-comes-up-with-workarou…
[4] https://github.com/guardianproject/orbot/pull/1191/files