Hi all,
Devices running versions of Android older than 7.1.1 can't verify certificates signed with Let's Encrypt's ISRG Root X1 root certificate, so they can't connect to domain fronts that use such certificates. [1] These devices (released in 2016 or earlier) still make up nearly 5% of active Android devices. [2]
There was a workaround in place at one point -- cross-signing Let's Encrypt certificates with a different, expired root certificate and relying on Android not to check the expiry date -- but I believe the cross-signature expired in early 2024. [3]
With the loss of Fastly and Azure, the only remaining fronts for Meek and Snowflake in the default config served by Moat will be cdn77.com and phpmyadmin.net, both of which use Let's Encrypt certificates that are signed with ISRG Root X1 and don't appear to be cross-signed.
It looks like there's some work in progress to address this issue in Lyrebird by adding the relevant certificates, so hopefully Meek and Snowflake will work in a future Lyrebird release. But what about the initial connection to Moat?
Orbot has moved from Fastly to CDN77 for its Moat front [4]. Are there any plans underway to make another front available, or should we move to CDN77 and plan for Moat being unavailable on older Android devices?
Thanks, Michael
[1] https://letsencrypt.org/2020/11/06/own-two-feet/ [2] https://apilevels.com/ [3] https://arstechnica.com/gadgets/2020/12/lets-encrypt-comes-up-with-workaroun... [4] https://github.com/guardianproject/orbot/pull/1191/files
Am 01.01.25 um 19:29 schrieb Michael Rogers via anti-censorship-team:
It looks like there's some work in progress to address this issue in Lyrebird by adding the relevant certificates, so hopefully Meek and Snowflake will work in a future Lyrebird release. But what about the initial connection to Moat?
This also uses Lyrebird's meek-lite transport. So, if Lyrebird is capable of handling the certificates, the connections to RdSys/Moat/Circumvention API will als work.
Cheers,
~tla
On 02/01/2025 17:02, Benjamin Erhart via anti-censorship-team wrote:
Am 01.01.25 um 19:29 schrieb Michael Rogers via anti-censorship-team:
It looks like there's some work in progress to address this issue in Lyrebird by adding the relevant certificates, so hopefully Meek and Snowflake will work in a future Lyrebird release. But what about the initial connection to Moat?
This also uses Lyrebird's meek-lite transport. So, if Lyrebird is capable of handling the certificates, the connections to RdSys/Moat/ Circumvention API will als work.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
Cheers, Michael
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/
On 14/01/2025 18:47, cohosh--- via anti-censorship-team wrote:
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/
Hi cohosh,
Thanks for following up on this.
I made a mistake about OnionShare's use of Moat. It *does* use obfs4proxy for the meek connection (and will use Lyrebird in the next release), but then the HTTPS request through the meek tunnel fails on Android < 7.1.1 because bridges.torproject.org also uses a Let's Encrypt certificate. To fix that, I need to investigate whether we can get the libraries we're using for HTTPS (OkHttp and Conscrypt) to trust a pinned certificate.
Another thing I need to investigate is a difference in failure modes between Lyrebird 0.5.0 and obfs4proxy 0.0.14 on Android 7.0.
On Android 7.1.1, both versions of the transport are able to make meek connections via CDN77 and Azure and then connect to bridges.torproject.org through the tunnel.
On Android 7.0, both versions of the transport are able to make meek connections to Azure. (They then fail to connect to bridges.torproject.org through the tunnel as I mentioned above, but that's a separate issue.)
But on Android 7.0, obfs4proxy 0.0.14 can make a meek connection to CDN77 before failing to connect through the tunnel, whereas Lyrebird 0.5.0 fails with a timeout error instead.
When I first read your message I thought, "Aha, the out-of-date certificate pin is causing the meek connection to CDN77 to fail, that's why we see a timeout with Lyrebird but not with obfs4proxy." But now I have a doubt. If there's an out-of-date pin then shouldn't Lyrebird + CDN77 also fail on Android 7.1.1? Or is the pin only consulted if the platform's certificate store fails to validate the certificate (which would be the case for Android 7.0 + CDN77 + Lyrebird, but not for any other combination)?
Cheers, Michael
On 2025-01-15 12:47, Michael Rogers via anti-censorship-team wrote:
On 14/01/2025 18:47, cohosh--- via anti-censorship-team wrote:
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/
Hi cohosh,
Thanks for following up on this.
I made a mistake about OnionShare's use of Moat. It *does* use obfs4proxy for the meek connection (and will use Lyrebird in the next release), but then the HTTPS request through the meek tunnel fails on Android < 7.1.1 because bridges.torproject.org also uses a Let's Encrypt certificate. To fix that, I need to investigate whether we can get the libraries we're using for HTTPS (OkHttp and Conscrypt) to trust a pinned certificate.
Another thing I need to investigate is a difference in failure modes between Lyrebird 0.5.0 and obfs4proxy 0.0.14 on Android 7.0.
On Android 7.1.1, both versions of the transport are able to make meek connections via CDN77 and Azure and then connect to bridges.torproject.org through the tunnel.
On Android 7.0, both versions of the transport are able to make meek connections to Azure. (They then fail to connect to bridges.torproject.org through the tunnel as I mentioned above, but that's a separate issue.)
But on Android 7.0, obfs4proxy 0.0.14 can make a meek connection to CDN77 before failing to connect through the tunnel, whereas Lyrebird 0.5.0 fails with a timeout error instead.
When I first read your message I thought, "Aha, the out-of-date certificate pin is causing the meek connection to CDN77 to fail, that's why we see a timeout with Lyrebird but not with obfs4proxy." But now I have a doubt. If there's an out-of-date pin then shouldn't Lyrebird + CDN77 also fail on Android 7.1.1? Or is the pin only consulted if the platform's certificate store fails to validate the certificate (which would be the case for Android 7.0 + CDN77 + Lyrebird, but not for any other combination)?
Well, we never shipped the pin for Lyrebird. That was a trial patch I discarded because it didn't solve the problem we were seeing. Your last guess is correct. What we're doing in Snowflake isn't restrictively pinning certs. We're just appending the Let's Encrypt root cert to the root CA pool[0]. So if a platform, like Android 7.1.1 and above, already has a working cert in it's store, it doesn't need the pinned cert and just ignores it.
I am curious about the CDN77 meek connection working while the internal tunnelled connection fails on < Android 7.1.1. This sounds like exactly the issue we've been trying to debug with Tor Browser[1], when domain fronting stopped working with Fastly, which wasn't using Let's Encrypt. It didn't occur to me to think about certificate validation in the tunnelled connection... I guess I assumed that the application would do the right thing with the expired cross-signed root cert. But maybe the libraries used by Tor Browser and OnionShare are similar to the Go library in this way, and actually are enforcing the expiration date.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb...
anti-censorship-team@lists.torproject.org
-
Benjamin Erhart -
Cecylia Bocovich -
cohosh＠torproject.org
-
Michael Rogers