Hi all,
Devices running versions of Android older than 7.1.1 can't verify certificates signed with Let's Encrypt's ISRG Root X1 root certificate, so they can't connect to domain fronts that use such certificates. [1] These devices (released in 2016 or earlier) still make up nearly 5% of active Android devices. [2]
There was a workaround in place at one point -- cross-signing Let's Encrypt certificates with a different, expired root certificate and relying on Android not to check the expiry date -- but I believe the cross-signature expired in early 2024. [3]
With the loss of Fastly and Azure, the only remaining fronts for Meek and Snowflake in the default config served by Moat will be cdn77.com and phpmyadmin.net, both of which use Let's Encrypt certificates that are signed with ISRG Root X1 and don't appear to be cross-signed.
It looks like there's some work in progress to address this issue in Lyrebird by adding the relevant certificates, so hopefully Meek and Snowflake will work in a future Lyrebird release. But what about the initial connection to Moat?
Orbot has moved from Fastly to CDN77 for its Moat front [4]. Are there any plans underway to make another front available, or should we move to CDN77 and plan for Moat being unavailable on older Android devices?
Thanks, Michael
[1] https://letsencrypt.org/2020/11/06/own-two-feet/ [2] https://apilevels.com/ [3] https://arstechnica.com/gadgets/2020/12/lets-encrypt-comes-up-with-workaroun... [4] https://github.com/guardianproject/orbot/pull/1191/files
Am 01.01.25 um 19:29 schrieb Michael Rogers via anti-censorship-team:
It looks like there's some work in progress to address this issue in Lyrebird by adding the relevant certificates, so hopefully Meek and Snowflake will work in a future Lyrebird release. But what about the initial connection to Moat?
This also uses Lyrebird's meek-lite transport. So, if Lyrebird is capable of handling the certificates, the connections to RdSys/Moat/Circumvention API will als work.
Cheers,
~tla
On 02/01/2025 17:02, Benjamin Erhart via anti-censorship-team wrote:
Am 01.01.25 um 19:29 schrieb Michael Rogers via anti-censorship-team:
It looks like there's some work in progress to address this issue in Lyrebird by adding the relevant certificates, so hopefully Meek and Snowflake will work in a future Lyrebird release. But what about the initial connection to Moat?
This also uses Lyrebird's meek-lite transport. So, if Lyrebird is capable of handling the certificates, the connections to RdSys/Moat/ Circumvention API will als work.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
Cheers, Michael
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/
On 14/01/2025 18:47, cohosh--- via anti-censorship-team wrote:
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/
Hi cohosh,
Thanks for following up on this.
I made a mistake about OnionShare's use of Moat. It *does* use obfs4proxy for the meek connection (and will use Lyrebird in the next release), but then the HTTPS request through the meek tunnel fails on Android < 7.1.1 because bridges.torproject.org also uses a Let's Encrypt certificate. To fix that, I need to investigate whether we can get the libraries we're using for HTTPS (OkHttp and Conscrypt) to trust a pinned certificate.
Another thing I need to investigate is a difference in failure modes between Lyrebird 0.5.0 and obfs4proxy 0.0.14 on Android 7.0.
On Android 7.1.1, both versions of the transport are able to make meek connections via CDN77 and Azure and then connect to bridges.torproject.org through the tunnel.
On Android 7.0, both versions of the transport are able to make meek connections to Azure. (They then fail to connect to bridges.torproject.org through the tunnel as I mentioned above, but that's a separate issue.)
But on Android 7.0, obfs4proxy 0.0.14 can make a meek connection to CDN77 before failing to connect through the tunnel, whereas Lyrebird 0.5.0 fails with a timeout error instead.
When I first read your message I thought, "Aha, the out-of-date certificate pin is causing the meek connection to CDN77 to fail, that's why we see a timeout with Lyrebird but not with obfs4proxy." But now I have a doubt. If there's an out-of-date pin then shouldn't Lyrebird + CDN77 also fail on Android 7.1.1? Or is the pin only consulted if the platform's certificate store fails to validate the certificate (which would be the case for Android 7.0 + CDN77 + Lyrebird, but not for any other combination)?
Cheers, Michael
On 2025-01-15 12:47, Michael Rogers via anti-censorship-team wrote:
On 14/01/2025 18:47, cohosh--- via anti-censorship-team wrote:
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/meek/
Hi cohosh,
Thanks for following up on this.
I made a mistake about OnionShare's use of Moat. It *does* use obfs4proxy for the meek connection (and will use Lyrebird in the next release), but then the HTTPS request through the meek tunnel fails on Android < 7.1.1 because bridges.torproject.org also uses a Let's Encrypt certificate. To fix that, I need to investigate whether we can get the libraries we're using for HTTPS (OkHttp and Conscrypt) to trust a pinned certificate.
Another thing I need to investigate is a difference in failure modes between Lyrebird 0.5.0 and obfs4proxy 0.0.14 on Android 7.0.
On Android 7.1.1, both versions of the transport are able to make meek connections via CDN77 and Azure and then connect to bridges.torproject.org through the tunnel.
On Android 7.0, both versions of the transport are able to make meek connections to Azure. (They then fail to connect to bridges.torproject.org through the tunnel as I mentioned above, but that's a separate issue.)
But on Android 7.0, obfs4proxy 0.0.14 can make a meek connection to CDN77 before failing to connect through the tunnel, whereas Lyrebird 0.5.0 fails with a timeout error instead.
When I first read your message I thought, "Aha, the out-of-date certificate pin is causing the meek connection to CDN77 to fail, that's why we see a timeout with Lyrebird but not with obfs4proxy." But now I have a doubt. If there's an out-of-date pin then shouldn't Lyrebird + CDN77 also fail on Android 7.1.1? Or is the pin only consulted if the platform's certificate store fails to validate the certificate (which would be the case for Android 7.0 + CDN77 + Lyrebird, but not for any other combination)?
Well, we never shipped the pin for Lyrebird. That was a trial patch I discarded because it didn't solve the problem we were seeing. Your last guess is correct. What we're doing in Snowflake isn't restrictively pinning certs. We're just appending the Let's Encrypt root cert to the root CA pool[0]. So if a platform, like Android 7.1.1 and above, already has a working cert in it's store, it doesn't need the pinned cert and just ignores it.
I am curious about the CDN77 meek connection working while the internal tunnelled connection fails on < Android 7.1.1. This sounds like exactly the issue we've been trying to debug with Tor Browser[1], when domain fronting stopped working with Fastly, which wasn't using Let's Encrypt. It didn't occur to me to think about certificate validation in the tunnelled connection... I guess I assumed that the application would do the right thing with the expired cross-signed root cert. But maybe the libraries used by Tor Browser and OnionShare are similar to the Go library in this way, and actually are enforcing the expiration date.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf... [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb...
On 15/01/2025 20:03, Cecylia Bocovich via anti-censorship-team wrote:
On 2025-01-15 12:47, Michael Rogers via anti-censorship-team wrote:
On 14/01/2025 18:47, cohosh--- via anti-censorship-team wrote:
Hi Michael,
We addressed the original problem in Snowflake a while ago by pinning the ISRG Root X1 cert[0], and I wrote a quick patch for Lyrebird[1] that does the same. Although, indeed, it looks like the cert we've pinned[2] has expired, and we need to update that.
Thanks, good to know. The Moat library used by OnionShare Android doesn't use Lyrebird for its Moat connection but it looks like that's our best path forward.
This won't work quite yet without that above patch merged and also updated to use an up-to-date root cert. Is OnionShare using the original meek client[3]? If so, we can also work on adding cert pinning support there.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/snowflake/-/issues/40087 [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/lyrebird/-/merge_requests/62 [2] https://crt.sh/?id=3958242236 [3] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/meek/
Hi cohosh,
Thanks for following up on this.
I made a mistake about OnionShare's use of Moat. It *does* use obfs4proxy for the meek connection (and will use Lyrebird in the next release), but then the HTTPS request through the meek tunnel fails on Android < 7.1.1 because bridges.torproject.org also uses a Let's Encrypt certificate. To fix that, I need to investigate whether we can get the libraries we're using for HTTPS (OkHttp and Conscrypt) to trust a pinned certificate.
Another thing I need to investigate is a difference in failure modes between Lyrebird 0.5.0 and obfs4proxy 0.0.14 on Android 7.0.
On Android 7.1.1, both versions of the transport are able to make meek connections via CDN77 and Azure and then connect to bridges.torproject.org through the tunnel.
On Android 7.0, both versions of the transport are able to make meek connections to Azure. (They then fail to connect to bridges.torproject.org through the tunnel as I mentioned above, but that's a separate issue.)
But on Android 7.0, obfs4proxy 0.0.14 can make a meek connection to CDN77 before failing to connect through the tunnel, whereas Lyrebird 0.5.0 fails with a timeout error instead.
When I first read your message I thought, "Aha, the out-of-date certificate pin is causing the meek connection to CDN77 to fail, that's why we see a timeout with Lyrebird but not with obfs4proxy." But now I have a doubt. If there's an out-of-date pin then shouldn't Lyrebird + CDN77 also fail on Android 7.1.1? Or is the pin only consulted if the platform's certificate store fails to validate the certificate (which would be the case for Android 7.0 + CDN77 + Lyrebird, but not for any other combination)?
Well, we never shipped the pin for Lyrebird. That was a trial patch I discarded because it didn't solve the problem we were seeing. Your last guess is correct. What we're doing in Snowflake isn't restrictively pinning certs. We're just appending the Let's Encrypt root cert to the root CA pool[0]. So if a platform, like Android 7.1.1 and above, already has a working cert in it's store, it doesn't need the pinned cert and just ignores it.
Thanks, this is good to know. In that case I can't explain the difference in failure modes between obfs4proxy 0.0.14 and Lyrebird 0.5.0 when connecting to CDN77 on Android 7.0.
One possibility is that it's somehow related to utls being re-added in 0.0.14-tor1. When I get some time I'll test with that version to see if it behaves like 0.0.14 or 0.5.0.
I am curious about the CDN77 meek connection working while the internal tunnelled connection fails on < Android 7.1.1. This sounds like exactly the issue we've been trying to debug with Tor Browser[1], when domain fronting stopped working with Fastly, which wasn't using Let's Encrypt. It didn't occur to me to think about certificate validation in the tunnelled connection... I guess I assumed that the application would do the right thing with the expired cross-signed root cert. But maybe the libraries used by Tor Browser and OnionShare are similar to the Go library in this way, and actually are enforcing the expiration date.
So just to make sure we're on the same page, on Android < 7.1.1, the meek connection to CDN77 works and the tunnelled connections fails with obfs4proxy 0.0.14, but with Lyrebird 0.5.0 there's a timeout instead. I don't know yet whether the timeout is happening while making the meek connection or the tunnelled connection, but I suspect it's the meek connection because the stack used for the tunneled connection is the same in both cases.
For both Azure and CDN77, the tunnelled connection fails on Android < 7.1.1 even when we use Conscrypt to get an up-to-date TLS implementation, because Conscrypt is still relying on the phone's certificate store. I would guess that the same might be true for whatever TLS implementation Tor Browser uses on Android?
Until recently, I thought we'd solved this issue by using Conscrypt. But it turns out that it was just the Let's Encrypt cross-signing workaround, which has now expired, so the problem has re-emerged.
[0] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/snowflake/-/merge_requests/151/ diffs#a8b4f0bf23e091c376d7ebea60b394b7278b6250_0_49 [1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/lyrebird/-/issues/40012
Cheers, Michael
Hi again :)
On 20/01/2025 13:30, Michael Rogers via anti-censorship-team wrote:
I made a mistake about OnionShare's use of Moat. It *does* use obfs4proxy for the meek connection (and will use Lyrebird in the next release), but then the HTTPS request through the meek tunnel fails on Android < 7.1.1 because bridges.torproject.org also uses a Let's Encrypt certificate. To fix that, I need to investigate whether we can get the libraries we're using for HTTPS (OkHttp and Conscrypt) to trust a pinned certificate.
Another thing I need to investigate is a difference in failure modes between Lyrebird 0.5.0 and obfs4proxy 0.0.14 on Android 7.0.
On Android 7.1.1, both versions of the transport are able to make meek connections via CDN77 and Azure and then connect to bridges.torproject.org through the tunnel.
On Android 7.0, both versions of the transport are able to make meek connections to Azure. (They then fail to connect to bridges.torproject.org through the tunnel as I mentioned above, but that's a separate issue.)
But on Android 7.0, obfs4proxy 0.0.14 can make a meek connection to CDN77 before failing to connect through the tunnel, whereas Lyrebird 0.5.0 fails with a timeout error instead.
An update on this:
* I've persuaded OkHttp to validate the Let's Encrypt cert from bridges.torproject.org on Android < 7.1.1 by using a custom X509TrustManager. * This has revealed that the difference in failure modes between obfs4proxy 0.0.14 and Lyrebird 0.5.0 when using CDN77 on Android < 7.1.1 was not what I thought. * Lyrebird fails with a timeout before making the meek connection. * obfs4proxy fails with an SSLHandshakeException after making the meek connection. I thought this was due to OkHttp failing to validate the certificate, but it's actually due to the tunneled connection to bridges.torproject.org being closed before OkHttp even sees the certificate.
So neither 0.0.14 nor 0.5.0 can make a tunneled connection via CDN77 on Android < 7.1.1. The failure modes are different but they both fail to create a usable connection.
Since both versions *can* make a usable connection on Android >= 7.1.1, where the root cert used by Let's Encrypt is in the device's certificate store, I think the issue must be related to the PT's TLS handshake relying on the device's certificate store. So to get this working, we need to be able to validate Let's Encrypt certs at the PT layer as well as the proxied-application layer.
I'll start working on this next, but if anyone on the list has experience with customising certificate validation in Go, I'd be grateful for any hints you can offer!
Thanks, Michael
On 04/02/2025 11:30, Michael Rogers via anti-censorship-team wrote:
Since both versions *can* make a usable connection on Android >= 7.1.1, where the root cert used by Let's Encrypt is in the device's certificate store, I think the issue must be related to the PT's TLS handshake relying on the device's certificate store. So to get this working, we need to be able to validate Let's Encrypt certs at the PT layer as well as the proxied-application layer.
I'll start working on this next, but if anyone on the list has experience with customising certificate validation in Go, I'd be grateful for any hints you can offer!
Using the patch from Lyrebird !62 [1], but with the updated ISRG root cert that's currently in the snowflake repo [2], rather than the one referenced by the patch, gets Lyrebird working with CDN77 on Android < 7.1.1.
Since there isn't a snowflake release that includes the updated cert, I copied it into Lyrebird's meeklite package where it's used. If that's an acceptable solution for upstream I can open an MR.
Cheers, Michael
[1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
On 2025-02-04 09:21, Michael Rogers via anti-censorship-team wrote:
On 04/02/2025 11:30, Michael Rogers via anti-censorship-team wrote:
Since both versions *can* make a usable connection on Android >= 7.1.1, where the root cert used by Let's Encrypt is in the device's certificate store, I think the issue must be related to the PT's TLS handshake relying on the device's certificate store. So to get this working, we need to be able to validate Let's Encrypt certs at the PT layer as well as the proxied-application layer.
I'll start working on this next, but if anyone on the list has experience with customising certificate validation in Go, I'd be grateful for any hints you can offer!
Using the patch from Lyrebird !62 [1], but with the updated ISRG root cert that's currently in the snowflake repo [2], rather than the one referenced by the patch, gets Lyrebird working with CDN77 on Android < 7.1.1.
Since there isn't a snowflake release that includes the updated cert, I copied it into Lyrebird's meeklite package where it's used. If that's an acceptable solution for upstream I can open an MR.
Cheers, Michael
[1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/lyreb... [2] https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowf...
Thank you for your efforts to debug this problem and for working on the solution! It's totally fine to copy it into lyrebird for the MR. We can de-duplicate this later, and may decide to factor it out into a separate ptutil library.
Cecylia
On 04/02/2025 20:33, Cecylia Bocovich via anti-censorship-team wrote:
On 2025-02-04 09:21, Michael Rogers via anti-censorship-team wrote:
On 04/02/2025 11:30, Michael Rogers via anti-censorship-team wrote:
Since both versions *can* make a usable connection on Android >= 7.1.1, where the root cert used by Let's Encrypt is in the device's certificate store, I think the issue must be related to the PT's TLS handshake relying on the device's certificate store. So to get this working, we need to be able to validate Let's Encrypt certs at the PT layer as well as the proxied-application layer.
I'll start working on this next, but if anyone on the list has experience with customising certificate validation in Go, I'd be grateful for any hints you can offer!
Using the patch from Lyrebird !62 [1], but with the updated ISRG root cert that's currently in the snowflake repo [2], rather than the one referenced by the patch, gets Lyrebird working with CDN77 on Android < 7.1.1.
Since there isn't a snowflake release that includes the updated cert, I copied it into Lyrebird's meeklite package where it's used. If that's an acceptable solution for upstream I can open an MR.
Cheers, Michael
[1] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/lyrebird/-/merge_requests/62 [2] https://gitlab.torproject.org/tpo/anti-censorship/pluggable- transports/snowflake/-/blob/26f7ee4b0620b5b64f3b7df6b139891a7b0170c8/ common/certs/certs.go
Thank you for your efforts to debug this problem and for working on the solution! It's totally fine to copy it into lyrebird for the MR. We can de-duplicate this later, and may decide to factor it out into a separate ptutil library.
Cecylia
Great, I've opened !77 for this.
Cheers, Michael
anti-censorship-team@lists.torproject.org
-
Benjamin Erhart -
Cecylia Bocovich -
cohosh＠torproject.org
-
Michael Rogers