Hi,
As I said to Matt and Georg during a meeting, I've had some major headaches with getting Moat to work in Tails, and even debugging it has been hard. To give you an idea of the latter, I couldn't get access to Tor Launcher's logs, so I had to ugly-patch it to always spam the full log into the clipboard just to get it some how.
For the former, the actual error I see is this prompt, right after clicking "Request a bridge...":
Unable to obtain a bridge from BridgeDB.
NS_ERROR_NET_INTERRUPT
I've attached the logs of Tor Launcher and obfs4proxy when this happens. To me it certainly seems like the problem is somewhere in their interaction.
Note that Tails runs Tor Launcher as a standalone XUL application (via `firefox --app`, using the executable from Tor Browser) and not a (system) extension as in Tor Browser. This might be responsible for the issues I'm having, but if so I have no idea why, or even how to approach debugging it.
Any ideas?
How to reproduce inside Tails =============================
If you feel adventurous and would like to try to reproduce this and investigate further, you can fetch an image built from the feature branch here:
https://nightly.tails.boum.org/build_Tails_ISO_feature-8243-meek/lastSuccess...
<Note> The Tails project is in the process of migrating from Redmine to GitLab, and as a result our Jenkins isn't building images for us. Therefore the last images available at the moment are built yesterday (2020-05-19) and lack a small fix I just pushed: enable obfs4proxy debug logging. You can enable the logging yourself by running the following as root:
. /usr/local/lib/tails-shell-library/tor.sh tor_control_setconf 'ClientTransportPlugin="obfs2,obfs3,obfs4,meek_lite exec /usr/local/lib/obfs4proxy/obfs4proxy -enableLogging=true -logLevel DEBUG managed"'
</Note>
Booting the .iso in a VM of you choice is probably the easiest approach.
If you want to look at the code of the feature branch, feature/8243-meek [0], you can do so here:
https://git.tails.boum.org/tails/log/?h=feature/8243-meek
or if you prefer to Git clone it:
git clone https://git-tails.immerda.ch/tails && \ cd tails && \ git checkout feature/8243-meek
The build script which installs Tor Browser and Tor Launcher is probably the most interesting bit for you (my guess is that you don't need to look at it, though):
https://git.tails.boum.org/tails/tree/config/chroot_local-hooks/10-tbb?h=fea...
Once you boot Tails and reach the Greeter you'll have to set two "Additional settings" by pressing the + button (in the GUI, not on the keyboard):
* An administrative password so we can run stuff as root * Enable Tor Launcher via: Network configuration → Configure a Tor bridge or local proxy
As soon as you have logged in and your network is up you need to run this as root:
/usr/local/lib/do_not_ever_run_me
which disables the firewall completely (I haven't bothered opening up properly for Moat yet). This command must be rerun if your network reconnects.
As you may notice Tor Launcher will start automatically once you log in and have a network connection, but if you want to run it again without having to reboot (very likely!) you can do so by running this as root:
tails-tor-launcher
Some other things that might be useful when debugging -----------------------------------------------------
The path to the obfs4proxy executable (copied straight from the Tor Browser bundle):
/usr/local/lib/obfs4proxy/obfs4proxy
We run Tor Launcher under a dedicated 'tor-launcher' user whose $HOME is:
/home/tor-launcher
The Tor Launcher's app profile is located in:
/home/tor-launcher/.tor-launcher/profile.default
The obfs4proxy log can be found in:
/home/tor-launcher/Tor/pt_state/obfs4proxy.log
And as I already mentioned, Tor Launcher is patched to dump its full debug log into the clipboard for each log event, so just paste into Gedit or whatever. I also recommend quitting Tor Launcher as soon as you have done this, as it otherwise will keep spamming your clipboard and probably mess things up for you. :)
Cheers!
[0] It's in the branch for Meek because when we had (successfully) implemented a PoC for it, it occurred to us that we need to provide Moat at the same time as an alternative. Details: https://lists.torproject.org/pipermail/tor-dev/2020-March/014189.html
On 5/20/20 8:54 AM, anonym wrote:
Hi,
As I said to Matt and Georg during a meeting, I've had some major headaches with getting Moat to work in Tails, and even debugging it has been hard. To give you an idea of the latter, I couldn't get access to Tor Launcher's logs, so I had to ugly-patch it to always spam the full log into the clipboard just to get it some how.
For the former, the actual error I see is this prompt, right after clicking "Request a bridge...":
Unable to obtain a bridge from BridgeDB. NS_ERROR_NET_INTERRUPTI've attached the logs of Tor Launcher and obfs4proxy when this happens. To me it certainly seems like the problem is somewhere in their interaction.
Note that Tails runs Tor Launcher as a standalone XUL application (via `firefox --app`, using the executable from Tor Browser) and not a (system) extension as in Tor Browser. This might be responsible for the issues I'm having, but if so I have no idea why, or even how to approach debugging it.
Any ideas?
Mark and I did a quick comparison by running Tor Browser with Moat on a regular desktop system (outside of Tails). Comparing the Tor Launcher logging, everything is the same except in the Tails case no data is received from obfs4proxy. When things are working you should see something like:
[...] TorLauncher VERB: Moat JSON request: {"data":[{"version":"0.1.0","type":"client-transports","supported":["obfs4"]}]} [...] TorLauncher DBUG: Moat onDataAvailable: 63834 bytes [...] TorLauncher DBUG: Moat onDataAvailable: 25463 bytes [...] TorLauncher INFO: Moat response HTTP status: 200 [...] TorLauncher VERB: Moat JSON response: {"data": [{"id": "1", "type": "moat-challenge", ...
The obfs4proxy log looks like this: ... [NOTICE]: obfs4proxy-0.0.11 - launched ... [INFO]: obfs4proxy - initializing client transport listeners ... [INFO]: meek_lite - registered listener: 127.0.0.1:58179 ... [INFO]: obfs4proxy - accepting connections ... [DEBUG]: meek_lite - HPKP enabled for host: ajax.aspnetcdn.com ... [INFO]: meek_lite([scrubbed]:443) - closed connection ... [DEBUG]: meek_lite - HPKP enabled for host: ajax.aspnetcdn.com ... [NOTICE]: obfs4proxy - terminated
Our gut feeling is that the problem lies between obfs4proxy and the frontend server. Does the meek-azure bridge work? (it uses the same frontend server as moat)
Maybe someone like dcf, who has a deeper understanding of meek and Go, could provide some debugging ideas.
-- Kathy
For some basic debugging, you should check that these commands work to talk to the endpoint of the meek tunnel:
$ curl https://onion.azureedge.net/ I’m just a happy little web server. $ curl -H 'Host: onion.azureedge.net' https://ajax.aspnetcdn.com/ I’m just a happy little web server.
If those work, you can try communicating with Moat through the tunnel. You will need to run obfs4proxy and pass it specially formatted SOCKS credentials that encode the url= and front= parameters. You will have to change the XXXXX port number to match what obfs4proxy outputs in the CMETHOD line.
$ TOR_PT_MANAGED_TRANSPORT_VER=1 TOR_PT_STATE_LOCATION=pt_state TOR_PT_CLIENT_TRANSPORTS=meek_lite /usr/local/lib/obfs4proxy/obfs4proxy VERSION 1 CMETHOD meek_lite socks5 127.0.0.1:XXXXX CMETHODS DONE
$ curl --proxy socks5://url%3Dhttps%3A%2F%2Fonion.azureedge.net:%2F@127.0.0.1:XXXXX/ https://bridges.torproject.org/moat/ {"errors": [{"id": "0", "type": "", "version": "0.1.0", "code": 501, "status": "Not Implemented", "detail": "moat version 0.1.0 does not implement GET /moat/"}]}
$ curl --proxy socks5://url%3Dhttps%3A%2F%2Fonion.azureedge.net%2F%3Bfront=ajax.aspnetcdn.co:m@127.0.0.1:XXXXX/ https://bridges.torproject.org/moat/ {"errors": [{"id": "0", "type": "", "version": "0.1.0", "code": 501, "status": "Not Implemented", "detail": "moat version 0.1.0 does not implement GET /moat/"}]}
Hi!
David Fifield:
For some basic debugging, you should check that these commands work to talk to the endpoint of the meek tunnel:
$ curl https://onion.azureedge.net/ I’m just a happy little web server. $ curl -H 'Host: onion.azureedge.net' https://ajax.aspnetcdn.com/ I’m just a happy little web server.
Looking into this made me eventually realize that there was no DNS resolution possible in the context Tor Launcher is running from inside Tails: it was using little-t-tor (via DNSPort) as resolver, but at the time we're not connected to the Tor network, *and* we even have DisableNetwork set, so that is a no go and we find ourselves in a catch-22 situation. I fixed it by making Tor Launcher use the resolver obtained through DHCP instead. Now I have a basic PoC for Moat in Tails! \o/
Thank you so much for kicking me in the right direction!
If those work, you can try communicating with Moat through the tunnel. You will need to run obfs4proxy and pass it specially formatted SOCKS credentials that encode the url= and front= parameters. You will have to change the XXXXX port number to match what obfs4proxy outputs in the CMETHOD line.
$ TOR_PT_MANAGED_TRANSPORT_VER=1 TOR_PT_STATE_LOCATION=pt_state TOR_PT_CLIENT_TRANSPORTS=meek_lite /usr/local/lib/obfs4proxy/obfs4proxy VERSION 1 CMETHOD meek_lite socks5 127.0.0.1:XXXXX CMETHODS DONE
$ curl --proxy socks5://url%3Dhttps%3A%2F%2Fonion.azureedge.net:%2F@127.0.0.1:XXXXX/ https://bridges.torproject.org/moat/ {"errors": [{"id": "0", "type": "", "version": "0.1.0", "code": 501, "status": "Not Implemented", "detail": "moat version 0.1.0 does not implement GET /moat/"}]}
$ curl --proxy socks5://url%3Dhttps%3A%2F%2Fonion.azureedge.net%2F%3Bfront=ajax.aspnetcdn.co:m@127.0.0.1:XXXXX/ https://bridges.torproject.org/moat/ {"errors": [{"id": "0", "type": "", "version": "0.1.0", "code": 501, "status": "Not Implemented", "detail": "moat version 0.1.0 does not implement GET /moat/"}]}
The errors are really opaque when DNS isn't available! In the Tor Launcher case I got the not-so-informative "NS_ERROR_NET_INTERRUPT" I mentioned in my initial post, and for the above curl commands the failure is:
(35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to bridges.torproject.org
Neither of these got me thinking about DNS resolution. It wasn't until I fired up a packet sniffer that I saw that DNS resolution failed for ajax.aspnetcdn.com. :)
So, case closed!
Cheers!
-
anonym -
David Fifield -
Kathleen Brade