We're making progress on meek (https://trac.torproject.org/projects/tor/wiki/doc/meek), the transport that hides your traffic in HTTPS requests to an unblockable web site. It's already doing a good job at reliably transfering bits. What we'd like to do next is remove trivial means of blocking it based on network fingerprinting. We're currently using the golang HTTPS library, and at https://trac.torproject.org/projects/tor/wiki/doc/meek#Distinguishability you can see meek's TLS differs from Chromium's in its ciphersuites and extensions.
It seems the right thing to do is mimic a browser, and I can think of at least three ways to do that: 1. Try really hard, using NSS or some other library, to look like a particular browser. 2. Run a second browser, apart from Tor Browser, that receives commands from a client PT program and makes the HTTPS requests it is commanded to. 3. Run a browser plugin *inside* Tor Browser, that makes HTTPS requests *directly on the Internet, without going through Tor*. That is, the plugin receives commands from the client PT program, and then bypasses all of Tor Browser's proxy settings in order to send HTTPS requests to the web site fronting the circumvention.
It's the third option I want to ask about. The first option puts us on the parrot treadmill. The second has the usability and distribution problems of running two browsers at once. The third is slightly crazy, but more usable and easier to deploy. The third option is definitely what I would do if I were using a browser other than Tor Browser and designing purely for circumvention and not anonymity.
Is it even possible to run a plugin that bypasses the proxy settings? (I know a binary plugin like Flash can do it--but does that use the browser's HTTP engine or do they cook up their own HTTP with sockets?) Would Tor Browser's customizations, for example the modified User-Agent string, defeat the purpose of looking like an ordinary browser? Anything else I'm missing?
This is just brainstorming; negative answers won't be the end of the project. The first option might not be too bad if we use NSS directly.
David Fifield
It seems the right thing to do is mimic a browser, and I can think of at least three ways to do that:
- Try really hard, using NSS or some other library, to look like a particular browser.
- Run a second browser, apart from Tor Browser, that receives commands from a client PT program and makes the HTTPS requests it is commanded to.
- Run a browser plugin *inside* Tor Browser, that makes HTTPS requests *directly on the Internet, without going through Tor*. That is, the plugin receives commands from the client PT program, and then bypasses all of Tor Browser's proxy settings in order to send HTTPS requests to the web site fronting the circumvention.
It's the third option I want to ask about. The first option puts us on the parrot treadmill. The second has the usability and distribution problems of running two browsers at once.
Usability might not be such an issue if you're using a headless browser. Distribution still would be.
On Sat, Feb 22, 2014 at 05:47:55PM +0000, Arlo Breault wrote:
It seems the right thing to do is mimic a browser, and I can think of at least three ways to do that: 1. Try really hard, using NSS or some other library, to look like a particular browser. 2. Run a second browser, apart from Tor Browser, that receives commands from a client PT program and makes the HTTPS requests it is commanded to. 3. Run a browser plugin *inside* Tor Browser, that makes HTTPS requests *directly on the Internet, without going through Tor*. That is, the plugin receives commands from the client PT program, and then bypasses all of Tor Browser's proxy settings in order to send HTTPS requests to the web site fronting the circumvention. It's the third option I want to ask about. The first option puts us on the parrot treadmill. The second has the usability and distribution problems of running two browsers at once.Usability might not be such an issue if you're using a headless browser. Distribution still would be.
Thanks Arlo. I took this idea and some from IRC discussion and summarized them here:
https://trac.torproject.org/projects/tor/wiki/doc/meek#HowtolooklikebrowserH...
1. Use your own HTTPS/TLS library, and take care to make sure your ciphersuites and extensions match those of a browser. There are [https://www.mozilla.org/projects/security/pki/python-nss/ Python bindings for NSS] that might make it easier. Chromium is [https://code.google.com/p/chromium/issues/detail?id=62803 moving to OpenSSL] in the future. 2. Use a separate (headless) browser as an instrument for making HTTPS requests. This is what [https://raw.github.com/wiki/gsathya/htpt/Overall_architecture2.png htpt plans to do]. [http://phantomjs.org/ PhantomJS] is a headless WebKit that is scriptable with JavaScript. Its compressed size is 7–13 MB. [https://github.com/ariya/phantomjs/blob/master/examples/postserver.js This postserver.js example] shows it running its own web server, which we could use as a means of communication: meek-client on localhost ←HTTP→ PhantomJS on localhost ←HTTPS→ www.google.com. Another option is to write an extension for some other browser and communicate with it using some custom IPC. 3. Use an [https://developer.mozilla.org/en/Extensions extension] in Tor Browser itself. The plugin bypasses Tor Browser's normal proxy settings in order to issue HTTPS requests directly to the front domain. * [tor-dev] Feasibility of using a Tor Browser plugin as a PT component? https://lists.torproject.org/pipermail/tor-dev/2014-February/006266.html GeKo says that [https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsISocket... nsISocketTransportService] is what we want to look at. * [https://stackoverflow.com/questions/10173811/how-to-connect-to-a-remote-serv... How to connect to a remote server using nsISocketTransportService in a firefox extension?] * [https://code.google.com/p/weaponry/source/browse/trunk/xulrunner/weaponry/di... WeaponryRawHttpRequest.js] is doing what we want. [https://developer.mozilla.org/en-US/docs/Mozilla/XPCOM XPCOM] (a Firefox API) allows you to create [https://developer.mozilla.org/en-US/docs/WebAPI/TCP_Socket TCP sockets].
David Fifield
On 2014-02-22 21:36, David Fifield wrote:
2. Run a second browser, apart from Tor Browser, that receives commands from a client PT program and makes the HTTPS requests it is commanded to.
[..]
If all is well, there should be a paper at PETS2014 which solves exactly this.... (and with some tweaks meek was able to work over it ;)
Greets, Jeroen
On 22/02/14 04:08, David Fifield wrote:
- Run a second browser, apart from Tor Browser, that receives commands from a client PT program and makes the HTTPS requests it is commanded to.
You might want to look at MozRepl. More summary here:
https://lists.torproject.org/pipermail/tor-dev/2013-November/005833.html
I told gsathya about this and he had a brief look during NYC and thought it was potentially feasible, but AFAIK we haven't done a complete analysis of this yet.
X
On 02/22/2014 08:54 PM, Ximin Luo wrote:
On 22/02/14 04:08, David Fifield wrote:
- Run a second browser, apart from Tor Browser, that receives
commands from a client PT program and makes the HTTPS requests it is commanded to.
You might want to look at MozRepl. More summary here:
https://lists.torproject.org/pipermail/tor-dev/2013-November/005833.html
I told gsathya about this and he had a brief look during NYC and thought it was potentially feasible, but AFAIK we haven't done a complete analysis of this yet.
I have been working on this for a related project (with slightly different goals -- I need to drive TBB itself from automation). Unfortunately, MozRepl does *not* AFAICT expose any machine-friendly interface, just the telnet interface geared for humans. So I forked it and have been hacking up a strictly machine-oriented interface, here: https://github.com/zackw/firefox-puppeteer It's not even close to done, and the current design sketch (see the README) is extremely focused on my particular needs (e.g. use of ZeroMQ as wire protocol), but I'd be delighted to make it useful for more people than just me (especially if that comes with coding assistance ;-)
On the larger topic, looping network traffic back through the same process that originated it strikes me as asking for deadlocks, so I'd be inclined to go with (2) over (3) if it were me coding it. That said, in Firefox, extensions do have the ability to override the normal proxy settings, although it's a bit awkward (there doesn't appear to be a way to say "do _this HTTP request_ with no proxy", only "ignore what the prefs say if this function says so, for any URL"; but I could have missed something).
zw
On Fri, Feb 21, 2014 at 08:08:59PM -0800, David Fifield wrote:
We're making progress on meek (https://trac.torproject.org/projects/tor/wiki/doc/meek), the transport that hides your traffic in HTTPS requests to an unblockable web site. It's already doing a good job at reliably transfering bits. What we'd like to do next is remove trivial means of blocking it based on network fingerprinting. We're currently using the golang HTTPS library, and at https://trac.torproject.org/projects/tor/wiki/doc/meek#Distinguishability you can see meek's TLS differs from Chromium's in its ciphersuites and extensions.
It seems the right thing to do is mimic a browser, and I can think of at least three ways to do that:
- Try really hard, using NSS or some other library, to look like a particular browser.
- Run a second browser, apart from Tor Browser, that receives commands from a client PT program and makes the HTTPS requests it is commanded to.
- Run a browser plugin *inside* Tor Browser, that makes HTTPS requests *directly on the Internet, without going through Tor*. That is, the plugin receives commands from the client PT program, and then bypasses all of Tor Browser's proxy settings in order to send HTTPS requests to the web site fronting the circumvention.
It's the third option I want to ask about. The first option puts us on the parrot treadmill. The second has the usability and distribution problems of running two browsers at once. The third is slightly crazy, but more usable and easier to deploy. The third option is definitely what I would do if I were using a browser other than Tor Browser and designing purely for circumvention and not anonymity.
I started trying to write a Firefox extension that makes HTTP requests outside of the proxy settings. I have one that works in Iceweasel 24.3 and does the Host header trick used by the transport. However it doesn't work in Tor Browser, and I'm looking for some insight as to why it might be so.
The source code of the extension is in the "firefox" directory of git clone -b extension https://www.bamsoftware.com/git/meek.git Instructions on how to try it are: https://developer.mozilla.org/en-US/docs/Building_an_Extension#Test. I also pasted the important JavaScript code at the end of this message.
The extension just makes an HTTPS request when you start the browser. The request is apparently for https://www.google.com/ but it actually goes to https://meek-reflect.appspot.com/. In Iceweasel with window.dump enabled, this is the output (in addition Wireshark shows me the request is happening):
*** LOG addons.xpi: startup *** LOG addons.xpi: checkForChanges *** LOG addons.xpi-utils: Opening database *** LOG addons.xpi: Add-on meek-http-helper@bamsoftware.com modified in app-profile *** LOG addons.xpi: Updating database with changes to installed add-ons *** LOG addons.xpi-utils: Updating add-on states *** LOG addons.xpi-utils: Writing add-ons list onStartRequest onDataAvailable 38:[object Uint8Array] onStopRequest
"onStartRequest", "onDataAvailable", and "onStopRequest" are output from the extension. 38 is the length of the "I’m just a happy little web server.\n" response from the bridge. I configured a proxy in Iceweasel and the request bypasses the proxy. Everything looks right.
Here is the output when I run in TBB 3.5.2.1. There's "onStartRequest" and "onStartRequest" but no "onDataAvailable" with the data. Wireshark doesn't show any HTTPS request happening.
*** LOG addons.xpi: startup *** LOG addons.xpi: checkForChanges *** LOG addons.xpi-utils: Opening database *** LOG addons.xpi: Add-on meek-http-helper@bamsoftware.com modified in app-profile *** LOG addons.xpi: Updating database with changes to installed add-ons *** LOG addons.xpi-utils: Updating add-on states *** LOG addons.xpi-utils: Writing add-ons list no loadgroup notificationCallbacks for https://www.google.com/ onStartRequest onStopRequest
"no loadgroup notificationCallbacks" is a string from HTTPS Everywhere. Any ideas as to why it doesn't work? I've never written a browser extension before so I'd appreciate tips on how to approach debugging it.
David Fifield
var FRONT_URL = "https://www.google.com/"; var HOST = "meek-reflect.appspot.com";
// Create a "direct" nsIProxyInfo that bypasses the default proxy. // https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIProtoc... var pps = Components.classes["@mozilla.org/network/protocol-proxy-service;1"] .getService(Components.interfaces.nsIProtocolProxyService); // https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIProxyI... var proxy = pps.newProxyInfo("direct", "", 0, 0, 0xffffffff, null);
// https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIIOServ... var ioService = Components.classes["@mozilla.org/network/io-service;1"] .getService(Components.interfaces.nsIIOService); var httpProtocolHandler = ioService.getProtocolHandler("http") .QueryInterface(Components.interfaces.nsIHttpProtocolHandler); var uri = ioService.newURI(FRONT_URL, null, null); // Construct an HTTP channel with the proxy bypass. // https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIHttpCh... var channel = httpProtocolHandler.newProxiedChannel(uri, proxy, 0, null) .QueryInterface(Components.interfaces.nsIHttpChannel); // Set the host we really want. channel.setRequestHeader("Host", HOST, false); channel.redirectionLimit = 0; // https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIUpload... // channel.requestMethod = "POST";
var listener = new StreamListener(); channel.asyncOpen(listener, null);
// https://developer.mozilla.org/en-US/docs/Creating_Sandboxed_HTTP_Connections function StreamListener() { this.onStartRequest = function(aRequest, aContext) { console.log("onStartRequest\n"); dump("onStartRequest\n"); }; this.onStopRequest = function(aRequest, aContext, aStatus) { dump("onStopRequest\n"); }; this.onDataAvailable = function(aRequest, aContext, aStream, aSourceOffset, aLength) { dump("onDataAvailable\n"); var a = new Uint8Array(aLength); var input = Components.classes["@mozilla.org/binaryinputstream;1"] .createInstance(Components.interfaces.nsIBinaryInputStream); input.setInputStream(aStream); input.readByteArray(aLength, a); dump(aLength + ":" + a + "\n"); }; }
On Mon, Mar 10, 2014 at 08:23:35PM -0700, David Fifield wrote:
I started trying to write a Firefox extension that makes HTTP requests outside of the proxy settings. I have one that works in Iceweasel 24.3 and does the Host header trick used by the transport. However it doesn't work in Tor Browser, and I'm looking for some insight as to why it might be so.
I started https://trac.torproject.org/projects/tor/ticket/11183 for development of the browser extension. XUL and XPCOM experts are invited to comment.
David Fifield
On 3/10/14, 11:23 PM, David Fifield wrote:
I started trying to write a Firefox extension that makes HTTP requests outside of the proxy settings. I have one that works in Iceweasel 24.3 and does the Host header trick used by the transport. However it doesn't work in Tor Browser, and I'm looking for some insight as to why it might be so.
The source code of the extension is in the "firefox" directory of git clone -b extension https://www.bamsoftware.com/git/meek.git Instructions on how to try it are: https://developer.mozilla.org/en-US/docs/Building_an_Extension#Test. I also pasted the important JavaScript code at the end of this message.
I looked at this for a few minutes but ran out of time for today.
When I dump aStatus in your onStopRequest function, I get 2152398890 which is 0x804B002A which is NS_ERROR_UNKNOWN_PROXY_HOST (see https://developer.mozilla.org/en-US/docs/Table_Of_Errors).
I am not sure what that means but it sounds interesting.
On Tue, Mar 11, 2014 at 05:22:37PM -0400, Mark Smith wrote:
On 3/10/14, 11:23 PM, David Fifield wrote:
I started trying to write a Firefox extension that makes HTTP requests outside of the proxy settings. I have one that works in Iceweasel 24.3 and does the Host header trick used by the transport. However it doesn't work in Tor Browser, and I'm looking for some insight as to why it might be so.
The source code of the extension is in the "firefox" directory of git clone -b extension https://www.bamsoftware.com/git/meek.git Instructions on how to try it are: https://developer.mozilla.org/en-US/docs/Building_an_Extension#Test. I also pasted the important JavaScript code at the end of this message.
I looked at this for a few minutes but ran out of time for today.
When I dump aStatus in your onStopRequest function, I get 2152398890 which is 0x804B002A which is NS_ERROR_UNKNOWN_PROXY_HOST (see https://developer.mozilla.org/en-US/docs/Table_Of_Errors).
I am not sure what that means but it sounds interesting.
Thanks for finding this clue. It appears to me that the critical difference is in the network.proxy.socks_remote_dns setting. My Iceweasel had it false and Tor Browser had it true. If it make it true in Iceweasel, the extension fails; if I make it false in Tor Browser, the extension succeeds. I'll see if there's a way to make it work with socks_remote_dns=true.
The NS_ERROR_UNKNOWN_PROXY_HOST was coming from the "" hostname passed to newProxyInfo. If I change it to "127.0.0.1", I instead get the error 0x804b0048 NS_ERROR_PROXY_CONNECTION_REFUSED, and I see that Firefox is actually trying to connect to 127.0.0.1, despite that a "direct" proxy type is in use.
David Fifield
On Tue, Mar 11, 2014 at 10:31:16PM -0700, David Fifield wrote:
On Tue, Mar 11, 2014 at 05:22:37PM -0400, Mark Smith wrote:
On 3/10/14, 11:23 PM, David Fifield wrote:
I started trying to write a Firefox extension that makes HTTP requests outside of the proxy settings. I have one that works in Iceweasel 24.3 and does the Host header trick used by the transport. However it doesn't work in Tor Browser, and I'm looking for some insight as to why it might be so.
The source code of the extension is in the "firefox" directory of git clone -b extension https://www.bamsoftware.com/git/meek.git Instructions on how to try it are: https://developer.mozilla.org/en-US/docs/Building_an_Extension#Test. I also pasted the important JavaScript code at the end of this message.
I looked at this for a few minutes but ran out of time for today.
When I dump aStatus in your onStopRequest function, I get 2152398890 which is 0x804B002A which is NS_ERROR_UNKNOWN_PROXY_HOST (see https://developer.mozilla.org/en-US/docs/Table_Of_Errors).
I am not sure what that means but it sounds interesting.
Thanks for finding this clue. It appears to me that the critical difference is in the network.proxy.socks_remote_dns setting. My Iceweasel had it false and Tor Browser had it true. If it make it true in Iceweasel, the extension fails; if I make it false in Tor Browser, the extension succeeds. I'll see if there's a way to make it work with socks_remote_dns=true.
I made a mistake when I said this. The issue turns out not to be the socks_remote_dns setting. The extension works in Iceweasel whether socks_remote_dns is true or false, but works in Tor Browser only when it is false. Mike traced the cause to this patch in Tor Browser, which guard against DNS leaks by prohibiting name lookups when socks_remote_dns=true, which is its default setting. https://gitweb.torproject.org/tor-browser.git/commitdiff/5069a3ee8fa51546a8a... If I undo that patch, the extension works in Tor Browser with no other changes.
I wrote a summary of the situation here: https://trac.torproject.org/projects/tor/ticket/11183#comment:6
David Fifield
-
Arlo Breault -
David Fifield -
Jeroen Massar -
Mark Smith -
Ximin Luo -
Zack Weinberg