Introducing Snowflake (webrtc pt)
Hi all, Snowflake is a webrtc pluggable transport inspired by flashproxy. (https://gitweb.torproject.org/pluggable-transports/snowflake.git) Arlo, David, and I have made lots of progress on it lately, and it now appears to have reached minimum viability. The following should result in a 100% bootstrap over WebRTC: ``` git clone https://git.torproject.org/pluggable-transports/snowflake.git cd snowflake/client go get go build tor -f torrc ``` In case you have a moment, it'd be cool if you tried it out and let us know whether it works for you. I'd really appreciate hearing about all the thoughts / concerns / issues before we move forward. -- More info -- Q: What are the benefits of this PT compared with other PTs? Snowflake combines the advantages of flashproxy and meek. Primarily: - It has the convenience of Meek, but can support magnitudes more users with negligible CDN costs. (Domain fronting only used for brief signalling / NAT-piercing to setup P2P webrtc which handles the actual traffic) - Arbitrarily high numbers of volunteer proxies are possible like in flashproxy, but NATs are no longer a usability barrier - no need for manual port forwarding! Q: How does it work? 1. Volunteers visit websites which host the "snowflake" proxy. (just like flashproxy) 2. Tor clients automatically find available browser proxies over the domain fronted signaling channel. 3. Tor client and browser proxy establish a WebRTC peer connection. 4. Proxy connects to some relay. 5. Tor occurs. Q: Why is it called Snowflake? There's a bunch of "ICE" negotiation happening for WebRTC, and it also involves a great abundance of ephemeral and short-lived (and special!) volunteer proxies... Anyhow, if Snowflake seems like it would be useful / desired here, it would be awesome if we had more help getting it stable, polished, audited, deployable, etc... Plenty of work to do! <3, ~serene P.S. Make sure you're using a recent version of Go (1.5+) P.P.S. The repo is available in these locations: - https://gitweb.torproject.org/pluggable-transports/snowflake.git - https://github.com/keroserene/snowflake
On Mon, 25 Jan 2016 14:34:42 -0800 Serene <keroserene@riseup.net> wrote:
Anyhow, if Snowflake seems like it would be useful / desired here, it would be awesome if we had more help getting it stable, polished, audited, deployable, etc...
Neat. Yes, this will be useful. What are your plans for getting https://github.com/keroserene/go-webrtc to build completely in a deterministic manner? The several hours isn't per platform right? (The "easy way" is not going to cut it for distribution). What are your plans for actually getting the server side to scale well? Since you're using cgo you will run into Really Interesting behavior wrt OS threads as you try to increase concurrency. Regards, -- Yawning Angel
On Mon, Jan 25, 2016 at 2:53 PM, Yawning Angel <yawning@schwanenlied.me> wrote:
What are your plans for getting https://github.com/keroserene/go-webrtc to build completely in a deterministic manner? The several hours isn't per platform right? (The "easy way" is not going to cut it for distribution).
Totally. Building native webrtc requires google's depot_tools, and `gclient sync` takes a long time (over an hour on my machine) because chromium. Per platform takes a few minutes. The plan is to provide a build script which: - gclient syncs to some known & tested commit hash on a release branch. - Apply a patch to remove unnecessary video/audio webrtc media API stuff (only datachannels are needed for our use-case) This will also greatly reduce the archive size. - Build the archive we need using a custom gyp or ninja file. - Concat and copy the archive over for go-webrtc Does that sound like a reasonable deterministic build?
What are your plans for actually getting the server side to scale well? Since you're using cgo you will run into Really Interesting behavior wrt OS threads as you try to increase concurrency.
Right now, the server side is the same websocket relay from flashproxy. Webrtc currently happens just between the client and browser proxy - this already yields all the benefits listed above (as we assume the volunteer proxy has no problem connecting to Tor) However, it might be worth having webrtc on both sides. We already have prepared a webrtc server plugin, which the client plugin has successfully connected to directly. To use a snowflake proxy in between would require establishing two separate webrtc peerconnections per circuit. Maybe that's worth doing - but I'm not sure about the plan there, if we do decide to go that route. Best, ~s
What are your plans for getting https://github.com/keroserene/go-webrtc to build completely in a deterministic manner?
Just opened an issue, so that's about as far as we are in the planning stages. https://github.com/keroserene/go-webrtc/issues/29
The several hours isn't per platform right? (The "easy way" is not going to cut it for distribution).
No, that's just to do the `gclient sync`. Building itself is only ~10 mins on my machine.
What are your plans for actually getting the server side to scale well? Since you're using cgo you will run into Really Interesting behavior wrt OS threads as you try to increase concurrency.
The proxy ("snowflake") to server connection is using the currently deployed WebSocket bridge. The WebRTC part is only to avoid the incoming connecting to the client. I don't think the current websocket server pt uses cgo, right? It's a native implementation. But I should look ...
On Mon, 25 Jan 2016 15:32:55 -0800 Serene <keroserene@riseup.net> wrote: [snip]
What are your plans for actually getting the server side to scale well? Since you're using cgo you will run into Really Interesting behavior wrt OS threads as you try to increase concurrency.
Right now, the server side is the same websocket relay from flashproxy. Webrtc currently happens just between the client and browser proxy - this already yields all the benefits listed above (as we assume the volunteer proxy has no problem connecting to Tor)
This seems sensible.
However, it might be worth having webrtc on both sides. We already have prepared a webrtc server plugin, which the client plugin has successfully connected to directly. To use a snowflake proxy in between would require establishing two separate webrtc peerconnections per circuit. Maybe that's worth doing - but I'm not sure about the plan there, if we do decide to go that route.
Ah that's what that was. If you don't use it then, you won't have lots of misery fighting with cgo's quirks. Regards, -- Yawning Angel
On 26 Jan 2016, at 09:34, Serene <keroserene@riseup.net> wrote:
Hi all,
Snowflake is a webrtc pluggable transport inspired by flashproxy. (https://gitweb.torproject.org/pluggable-transports/snowflake.git) Arlo, David, and I have made lots of progress on it lately, and it now appears to have reached minimum viability.
The following should result in a 100% bootstrap over WebRTC: ``` git clone https://git.torproject.org/pluggable-transports/snowflake.git cd snowflake/client go get go build tor -f torrc ``` In case you have a moment, it'd be cool if you tried it out and let us know whether it works for you. I'd really appreciate hearing about all the thoughts / concerns / issues before we move forward.
I get about this far on OS X, I'm behind a NAT: Jan 26 12:25:50.063 [notice] Tor v0.2.7.6 running on Darwin with Libevent 2.0.22-stable, OpenSSL 1.0.2e and Zlib 1.2.8. … Jan 26 12:25:50.071 [notice] Opening Socks listener on 127.0.0.1:9050 Jan 26 12:25:50.000 [notice] Parsing GEOIP IPv4 file /opt/local/share/tor/geoip. Jan 26 12:25:50.000 [notice] Parsing GEOIP IPv6 file /opt/local/share/tor/geoip6. Jan 26 12:25:50.000 [notice] Bootstrapped 0%: Starting Jan 26 12:25:50.000 [notice] Delaying directory fetches: No running bridges Jan 26 12:25:52.000 [notice] Bootstrapped 5%: Connecting to directory server Jan 26 12:25:52.000 [notice] Bootstrapped 10%: Finishing handshake with directory server Any hints? Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F
I get about this far on OS X, I'm behind a NAT:
Jan 26 12:25:50.063 [notice] Tor v0.2.7.6 running on Darwin with Libevent 2.0.22-stable, OpenSSL 1.0.2e and Zlib 1.2.8. … Jan 26 12:25:50.071 [notice] Opening Socks listener on 127.0.0.1:9050 Jan 26 12:25:50.000 [notice] Parsing GEOIP IPv4 file /opt/local/share/tor/geoip. Jan 26 12:25:50.000 [notice] Parsing GEOIP IPv6 file /opt/local/share/tor/geoip6. Jan 26 12:25:50.000 [notice] Bootstrapped 0%: Starting Jan 26 12:25:50.000 [notice] Delaying directory fetches: No running bridges Jan 26 12:25:52.000 [notice] Bootstrapped 5%: Connecting to directory server Jan 26 12:25:52.000 [notice] Bootstrapped 10%: Finishing handshake with directory server
Any hints?
Can you paste a sanitized version of your snowflake.log (same dir as client) so we can see how far it gets? The pt should make a domain-fronted request to, https://snowflake-reg.appspot.com/ with an sdp offer and receive an answer, then try to negotiate a session with a snowflake (proxy) using, stun:stun.l.google.com:19302
On 26 Jan 2016, at 12:47, Arlo Breault <arlo@torproject.org> wrote:
I get about this far on OS X, I'm behind a NAT:
Jan 26 12:25:50.063 [notice] Tor v0.2.7.6 running on Darwin with Libevent 2.0.22-stable, OpenSSL 1.0.2e and Zlib 1.2.8. … Jan 26 12:25:50.071 [notice] Opening Socks listener on 127.0.0.1:9050 Jan 26 12:25:50.000 [notice] Parsing GEOIP IPv4 file /opt/local/share/tor/geoip. Jan 26 12:25:50.000 [notice] Parsing GEOIP IPv6 file /opt/local/share/tor/geoip6. Jan 26 12:25:50.000 [notice] Bootstrapped 0%: Starting Jan 26 12:25:50.000 [notice] Delaying directory fetches: No running bridges Jan 26 12:25:52.000 [notice] Bootstrapped 5%: Connecting to directory server Jan 26 12:25:52.000 [notice] Bootstrapped 10%: Finishing handshake with directory server
Any hints?
Can you paste a sanitized version of your snowflake.log (same dir as client) so we can see how far it gets?
The pt should make a domain-fronted request to, https://snowflake-reg.appspot.com/ with an sdp offer and receive an answer, then try to negotiate a session with a snowflake (proxy) using, stun:stun.l.google.com:19302
I'm connected using dual-stack IPv4 and IPv6, but I'm not sure if that's the issue. ----- 2016/01/26 12:23:15 starting 2016/01/26 12:23:18 OnNegotiationNeeded 2016/01/26 12:23:18 OnIceCandidate {"candidate":"candidate:4081377822 1 udp 2122262783 2001:... 49215 typ host generation 0 ufrag MhDVA7+9JyUZKlqY","sdpMid":"data","sdpMLineIndex":0} 2016/01/26 12:23:18 OnIceCandidate {"candidate":"candidate:44203497 1 udp 2122194687 10…. 60053 typ host generation 0 ufrag MhDVA7+9JyUZKlqY","sdpMid":"data","sdpMLineIndex":0} 2016/01/26 12:23:18 OnIceCandidate {"candidate":"candidate:3183943406 1 tcp 1518283007 2001:... 64926 typ host tcptype passive generation 0 ufrag MhDVA7+9JyUZKlqY","sdpMid":"data","sdpMLineIndex":0} 2016/01/26 12:23:18 OnIceCandidate {"candidate":"candidate:1277264153 1 tcp 1518214911 10…. 64927 typ host tcptype passive generation 0 ufrag MhDVA7+9JyUZKlqY","sdpMid":"data","sdpMLineIndex":0} 2016/01/26 12:23:21 OnIceCandidate {"candidate":"candidate:1036294592 1 udp 1685987071 180... 60053 typ srflx raddr 10…. rport 60053 generation 0 ufrag MhDVA7+9JyUZKlqY","sdpMid":"data","sdpMLineIndex":0} 2016/01/26 12:23:21 OnIceComplete 2016/01/26 12:23:21 ---------------- {"type":"offer","sdp":"v=0\r\no=- 8307480261278365747 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=msid-semantic: WMS\r\nm=application 60053 DTLS/SCTP 5000\r\nc=IN IP4 180….\r\na=candidate:4081377822 1 udp 2122262783 2001:... 49215 typ host generation 0\r\na=candidate:44203497 1 udp 2122194687 10…. 60053 typ host generation 0\r\na=candidate:3183943406 1 tcp 1518283007 2001:... 64926 typ host tcptype passive generation 0\r\na=candidate:1277264153 1 tcp 1518214911 10…. 64927 typ host tcptype passive generation 0\r\na=candidate:1036294592 1 udp 1685987071 180…. 60053 typ srflx raddr 10…. rport 60053 generation 0\r\na=ice-ufrag:MhDVA7+9JyUZKlqY\r\na=ice-pwd:8P3MROhYGYJJ6PBD4sxh5Hs6\r\na=fingerprint:sha-256 6A:2F:9D:5D:DF:83:9C:0A:2F:73:CF:1E:2D:AF:FF:F4:05:08:C8:C5:30:37:1C:A0:CE:C8:C6:1A:D0:E0:12:44\r\na=setup:actpass\r\na=mid:data\r\na=sctpmap:5000 webrtc-datachannel 1024\r\n"} 2016/01/26 12:23:21 ---------------- 2016/01/26 12:23:21 waiting for answer 2016/01/26 12:23:21 Sending offer via meek channel... Target URL: https://snowflake-reg.appspot.com/ Front URL: www.google.com 2016/01/26 12:23:25 MeekChannel Response: 200 OK 2016/01/26 12:23:25 Received Answer: v=0 o=- 1195921414501541321 2 IN IP4 127.0.0.1 s=- t=0 0 a=msid-semantic: WMS m=application 48952 DTLS/SCTP 5000 c=IN IP4 199.241.201.138 b=AS:30 a=candidate:2885715399 1 udp 2122260223 192.168…. 48952 typ host generation 0 a=candidate:3800267063 1 tcp 1518280447 192.168…. 0 typ host tcptype active generation 0 a=candidate:759726963 1 udp 1686052607 199... 48952 typ srflx raddr 192.168…. rport 48952 generation 0 a=ice-ufrag:gW3Squmad22xQeoQ a=ice-pwd:OAGHWixl0ZICWg2JYTXOri3W a=fingerprint:sha-256 A5:1A:FC:85:52:BE:D6:68:08:52:BE:66:B7:84:08:45:0F:63:47:70:1E:4F:E1:7D:1A:EC:67:47:D4:74:D3:07 a=setup:active a=mid:data a=sctpmap:5000 webrtc-datachannel 1024 2016/01/26 12:25:48 WebRTC: interrupted ----- Tim Wilson-Brown (teor) teor2345 at gmail dot com PGP 968F094B teor at blah dot im OTR CAD08081 9755866D 89E2A06F E3558B7F B5A9D14F
On Mon, Jan 25, 2016 at 02:34:42PM -0800, Serene wrote:
Snowflake is a webrtc pluggable transport inspired by flashproxy. (https://gitweb.torproject.org/pluggable-transports/snowflake.git) Arlo, David, and I have made lots of progress on it lately, and it now appears to have reached minimum viability.
I started a wiki page. https://trac.torproject.org/projects/tor/wiki/doc/Snowflake
Serene wrote:
Q: Why is it called Snowflake? There's a bunch of "ICE" negotiation happening for WebRTC, and it also involves a great abundance of ephemeral and short-lived (and special!) volunteer proxies...
Anyhow, if Snowflake seems like it would be useful / desired here, it would be awesome if we had more help getting it stable, polished, audited, deployable, etc... Plenty of work to do!
This is really great work, Serene ^_^ Once it is a bit more stable (and perhaps audited!), I'd be happy to incorporate Snowflake into Cupcake if that's useful. I am curious why you chose CoffeeScript for the proxy, rather than JavaScript. woot, Griffin -- “I did then what I knew then, & when I knew better, I did better.” ― Maya Angelou
On Mon, Jan 25, 2016 at 5:55 PM, Tim Wilson-Brown - teor <teor2345@gmail.com> wrote:
I'm connected using dual-stack IPv4 and IPv6, but I'm not sure if that's the issue. ----- [snip] 2016/01/26 12:23:21 Sending offer via meek channel... Target URL: https://snowflake-reg.appspot.com/ Front URL: www.google.com 2016/01/26 12:23:25 MeekChannel Response: 200 OK
2016/01/26 12:23:25 Received Answer: [snip]
Since the offer & answer exchanged successfully, I'm fairly certain this failed during ICE negotiation. In most cases, the public ip candidates given by STUN servers are sufficient for peers to negotiate a working p2p path. But sometimes, maybe 14% of the time [1], this isn't enough -- usually due to symmetric NAT / lack of port preservation, and a TURN relay is required as a last resort. I haven't yet configured TURN in either the client or the proxy, so this seems the most likely cause. I can update that to see if it works for you. We should probably setup some sort of Snowflake test suite for various NAT topologies... Later on, this may mean more complications for that fraction of users (TURN relays could get blocked in filtered zones). We do have some plan to inform clients of temporary TURN servers through the domain fronted signalling channel eventually, which may help. Also, once snowflake proxies & clients are multiplexed, perhaps the likelihood that TURN continuous to be absolutely required for every pair of peers greatly decreases and we might be fine again. More thought is needed here. [1] old numbers from http://webrtcstats.com
On Mon, Jan 25, 2016 at 9:15 PM, Griffin Boyce <griffin@cryptolab.net> wrote:
This is really great work, Serene ^_^ Once it is a bit more stable (and perhaps audited!), I'd be happy to incorporate Snowflake into Cupcake if that's useful.
Thanks Griffin! ^_^ That would indeed be useful and wonderful -- let's definitely make Snowflake frosted Cupcakes!
I am curious why you chose CoffeeScript for the proxy, rather than JavaScript.
I happened to be familiar with Coffeescript, enjoy its structures and idioms, and find I'm much quicker & productive writing in it. It's reasonably debuggable in-browser, and the generated javascript, though not "pretty", is very "correct" - so it seemed a fun choice to me :) ~s
Cool! Does the server (hosting a bridge) work also out of the box behind NAT? Cheers, Patrick
Hi Patrick -- yes, they're just WebRTC peers, which automatically and easily traverse NATs in most cases. "Hosting a bridge" for snowflake can be accomplished by leaving a tab open in your browser (or later on, running an extension) Cheers, ~serene On Tue, Mar 15, 2016 at 2:25 PM, Patrick Schleizer <patrick-mailinglists@whonix.org> wrote:
Cool!
Does the server (hosting a bridge) work also out of the box behind NAT?
Cheers, Patrick _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Can a snowflake bridge also be hosted by running from command line only without an open browser? If it could be dumbed down for the integrator up to a "sudo apt-get install snowflage-bridge", this would open up for huge opportunities getting more bridges. Freedombox and related box projects could pre-install it. Tails and Whonix users could optionally install the package and therefore easily start contributing without asking something which is impossible for most users for all practical purposes - setting up a port forwarding. Cheers, Patrick Serene:
Hi Patrick -- yes, they're just WebRTC peers, which automatically and easily traverse NATs in most cases. "Hosting a bridge" for snowflake can be accomplished by leaving a tab open in your browser (or later on, running an extension)
Cheers, ~serene
On Tue, Mar 15, 2016 at 2:25 PM, Patrick Schleizer <patrick-mailinglists@whonix.org> wrote:
Cool!
Does the server (hosting a bridge) work also out of the box behind NAT?
Cheers, Patrick _______________________________________________ tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
tor-dev mailing list tor-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
Can a snowflake bridge also be hosted by running from command line only without an open browser?
Not yet, but I opened a ticket for it, https://github.com/keroserene/snowflake/issues/30
participants (8)
-
Arlo Breault -
David Fifield -
Griffin Boyce -
Patrick Schleizer -
Serene -
Serene -
Tim Wilson-Brown - teor -
Yawning Angel