[tor-talk] Pluggable transports for other projects...

Thu Jan 30 17:55:48 UTC 2014

TheMindwareGroup <themindwaregroup at gmail.com> writes:

> Bypassing DPI filters is a constantly evolving art form and entire
> field of research all on its own, and just like encryption extremely
> difficult to do well (check out bit torrent and emule obfuscation for
> example, a nice effort but when scrutinized not particularily effective).
>
> It would be nice if pluggable transports stenographic tunnels where
> easily available for other projects to use also maybe with an easy
> socks proxy interface like Tor uses or an api interface (dont know how
> you have implemented it).
>
> I am working on my own project but most of my effort is on getting the
> core up and running, I could throw in some transformation but it isnt
> going to be anywhere near as good as what Fornat Transforming
> Encryption or stegotorus could provide. Its the same problem with most
> programs getting over DPI firewalls is always gonna be a big problem.
>
> If you could post a page about how it works and how to use it on you
> site, specifically how to interface with it, it would be most helpful
> for other developers.
>
> ~Shadowman
>
> ~TheMindwareGroup
> TheMindwareGroup at gmail.com PGP: 0xf4b6586f

Greetings Shadowman,

I admit that the documentation resources on how to integrate PTs on
your project are not particularly good. This is mainly because not
many projects had expressed interest in the past and we had more
urgent things to do. Recently a few projects have asked us for help on
integrating PTs into their system. This is a good sign but it also
means that we should improve our documentation. A related trac ticket
is https://trac.torproject.org/projects/tor/ticket/10629

For starters, I suggest you read
https://gitweb.torproject.org/torspec.git/blob/HEAD:/pt-spec.txt .
This will give you an idea of how Tor uses pluggable transports. That
document can be vastly improved and I plan to do it "soon". Till then,
patches are more than welcome (I especially need to add some sections
and structure to that document, since it doesn't have a nice flow.)

Now, let me give you a small description of how Tor manages its
pluggable transports. It will not be ultra robust, but it might help
you understand what's going on:

Client side:

0) The user is supposed to use the torrc configuration file to inform
   Tor about bridges, their pluggable transports and their
   IP:port. This happens using the Bridge and ClientTransportPlugin
   torrc directives (see the man page, the pt-spec or the source
   code).

1) Tor now knows which bridges the user wants to connect to and which
   pluggable transports it needs to use. It gets the filepath of the
   pluggable transports from the ClientTransportPlugin line and gets
   ready to start them up.

   Before starting the PT, Tor needs to prepare some managerial
   information that the PT needs to know. For example, the directory
   that the PT should use if it needs to store permanent state, which
   transports the PT application should initialize (if it supports
   more than one transport) etc. To do so, Tor sets up the TOR_PT_*
   environment variables accordingly. You can find more info about
   those in the pt-spec.txt.

   After Tor prepares the environment variables, it fires up the PT
   (using execve() or whatever).

2) The PT application now parses the environment variables and
   initializes the PTs that Tor asked it to use. Each PT starts up a
   SOCKS listener that Tor is supposed to connect to. After the
   initialization has been done, the PT sends a 'CMETHOD' line to
   stdout that informs Tor of the IP:port that the PT SOCKS listener
   is listening on, the SOCKS version that it uses etc. You can read
   more about CMETHOD lines in pt-spec.

   We've developed helper libraries that manage this step for you,
   check out pyptlib, goptlib and liballium [0]. For example, you
   might want to skim over this:
   http://pyptlib.readthedocs.org/en/latest/API.html#general-overview

3) Tor now knows which SOCKS listener it should connect to for each
   PT. When Tor wants to connect to a bridge that uses a PT, it does a
   SOCKS CONNECT request to the appropriate SOCKS listener and asks it
   to CONNECT to the destination (the destination should of course be
   the server-side PT IP:port).

   In case the PT requires further parameters for each connection
   (e.g. if the server-side PT protocol requires a shared-secret) the
   SOCKS handshake is (ab)used to transfer those parameters to the
   PT. Currently we are using the username/password fields of the
   SOCKS handshake to transfer those parameters, but we are not very
   happy about this. For more info see:
   https://trac.torproject.org/projects/tor/ticket/10671

4) When the PT receives a CONNECT request to its SOCKS listener, it
   connects to the destination and starts relaying traffic between Tor
   and the destination. Specifically, everytime the PT receives
   application-layer traffic from Tor, it obfuscates it appropriately,
   and relays it to the destination.

That's how the client side works. Now let's look at the server (bridge) side.

Server side:

0) The bridge operator is supposed to use the torrc configuration file
   to inform Tor about the PTs it wants to support in her bridge. This
   happens using the ServerTransportPlugin torrc directive (again see
   the man page, the pt-spec or the source code).

1) Tor, again, prepares the environment variables that need to be
   passed to the PT and spawns it.

   In the server-side there is even more information that Tor needs to
   pass to the PT. Take a look at the server-only TOR_PT_* environment
   variables in pt-spec.txt. For example, Tor needs to instruct the PT
   to bind on a specific IP:port, so that the PT listener stays in the
   same place even after Tor reboots (Tor saves the IP:port in its
   state file).

2) The PT, again, parses the environment variables, fires up its
   listeners and informs Tor of its IP:port using SMETHOD lines in
   stdout.

   The pyptlib/goptlib/liballium libraries help with this step too.

3) When the PT receives traffic from a client, it deobfuscates it and
   passes it to Tor.

   In the server side, the PT uses a special protocol to communicate
   with Tor. It's called the "Extended ORPort" protocol and its
   basically a round of metadata transfer before the actual
   application-layer traffic are forwarded to Tor. This metadata
   includes the actual IP address of the client and the name of the PT
   used. We use this information to keep statistics about the clients
   that each bridge sees.

   The Extended ORPort protocol is specified here:
   https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/196-transport-control-ports.txt
   https://gitweb.torproject.org/torspec.git/blob/HEAD:/proposals/217-ext-orport-auth.txt
   Ignore the TransportControlPort part of proposal 196, as that's not
   implemented. Also, don't ask me why this stuff are not in
   pt-spec.txt because I don't know; we should fix this.

   For the extended OR port, you may also want to look at
   some existing source code:
   https://gitweb.torproject.org/pluggable-transports/obfsproxy.git/blob/b88efc0ce3f145d5e767d5b938889e99b47021b7:/obfsproxy/network/extended_orport.py
   https://gitweb.torproject.org/pluggable-transports/goptlib.git/blob/abeea884f554b4119ebd84974c612c0dca6ce941:/pt.go#l581

So... this is how PTs work with Tor. Parts of it work like this
because we wanted PTs to work transparently; so that the end user
doesn't need to care at all about how PTs work and why they work like
that.

If you don't need all of this, you might enjoy the simpler "external
mode" functionality of PTs, as is specified in pt-spec.txt.

I already wrote a bit too much and I'm not really sure what you are
actually interested in so I'll shut up and wait for any questions you
might have.

Also, feel free to drop by #tor-dev at the OFTC IRC network for a more
synchronous communication experience. We also have biweekly PT
meetings in IRC every second Friday. Tomorrow is one of those Fridays
and we are going to gather there at 17:00 UTC. Feel free to join.

(BTW, I've been meaning to write a "How to integrate PTs with your
project" document for a while. I'll probably reuse parts of this post
when I do so.)

Cheers!

[0]: https://trac.torproject.org/projects/tor/wiki/doc/PluggableTransports#OtherPTsoftware