- tor-dev - lists.torproject.org

Proposal 229: Further SOCKS5 extensions
by Nick Mathewson 28 Feb '14

28 Feb '14

On Fri, Feb 28, 2014 at 3:53 AM, Yawning Angel <yawning(a)schwanenlied.me> wrote: > Hello all, > > Based on feedback from nickm/arma, I have extended the proposal to also > cover Tor's SOCKS port, including response codes related to hidden > service status. > > Additionally, instead of having a username/password field pre-defined, > the extended authentication method has been generalized to contain > key/value pairs for extra flexibility in the future. > > Questions, comments, feedback appreciated as always, This is now proposal 229! (I've wrapped the lines a little, but made no additional changes.) After adding it to Git, I made some redundancy tweaks and asked a bunch of annoying questions that we'll probably want to figure out before implementing anything like this. Filename: 229-further-socks5-extensions.txt Title: Further SOCKS5 extensions Author: Yawning Angel Created: 25-Feb-2014 Status: Draft 0. Abstract We propose extending the SOCKS5 protocol to allow passing more per-session metadata, and to allow returning more meaningful response failure codes back to the client. 1. Introduction The SOCKS5 protocol is used by Tor both as the primary interface for applications to transfer data, and as the interface by which Tor communicates with pluggable transport implementations. While the current specifications allow for passing a limited amount of per-session metadata via hijacking the Username/Password authentication method fields, this solution is limited in that the amount of payload that can be conveyed is restricted to 510 bytes, does not allow the SOCKS server to return a response, and precludes using authentication on the SOCKS port. The first part of this proposal defines a new authentication method to overcome both of these limitations. The second part of this proposal defines a range of SOCKS5 response codes that can be used to signal Tor specific error conditions when processing SOCKS requests. 2. Proposal 2.1. Tor Extended SOCKS5 Authentication We introduce a new authentication method to the SOCKS5 protocol. The METHOD number to be returned to indicate support for or select this method is X'97', which belongs to the "RESERVED FOR PRIVATE METHODS" range in RFC 1928. After the authentication method has been negotiated following the standard SOCKS5 protocol, the actual authentication phase begins. If any requirement labeled with a "MUST" below in this protocol is violated, the party receiving the violation MUST close the connection. All multibyte numeric values in this protocol MUST be transmitted in network (big-endian) byte order. The initiator will send an Extended Authentication request: +----+----------+-------+-------------+-------+-------------+--- |VER | NR PAIRS | KLEN1 | KEY1 | VLEN1 | VALUE1 | ... +----+----------+-------+-------------+-------+-------------+--- | 1 | 2 | 2 | KLEN1 bytes | 2 | VLEN1 bytes | ... +----+----------+-------+-------------+-------+-------------+--- VER: 8 bits (unsigned integer) This field specifies the version of the authentication method. It MUST be set to X'01'. NR PAIRS: 16 bits (unsigned integer) This field specifies the number of key/value pairs to follow. KLEN: 16 bits (unsigned integer) This field specifies the length of the key in bytes. It MUST be greater than 0. KEY: variable length This field contains the key associated with the subsequent VALUE field as an ASCII string, without a NUL terminator. VLEN: 16 bits (unsigned integer) This field specifies the length of the value in bytes. It MAY be X'0000', in which case the corresponding VALUE field is omitted. VALUE: variable length, optional The value corresponding to the KEY. The responder will verify the contents of the Extended Authentication request and send the following response: +----+--------+----------+-------+-------------+-------+-------------+--- |VER | STATUS | NR PAIRS | KLEN1 | KEY1 | VLEN1 | VALUE1 | ... +----+--------+----------+-------+-------------+-------+-------------+--- | 1 | 1 | 2 | 2 | KLEN1 bytes | 2 | VLEN1 bytes | ... +----+--------+----------+-------+-------------+-------+-------------+--- VER: 8 bits (unsigned integer) This field specifies the version of the authentication method. It MUST be set to X'01'. STATUS: 8 bits (unsigned integer) The status of the Extended Authentication request where: * X'00' SUCCESS * X'01' AUTHENTICATION FAILED * X'02' INVALID ARGUMENTS If a server sends a response indicating failure (STATUS value other than X'00') it MUST close the connection. [XXXX What should a client if it gets a value here it does not recognize?] NR PAIRS, KLEN, KEY, VLEN, VALUE: These fields have the same format as they do in Extended Authentication requests. The currently defined KEYs are: * "USERNAME" The username for authentication. * "PASSWD" The password for authentication. [XXXX What do these do? What is their behavior? Are they client-only? Right now, Tor uses SOCKS5 usernames and passwords in two ways: 1) as a way to control isolation, when receiving them from a SOCKS client. 2) as a way to encode arbitrary data, when sending data to a PT. Neither of these seem necessary any more. We can turn 1 into a new KEY, and we can turn 2 into a new set of keys. -NM] [XXX - Add some more here, Stream isolation? -YA] [XXXX What should a client if it gets a value here it does not recognize? -NM] [XXXX Should we recommend any namespace conventions for these? -NM] 2.2. Tor Extended SOCKS5 Reply Codes We introduce the following additional SOCKS5 reply codes to be sent in the REP field of a SOCKS5 message. Implementations MUST NOT send any of the extended codes unless the initiator has indicated that it understands the "Tor Extended SOCKS5 Authentication" as part of the version identifier/method selection SOCKS5 message. [Actually, should this perhaps be controlled by additional KEY? (I'm not sure.) -NM] Where: * X'E0' Hidden Service Not Found The requested Tor Hidden Service was not reachable. * X'E1' Hidden Service Not Reachable The requested Tor Hidden Service was not found. * X'F0' Temporary Pluggable Transport failure, retry immediately Pluggable transports SHOULD return this status code if the connection attempt failed, but the pluggable transport believes that subsequent connections with the same parameters are likely to succeed. Example: The ScrambleSuit Session Ticket handshake failed, but reconnecting is likely to succeed as it will use the UniformDH handshake. * X'F1' Pluggable transport protocol failure, invalid bridge Pluggable transports MUST return this status code if the connection attempt failed in a manner that indicates that the remote peer is not likely to accept connections at a later time. Example: The obfs3 handshake failed. * X'F2' Pluggable transport internal error Pluggable transports SHOULD return this status code if the connection attempt failed due to an internal error in the pluggable transport implementation. Tor might wish to restart the pluggable transport executable, or retry after a delay. 3. Compatibility SOCKS5 negotiates authentication methods so backward and forward compatibility is obtained for free, assuming a non-broken SOCKS5 implementation on the responder side that ignores unrecognised authentication methods in the negotiation phase. 4. Security Considerations Identical security considerations to RFC 1929 Username/Password authentication applies when doing Username/Password authentication using the keys reserved for such. As SOCKS5 is sent in cleartext, this extension (like the rest of the SOCKS5 protocol) MUST NOT be used in scenarios where sniffing is possible. The authors of this proposal note that binding any of the Tor (and associated) SOCKS5 servers to non-loopback interfaces is strongly discouraged currently, so in the current model this is believed to be acceptable. 5. References Leech, M., Ganis, M., Lee, Y., Kuris, R., Koblas, D., Jones L., "SOCKS Protocol Version 5", RFC 1928, March 1996. Tor Project, "Tor's extensions to the SOCKS protocol" Leech, M. "Username/Password Authentication for SOCKS V5", RFC 1929, March 1996. Appelbaum, J., Mathewson, N., "Pluggable Transport Specification", June 2012. [XXX - Changelog (Remove when accepted) -YA] 2014-02-28 (Thanks to nickm/arma) * Generalize to also support tor * Add status codes for bug #6031 * Switch from always having a username/password field to making them just predefined keys. * Change the auth method number to 0x97 2014-02-28 (nickm's fault) * check it into git * clean text a little, fix redundancy * ask some questions

1 0

Pluggable transports meeting today (Friday 28th of February 2014)
by George Kadianakis 28 Feb '14

28 Feb '14

Greetings humans, this is an email to remind you that the regular biweekly pluggable transports meeting is going to happen today. Place is the #tor-dev IRC channel in OFTC. Time is 17:00 UTC (it's in an hour!). Cheers!

1 0

Translation priority
by Colin C. 28 Feb '14

28 Feb '14

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi everyone, Our contact at OpenITP has requested a list of resources / strings that we would like to potentially give priority to. If anyone has a project that is currently on our Transifex (or one that isn't and should be), and would like to see certain things translated with priority, please let me know and I will pass that information along. Thanks! - -- - -Phoul -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iQIcBAEBCAAGBQJTEE0BAAoJEJ01Zvx7KO7YSuMP/A1YvtZyOi+eb2vW0xqS1/+L YhffGKah5aGD8jEeAU/7kO7UQwOjOkTo7gCJGGtufIHcHDCxEqY2E6NyH0yhVQ1c qKMy6tZ/CL22ddSV0C2qz073R+YJ5oGjUM3rpYk7FdyxfvHBtbeb9GA/oYRxZGoQ QutNWFtl1C/UYzvBD5pGybH2dNHeHEHnCmCnMI4R9sWyBSneg/GasOwrGuNq+oKC Kc+ncfPs6C5/qqTdpgnd62EFE9LIYfChYP98FtQo8+azAcIWolaqM34j8F0IidQV yXIhWZLU7tU+efWwqAPTN7cPzIjKoBOatV1QkT/LRqhupDM27CBEm9ghGfQ1lszt nVG5GOMeobCnHJahGQiqyfngE/EcjkakOlFpygC7hfcjoiMhqdkt/r7Ar1euI9Oo HND6UvGJLP+CuRFfwAt31I5BMEHuqr0rPN3RZ8lu7QAl/L0sQDH8BQfAqoCjPurm SBq6GNbufQxzkvF4BY4c1Dw4ij6muwaV8VJBNulMVrEq5a857MD94FEm1qLlZ2OL z5p9uxfMfwWXQWVe4b8dKG2MhjphrtlFp+6zGPTYt5CN70B+F8gnVqkBrmgojoKl KnIlaULfuQBOI7E8VEge+iOAGnFUfQ5+s9hiRv82sxy0N9dx63t4Ce3U6m71dgUN CGwOz8wow4UfD0Cvdj1K =LUYK -----END PGP SIGNATURE-----

1 0

GSOC14 Idea
by Devang Thakkar 27 Feb '14

27 Feb '14

Hi, Its Devang here, a coding enthusiast studying at IIT Bombay. I am looking forward to contribute to Tor for the upcoming Google Summer of Code 2014 as a prospective student. So I wanted to know if there was a provision for Web Scraping using Tor. If there is, I would to know more about it or if there isn't, is it a feasible Summer of Code project? Devang Thakkar, First Year Undergraduate, Indian Institute of Technology, Bombay.

4 3

GSOC Proposal - Improve the testing framework Chutney
by Punit 27 Feb '14

27 Feb '14

Hello everyone, I would like to contribute to Tor as a part of my GSOC Project this year. I am mainly interested in improving the testing frameworks, since (from what information I have gathered) they require much work to be done (especially Chutney). This is with reference to the mail archive links http://archives.seul.org/or/dev/Apr-2013/msg00152.html http://archives.seul.org/or/dev/Apr-2013/msg00159.html and the Volunteer page project list - https://www.torproject.org/getinvolved/volunteer.html.en#chutneyExpansion In the mail, it is mentioned that Chutney needs to be more robust and has to have capabilities to actually test the network it launches. Right now, Chutney is able to launch a local Tor network, but much work is needed for actual tests. We need to feed data using some other program, and have to go through the log files to see if the network is running without errors. I think that Chutney can be turned into a great tool for testing Tor (for example, after every commit) and analyse how the Tor network works after the commit. What is needed is a way to prepare some Test cases (and to provide a simple interface for non programmers to make new Tests). Also, we need an integrated setup for running the network, pumping in data, run test cases, and show some statistics as to how the Tor network fared. I would like to know your views on the Proposal. Thanks, Punit

2 1

Any Tor-ish papers you'd like to nominate for the PET Award?
by Ian Goldberg 26 Feb '14

26 Feb '14

Is there a Tor-ish paper you'd like to nominate for the 2014 PET Award? See the CFN below. Thanks, - Ian Call for Nominations [Please forward and distribute] You are invited to submit nominations to the 2014 PET Award. The PET Award is presented annually to researchers who have made an outstanding contribution to the theory, design, implementation, or deployment of privacy enhancing technology. It is awarded at the annual Privacy Enhancing Technologies Symposium (PETS). The PET Award carries a prize of 3000 USD thanks to the generous support of Microsoft. The crystal prize itself is offered by the Office of the Information and Privacy Commissioner of Ontario, Canada. Any paper by any author written in the area of privacy enhancing technologies is eligible for nomination. However, the paper must have appeared in a refereed journal, conference, or workshop with proceedings published in the period from April 1, 2012 until March 30, 2014. The complete Award rules including eligibility requirements can be found at http://petsymposium.org/award/rules.php. Anyone can nominate a paper by sending an email message containing the following to award-chairs14(a)petsymposium.org: * Paper title * Author(s) * Author(s) contact information * Publication venue and full reference * Link to an available online version of the paper * A nomination statement of no more than 500 words. All nominations must be submitted by March 30, 2014. The Award Committee will select one or two winners among the nominations received. Winners must be present at the PET Symposium in order to receive the Award. This requirement can be waived only at the discretion of the PET Advisory board. More information about the PET award (including past winners) is available at http://petsymposium.org/award/ More information about the 2014 PET Symposium is available at http://petsymposium.org/2014.

1 0

Tor in Google Summer of Code 2014
by Damian Johnson 26 Feb '14

26 Feb '14

Interested in coding on Tor and getting paid for it by Google? If you are a student, we have good news for you: We have been accepted as a mentoring organization for Google Summer of Code 2014! https://www.google-melange.com/gsoc/homepage/google/gsoc2014 https://www.google-melange.com/gsoc/org2/google/gsoc2014/tor Here are the facts: The summer of code gives you the opportunity to work on your own Tor-related coding project with one of the Tor developers as your mentor. You can apply for a coding project related to Tor itself or to one of its many supplemental projects... https://www.torproject.org/getinvolved/volunteer.html.en#Projects Your mentor will help you when you're stuck with your project and guide you in becoming part of the Tor community. Google pays you 5,000 USD for the three months of your project, so that you can focus on coding and don't have to worry about how to pay your bills. Did we catch your attention? These are your next steps: Go look at the Google Summer of Code FAQ [1] to make sure you are eligible to participate. Have a look at our ideas list [2] to see if one of those projects matches your interests. If there is no project on that list that you'd want to work on, read the documentation on our website [3] and make up your own project idea! Come to the tor-dev@ list [4] or #tor-dev on OFTC [5] and let us know about your project idea. Communication is essential to success in the summer of code, and we're unlikely to accept students we haven't heard from before reading their application. So really, come to the list or IRC channel and talk to us! Finally, write down your project idea using our template [6] and submit your application to Google before March 21 [7]. We are looking forward to discussing your project idea with you! [1] https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2… [2] https://www.torproject.org/about/gsoc.html.en#Ideas [3] https://www.torproject.org/docs/documentation.html.en#UpToSpeed [4] https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev/ [5] https://www.torproject.org/about/contact.html.en#irc [6] https://www.torproject.org/about/gsoc.html.en#Template [7] https://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2…

1 0

Using MaxMind's GeoIP2 databases in tor, BridgeDB, metrics-*, Onionoo, etc.
by Karsten Loesing 26 Feb '14

26 Feb '14

Hi devs, you probably know that we use MaxMind's GeoIP database in various of our products (list may not be exhaustive): - tor: We ship little-t-tor with a geoip and a geoip6 file for clients to support excluding relays by country code and for relays to generate by-country statistics. - BridgeDB: I vaguely recall that the BridgeDB service uses GeoIP data to return only bridges that are not blocked in a user's country. Or maybe that was a feature yet to be implemented. - Onionoo: The Onionoo service uses MaxMind's city database to provide location information of relays. (It also uses MaxMind's ASN database to provide information on AS number and name.) - metrics-db: I'm planning to use GeoIP data to resolve bridge IP addresses to country codes in the bridge descriptor sanitizing process. - metrics-web: We have been using GeoIP data to provide statistics on relays by country. This is currently disabled because the implementation was eating too many resources, but I plan to put these statistics back. However, the GeoIP database that we currently use has a big shortcoming: it replaces valid country codes with A1 or A2 whenever MaxMind thinks that a relay is an "anonymizing proxy" or "satellite provider". That's why we currently repair their database by either automatically guessing what country code an A1 entry could have had [1, 2], or by manually looking it up in RIR delegation files [3, 4]. This is just a workaround. Also, I think BridgeDB doesn't repair its GeoIP database. Here's the good news: MaxMind now provides their databases in new formats which provide the A1/A2 information in *addition* to the correct country codes [5, 6]. We should switch! How do we switch? First option is to ship their binary database files and include their APIs [7] in our products. Looks there are APIs for C, Java, and Python, so all the languages we need for the tools listed above. Pros: we can kick out our parsing and lookup code. Cons: we need to check if their licenses are compatible, we have to kick out our parsing and lookup code and learn their APIs, and we add new dependencies. Another option is to write a new tool that parses their full databases and converts them into file formats we already support. (This would also allow us to provide a custom format with multiple database versions which would be pretty useful for metrics, see #6471.) Also, it looks like their license, Creative Commons Attribution-ShareAlike 3.0 Unported, allows converting their database to a different format. If we want to write such a tool, we have a few options: - We use their database specification [8] and write our own parser using a language of our choice (read: whoever writes it pretty much decides). We could skip the binary search tree part of their files and only process the contents. Whenever they change their format, we'll have to adapt. - We use their Python API [9] to build our parser, though it looks like that requires pip or easy_install and compiling their C API. I don't know enough about Python to assess what headaches that's going to cause. - We use their Java API [10] to build our parser, though we're probably forced to use Maven rather than Ant. I don't have much experience with Maven. Also, using Java probably makes me the default (and only) maintainer, which I'd want to avoid if possible. Thoughts? What other options did I miss, and what pros and cons that I overlooked? And is this something that people on this list would want to help with, once we agreed on one of the options? If so, please feel free to join the discussion now and maybe influence which path we're going to take. All the best, Karsten [1] https://gitweb.torproject.org/tor.git/blob/HEAD:/src/config/deanonymind.py [2] https://gitweb.torproject.org/onionoo.git/blob/HEAD:/geoip/deanonymind.py [3] https://gitweb.torproject.org/tor.git/blob/HEAD:/src/config/geoip-manual [4] https://gitweb.torproject.org/onionoo.git/blob/HEAD:/geoip/geoip-manual [5] http://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/ [6] http://dev.maxmind.com/geoip/geoip2/geolite2/ [7] http://dev.maxmind.com/geoip/geoip2/downloadable/ [8] https://github.com/marklr/MaxMind-IPDB-perl/blob/master/docs/MaxMind-IPDB-s… [9] https://pypi.python.org/pypi/geoip2 [10] https://github.com/maxmind/MaxMind-DB-Reader-java

2 4

Tor Browser Weekly IRC meetings at 19:00 UTC Wednesdays
by Mike Perry 26 Feb '14

26 Feb '14

At the dev meting, we agreed to hold weekly TBB meetings in #tor-dev on IRC (irc.oftc.net) ideally either before or after the tor-core meetings. It looks like we're going to shoot for Wednesdays at 11:00am PST/19:00 UTC. If we can agree on this time, Nick has said that the tor-core meetings may be able to follow ours, pending approval from the rest of the Tor folks. For the meetings themselves, I think we should try the following format, inspired by Scrum, but less rigid. Everyone makes 0 or more statements on the following three topics, with time for responses from others: 1. What have you been working on last week that other TBB people should be aware of? 2. Will you be working on anything TBB related next week that should/could involve other people? 3. Do you need anything else from anyone else to help you make progress or to get something reviewed+merged? If so, what do you need? These meetings won't be used for progress metrics/evaluation, so don't feel the need to inflate things, explain already-solved snags in detail, discuss your various non-TBB/life distractions, or anything like that. The purpose here is strictly TBB coordination. In particular, if you've been working on stuff that either isn't TBB-related or is too premature to be worth discussing in detail yet, you can skip items 1 and 2 entirely, or make them extremely brief (ie "Worked on the updater"). Similarly, if you're blocked on stuff that doesn't require other people's help/input to solve, you can skip item 3. Don't feel obligated to tell us about issues that involving others won't really help. However, when discussion is necessary, we can and should deviate from the #1/#2/#3 pattern to discuss people's work/questions in some detail without plowing straight through to the next person immediately. After everyone has their turn to say what's they've been doing/what's needed, we can do a little bit of freeform discussion on whatever other issues seem pressing and who should be doing what next, discuss any future releases/deadlines, and discuss recent support issues if any support people are present. We will hard-stop at 20:00 UTC for the tor-core dev meeting to start (but hopefully will finish much earlier than that). If we end up consistently finishing early, we can make it later so as to run more fluidly into the tor-core meeting, for people who want to attend both. -- Mike Perry

2 2

GSoC Ideas
by Vighnesh Birodkar 25 Feb '14

25 Feb '14

Hello I am found a couple of ideas from the Ideas Page interesting . I was a GSoC student for SimpleCV last year. In the past I've programmed in C,C++,Java and Python . Following are my queries . 1. Search for Hidden Services . I apologize in advance if there is something obviously wrong with my idea. Dark Web consists of information that cannot be crawled because it doesn't appear as hyperlinks in other pages . But someone somewhere will always have access to this information, either by entering search queries , through subscriptions or logging in. What if we can index all the pages a browser visits ? Users can voluntarily install and enable or disable a plugin in their browsers . This plugin will index process ( and maybe index ) pages locally and upload it's data to servers which will hold the global index . 2. Develop a Censorship Analyzer Will this be a part of any existing tor projects ? What is a student required to do to be considered suitable for this ? Thanks Vighnesh

3 3