- tor-dev - lists.torproject.org

Damian's Status Report - June 2013
by Damian Johnson 27 Jun '13

27 Jun '13

Hi all. Without much ado here's my status report for June... * Stem: Migrated to the Mock module Our homemade mocking framework has served us well, but over time it taught me one very important lesson: writing a mocking framework is hard. On the surface it seems pretty simple: apply and revert a set of monkey patches. But how do you monkey patch class methods? What about alias imports like the os module? And god forbid you want to mock python's open() function. I'm finally taking a lesson … [View More]from one of my coworkers and using a library for this. Python has several options but the most common is PyPI's mock module, which became part of the standard library in Python 3.3... http://www.voidspace.org.uk/python/mock/ https://gitweb.torproject.org/stem.git/commitdiff/101cf0b?hp=65846e8 ======================================== * Arm: Pruning the torTools utility Arm's torTools module was a wrapper around TorCtl that provided caching, thread safety, and a better API. Now that we're using stem it's obsolete, and my aim is to completely eliminate it from the codebase. This is easier said than done however. This month I pruned vast swaths of the module, reducing it from 2768 lines to 1020... before: https://gitweb.torproject.org/arm.git/blob/refs/heads/release:/src/util/tor… after: https://gitweb.torproject.org/arm.git/blob/HEAD:/src/util/torTools.py Doing this required some expansions on stem's part. Added functionality includes... * get_pid() and get_user() methods in the Controller * system functions for getting a process' start time and FreeBSD jail path * the system module's call() wasn't respecting exit codes ======================================== * Stem: Remote descriptor fetching Throughout the month Karsten and I have been bouncing ideas back and fourth concerning a stem API for remote descriptor fetching... https://trac.torproject.org/projects/tor/wiki/doc/stem#RemoteDescriptorFetc… https://lists.torproject.org/pipermail/tor-dev/2013-June/004957.html I have some s3krit ideas for improving this that will address both our use cases even better (spoiler: future style instances). We're coming up on a four day weekend so hopefully I'll be able to start implementing this soon! ======================================== * Stem: Expanded FAQ with answers to common Stack Overflow questions I took a pass over the tor related questions on Stack Overflow (http://stackoverflow.com/questions/tagged/tor) answering fifteen that concerned controller scripting. The vast majority of those unfortunately were some variation of 'how do I programmatically change my IP?' which I answered with a Stem FAQ entry... https://stem.torproject.org/faq.html#how-do-i-request-a-new-identity-from-t… The only question I thought was especially interesting went along the lines of 'How do I check the IPs of the exits I'm presently using?'. I answered this with a FAQ entry too... http://stackoverflow.com/questions/9777192/how-do-i-get-the-tor-exit-node-i… https://stem.torproject.org/faq.html#how-do-i-get-information-about-my-exits ======================================== ... and a handful of smaller tasks... * Investigated the descriptor version provided via tor's 'GETINFO ns/*' options. Contrary to to the spec it turns out these have been v3 all along, and stem now parses them as such. (#7953) * Our automated Jenkins test runs caught their first instance of tor regression. This concerned LOADCONF's behavior after merging a branch for ticket #6752 (#9122). * Handful of GSoC tasks (welcome/sorry emails after acceptance was announced, and org admin prep for the program's start) * Added msg_type argument to ControlMessage.from_str() (request by meejah, #8605) * Investigated cache consistency issue (thanks to Ravi, #7713) * Fixes for ONLINE target (patch by Jeremy, #8692) * Minor revisions to how consensus bandwidth-weights are validated (#6872) * Addressed an arm issue with certain TERMs (#9064) * Answered some questions from Sreenatha that came up while migrating Tor Weather to Stem (https://lists.torproject.org/pipermail/tor-dev/2013-June/005035.html) Cheers! -Damian [View Less]

1 0

Re: [tor-dev] [tor-talk] Client simulation
by Karsten Loesing 27 Jun '13

27 Jun '13

(Sorry for cross-posting, but I think this is a topic for tor-dev@, not tor-talk@. If you agree, please reply on tor-dev@ only. tor-talk@ people can follow the thread here: https://lists.torproject.org/pipermail/tor-dev/2013-June/thread.html) On 6/6/13 7:32 PM, Norman Danner wrote: > I have two questions regarding a possible research project. > > First, the research question: can one use machine-learning techniques > to construct a model of Tor client behavior? Or in a more … [View More]general form: > can one use <fill-in-the-blank> to construct a model of Tor client > behavior? A student of mine did some work on this over the last year, > and the results are encouraging, though not strong enough to do anything > with yet. > > Second, the meta-question: is it worthwhile to answer the first > question? It seems to me that if the answer to the first question is > "yes," then the solution could be used to (at least) provide better > simulations of Tor (e.g., via Shadow or ExperimenTor). This possibly > naive thought would imply that the answer to the second question is "yes." > > I'd be interested to hear responses to my second question, either > validating my naive thought or explaining why the first question isn't > worth answering. I'd accept responses to my first question, too, in > case this has already been done. Hi Norman, yes, it's worthwhile to answer this question! I can imagine how at least Shadow and the Tor path generator would benefit from better client models. User number estimates on the metrics website might benefit from them, too. I found two tickets where we asked similar questions before, and maybe there are more tickets like these: https://trac.torproject.org/projects/tor/ticket/2963 https://trac.torproject.org/projects/tor/ticket/6295 Some very early thoughts: - How do we make sure that we ask a representative set of people to instrument their clients and export data on their usage behavior? If we only ask people who read their favorite news site twice per day, our client model will be just that, but not representative for all Tor users. (Still, we would know more than we know now.) - Can we somehow aggregate usage information enough to make it safe for people to send actual usage reports to us? I could imagine having a torrc flag that is disabled by default and that, when enabled, writes sanitized usage information to disk. For this we need a very good idea what we're planning to do with the data, and we'll need to specify the aggregation approach in a tech report and get it reviewed by the community. Are your student's results available somewhere? Best, Karsten [View Less]

2 5

XMPP Pluggable Transport
by Alex Eftimiades 25 Jun '13

25 Jun '13

I have been working on creating an XMPP pluggable transport for Tor for a couple of weeks now, and someone suggested I send an email to this mailing list for suggestions. At this point, my priorities are ordered as follows: 1. Speed--I can watch low quality youtube videos on a bad internet connection, but there is a lot of down time while it buffers (not an unbearable amount of time, but enough to be annoying). This has been a big problem for a while, and I could really use some … [View More]suggestions on how to speed it up. 2. Full GTalk compatibility. It seems you need to be on a recipient's contact list to send a message to them (at least without supplying a resource). However, I currently use messages to send connection requests to JIDs with unknown resources. Then when they reply, their resource is saved in a table. This technique works fine for everything but GTalk, which of course is by far the fastest of the chat servers I have tested. 3. Error handling--I need to make sure I take everything into account. I do not have anything that can reestablish connections if one computer disconnects from the chat server and reconnects. In fact, if it acquires new resources (which is what happens on GTalk), the other computers will be stuck sending messages to a JID with the wrong resource. I am pretty sure I can come up with some ways of dealing with this. Right now, looking for potential problems is just as important as pointing out solutions to more obvious ones. 4. Security. I know security is important, but the nature of the program just does not seem to admit many security threats. There are really just three that I can see: - The server can be told to connect to any ip:port by a client. - The server and client exchange their Python interpreter's sys.maxsize values (which could lead to profiling a computer's OS). The former can be fixed with a whiltelist and the latter is not even necessary (I just have not gotten around to tweaking that out of the protocol yet). - DoS. This would be relatively easy to take into account by limiting the number of connections a bot is willing to accept. Then the chat server itself will limit the amount of data sent from any one JID (for better or worse). If you think you are able/willing to help, read on for a summery of the concept and protocol below. Even if you are not well versed in XMPP protocols (I myself am learning as I go), just helping to test the program would be a big help. Thanks, Alex Eftimiades A brief summery of how the program works can be found here: https://trac.torproject.org/projects/tor/ticket/9022#comment:42 , and is reproduced below: ---------------------------------------------<Summery of concept > --------------------------------------------------------------------------------------------------- When using hexchat, the data is sent to a local TCP socket running hexchat (call it hexchat1). Hexchat1 then reads the data (thereby stripping it of its TCP header) and passes it over a chat server to another hexchat program (call it hexchat2) that sends the data to the appropriate ip:port (giving it a new TCP header in the process). The client thinks it is sending the data to hexchat1, and the server thinks it is receiving data from hexchat2, but the data itself is never changed. It might be broken into smaller chunks or combined into bigger chunks, and it might be delivered at unpredictable rates, but it is never altered. ---------------------------------------------</Summery of concept > --------------------------------------------------------------------------------------------------- A more detailed overview of the protocol can be found in the protocol- spec here, though I will summarize it: ---------------------------------------------<Summery of protocol > --------------------------------------------------------------------------------------------------- When the program is started, it is told what JID(s) to use to connect to a chat server, and possibly the following optional parameters: 1. An ip:port to listen on. 2. A JID. Connection requests will be sent to this JID when a connection is is made to the ip:port specified in (1). 3. Another ip:port. This is the ip:port the bot connected from the JID in (2) should connect to. Note that any bot can connect to any ip:port that client asks for. This could be a 127.0.0.1 address, or a remote address (but will probably be the former). I plan to add a feature in which you can provide a list of acceptable ip:ports to connect to and refuse to connect to anything else (for security). However, that is not a priority at the moment. When a connection is made, the server and client exchange all JIDs they are using to connect to the chat server so they can each rotate through JIDs to send messages from and JIDs to send messages to. With the exception of initial connections, all messages are sent as IQ stanzas. When data is sent, it is base 64 encoded and sent with an ID number that is incremented each time a message is sent. When data is received, the id is checked against the id of the last message that resulted in writing data to a socket. The difference between these two ids is used as an index in a buffer. Then, messages are read from the buffer and sent to the appropriate socket. When a socket closes, a disconnect message is sent. Like when sending messages, an id stanza is included. When a disconnect message is received, the bot checks if there are any entries in the appropriate socket's message buffer. If not, it closes the socket and deletes it from the routing table. If there are messages in the buffer, the disconnect message is added to the buffer at the appropriate index as a string "disconnect" as though it were normal data. When a "disconnect" string is found in the buffer, the socket is closed and deleted from the bot's routing table. ---------------------------------------------</Summery of protocol > --------------------------------------------------------------------------------------------------- [View Less]

2 2

Re: [tor-dev] Gitian builds in VirtualBox with Vagrant
by Arlo Breault 24 Jun '13

24 Jun '13

> So this is a wrapper to install+launch a VirtualBox Ubuntu VM to use > Gitian in LXC mode? Sounds good! > > Exactly. Just codifies what it took to get it running in a VM. > > Hopefully by Monday I will have tagged TBB 3.0a2, and we will begin > building it. If your build matches (or comes close), I will merge this > stuff for 3.0a3. > > Note that I am about to commit some changes to mkbundle-*.sh, so be > sure to rebase your github branch for your build … [View More]

1 0

[Stegotorus] Fundamental problem with ack/retransmission mechanism
by vmonmoonshine＠gmail.com 24 Jun '13

24 Jun '13

Hello Zack, In testing the retransmission algorithm, I encounter situations that the ack/retransmission algorithm is unable to resolve by design. It happens when the header of a packet gets deliver but the middle part gets dropped. For example in the following scenario Server sends packet 1 len 4k packet 1 gets lost Server sends packet 2 len 64k the last 60kb of packet 2 gets lost client receives packet 2 partially (4k) and waits for the 60k to come. client sends ack to the server saying that … [View More]

4 6

Standalone flash proxy
by Arlo Breault 21 Jun '13

21 Jun '13

https://trac.torproject.org/projects/tor/ticket/7944 Hello, I wrote a little wrapper to provide a headless flash proxy. It requires node.js >= v0.8.x To install and run, > npm install -g flashproxy # adds the flashproxy bin to your path > flashproxy Or you can, > git clone -b nodejs https://github.com/arlolra/flashproxy.git > cd flashproxy/proxy > npm install > npm start I'm looking for testers. Most of the time the facilitator won't return … [View More]

2 1

Discussion on the crypto migration plan of the identity keys of Hidden Services
by George Kadianakis 19 Jun '13

19 Jun '13

Greetings, I'm supposed to write a Tor proposal for the migration of the long-term identity keys of Hidden Services. When I began writing the proposal, I realized that some of my choices might not be appreciated by Hidden Service operators, and that starting a discussion thread might be a good idea before writing the proposal. The problem with the current long-term HS identity keys is that they are RSA-1024 keys which are considered weak, and they need to be upgraded to a cryptosystem with … [View More]higher security properties. One of the main issues with this operation, is whether Hidden Services will be accessible using their *old* identity keys even after the migration. That is, when we change the identity keys of a Hidden Service, its onion also changes (since the onion is the truncated hash of its public key). This will be quite problematic for Hidden Services that have a well-established onion address. There are basically two ways to do this: a) After the transition, Hidden Services can be visited _only_ on their _new_ onion addresses. This is quite brutal, but it's the most secure and unambiguous option (might also be easier to implement and deploy). This change can be enforced both on the client-side, by rejecting any old RSA-1024 HS keys, and on the server-side, by only publishing the new keys in HS descriptors. To make the transition easier, we could prepare a tool that generates a new identity keypair before the flag day, so that Hidden Service operators can learn their future onion address beforehand and announce it to their users. b) After the transition, Hidden Services can use both old and new onion addresses. This might result in a more harmonious transition, where Hidden Services advertise their new onion address to users that visit them in their old address. .oO(It would also be interesting to do a redirection on the Tor protocol layer ("I got this descriptor by querying for the old onion address, but it also contains a new onion address. I should probably use the new one."), but I don't think it's possible to redirect the user without knowledge of the application-layer protocol (e.g. 302 for HTTP). Still, a Tor log message might be helpful.) The cons of this approach is that supporting both addresses might make the HS protocol more complicated and painful to implement, and it might also result in some Hidden Services never moving to the new onion addresses since clients can still visit them using the old insecure ones. This approach has a stricter variant, where the old addresses can only be used during a transitionary period (a few months?). After that, clients _have_ to use the new addresses. Of course, this means that we will have to do two flag days, coordinate Tor releases, and other no fun stuff. I'm probably moving towards the latter option because the former will make many people unhappy. Thoughts? (This is not a thread to select the cryptosystem we are going to use. It will derail the discussion, and we might also need to select a specific type of cryptosystem in the end (e.g. a discrete-log-based system) so that schemes like https://trac.torproject.org/projects/tor/ticket/8106 can be possible.). [View Less]

16 29

Running Atlas Application
by abhiram 18 Jun '13

18 Jun '13

Hello, I am trying to get the Atlas application to run. After installing the required python packages, I keep getting this error. Traceback (most recent call last): File "run.py", line 34, in <module> from tornado.util import b, bytes_type, import_object, ObjectDict ImportError: cannot import name b Its seems like a version problem with tornado. If so, what version should I be using. I am running python 2.7.3 and tornado 3.0.3. Thanks -- Abhiram Chintangal Graduate Student … [View More]

2 2

Re: [tor-dev] [liberationtech] Help test the new Tor Browser!
by Bernard Tyers - ei8fdb 17 Jun '13

17 Jun '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Congratulations Tor Project. Well done to Mike Perry and all the contributors. I've tested it on Mac OS X 10.6.8 and Debian 6.0 Squeeze and I had no technical issues on either. First launch (using clear Internet connection) took approx 40-50 seconds on each. (Debian was running as a VM on a Macbook Pro) The biggest usability hurdle for Tor (IMO) was having the browser launching separately to the Tor application. I've tested with users and this … [View More]was a huge confusion for them. It wasn't a browser as *they* understand one. Now it is. First prong of the attack: this is how privacy enhancing software should behave - the exact same as all other software. Now Tor is even better. Second prong of the attack. Run more exit nodes. - From the quick run through I did, here are some(possibly minor) suggestions: 1. The installed application icon is as follows http://diymobileusabilitytesting.net/bernard/skitches/tbb-3.0alpha1-icon1-2… However when the application is opened, the application icon is this http://diymobileusabilitytesting.net/bernard/skitches/tbb-3.0alpha1-icon2-2… It may be confusing for someone who was not familiar with the different icons used by the TP. 2. The copy displayed during the initial install ("Before the Tor Browser Bundle tries to connect to the Tor network, you need to provide information about this computer's Internet connection) could possibly be reworded to give some context as to *why* it is being asked for. (Possibly reposition the copy to above the connection steps) Alternatively, is it possible for the install to run these two tests and determine to correct outcome? Ie. 1. Run some "tests" to determine if the Internet connection is clear of obstacles, then 2. Run some "tests" to check if the Internet connection is censored/filtered. Based on the outcome of these tests, Tor could then configure the connection as necessary. I could see this step being confusing for users not familiar with their Internet connection. 3. It would be interesting to see the numbers of users who actually follow the "Test Tor Network Settings" link. Once the TBB has installed and displays the "Congratulations! This browser is configured to use Tor." page, are users guaranteed to be connected to the Tor network? If so is there any need for the Test Tor Network..." link? Is it possible to display that information on the startpage? It is also very nice that the user preferences have been altered to be more privacy enhancing (History, etc). Congratulations to all involved. It is great work. Bernard On 17 Jun 2013, at 17:02, Michael Carbone wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Congratulations Tor devs! Serious kudos -- this is exactly the > direction TBB needs to go. > > A couple minor things: the order of the addons in the toolbar seems > arbitrary (particularly the location of the Tor button, NoScript, and > HTTPS Everywhere). I'm sure it's not, but at minimum it might be good > in the about:tor splash page to have an arrow pointing to the location > of the TorButton in the toolbar if folks need to change settings. > Also, the search button image in about:tor is pixelated. > > This is a huge step forward in UX, very exciting! > > Michael > > On 06/17/2013 09:45 AM, Jacob Appelbaum wrote: >> Hi, >> >> I'm really excited to say that Tor Browser has had some really >> important changes. Mike Perry has really outdone himself - from >> deterministic builds that allow us to verify that he is honest to >> actually having serious usability improvements. I really mean it - >> the new TBB is actually awesome. It is blazing fast, it no longer >> has the sometimes confusing Vidalia UI, it is now fast to start, it >> now has a really nice splash screen, it has a setup wizard - you >> name it - nearly everything that people found difficult has been >> removed, replaced or improved. Hooray for Mike Perry and all that >> helped him! >> >> Here is Mike's email: >> >> https://lists.torproject.org/pipermail/tor-talk/2013-June/028440.html >> >> Here is the place to download it: >> >> https://people.torproject.org/~mikeperry/tbb-3.0alpha1-builds/official/ >> >> Please test it and please please tell us how we might improve it! - -------------------------------------- Bernard / bluboxthief / ei8fdb IO91XM / www.ei8fdb.org -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJRv4OOAAoJENsz1IO7MIrrCcQH/iic0Jy+xpAfFTXs29cxuQV2 Lcw/Im2uxxxwapQGK3+7hGWkfynwG+O/CvyN/RaFbCx6a2GywS8D++SAhSEpCVyL GMA6Vx8ZqiJ5KoqQkQ2Y2ENCMLkGIxgD374+bfSkHS5wkSmBesV2/DMva96PxO9e KZT9qZve/OwlXgsCKA0Z1CuHxPpxrbC9htNpRSJ31GUjNv+jZc6OIhDdAEbayx2W IBlgtsrb+glRe5gl1cRaBej3fnn6/zFoVoOMTQhwQEQr6xo8bvQUEcyNsHjMS6gW J6c3hSGcMmUnesvYCOv/x5BXGvC0FQBHHpk4+jh3zNeU3VAik59BLiVQ7e1PCL0= =ad2Q -----END PGP SIGNATURE----- [View Less]

1 0

difference between cached-descriptors and cached-consensus
by moataz ahmed 16 Jun '13

16 Jun '13

I have one question please what is the difference between cached-descriptors and cached-consensus Best Regards Moataz Ahmed,

2 1