- tor-dev - lists.torproject.org

Stem Tutorials
by Damian Johnson 19 Mar '13

19 Mar '13

Hi all. Stem is having its initial release later this month, and in anticipation of it I've been expanding its tutorials... https://stem.torproject.org/tutorial.html Are there any rough spots in these tutorials? Other categories that we should cover? Suggestions welcome! -Damian

3 3

Proposal: Capturing Traffic Statistics from Exit Relays
by Zack Weinberg 18 Mar '13

18 Mar '13

[This draft proposal is also available at git.torproject.org:users/zwol/torspec.git/proposals/ideas/xxx-exit-statistics.txt.] Filename: xxx-exit-statistics.txt Title: Capturing Traffic Statistics from Exit Relays Author: Zack Weinberg Created: 04-Feb-2013 Status: Draft 1. Motivation We propose to collect additional aggregate traffic information from exit nodes which will be of use to researchers, similar to how the project already collects statistics about its user population. We also propose to bring existing entry and exit statistics under the same umbrella so that anonymity protection can be applied in a principled fashion to the entire data set. There is some prior discussion in tickets #6002 and #6003. Exit nodes currently collect statistics on the destination TCP ports of exiting traffic, but this is insufficient information for research; in particular, there have been repeated requests for information about destination hosts. Individual researchers have instrumented exit nodes on an ad-hoc basis (e.g. [1]) but the project has historically been skeptical about both the value and the safety of doing so (e.g. [2] warns of running afoul of wiretapping statutes; I understand that the [1] paper was met with severe criticism, but I can't find that right now). A principled, network-wide policy and mechanism for collecting exit metrics can satisfy the demand for data for research purposes while protecting our users and minimizing additional legal risk to exit node operators. 2. Design At each exit node, we propose to measure the number of exiting TCP connections and total bytes transferred in each direction, per day, categorized three different ways: * TCP port * "Public suffix" + 1 domain component of destination (example.com, example.co.uk) * Country of IP address of destination * possibly: ASN of destination and ASN of exit (this may reveal potential traffic-correlation attackers that would not be visible any other way, but may also be too fine-grained to be safe). These are sufficiently coarse-grained that we believe they can be safely measured without damaging the network's anonymity guarantees; see below for further measures that will be taken to preserve anonymity. At the same time, they will enable interesting analyses of what the network is used _for_, in much the same way that the existing entry-side metrics enable analysis of _who_ uses the network. At the same time, we propose to rationalize entry-side data collection, which currently relies on directory queries rather than actual traffic to actual entry nodes. This will improve accuracy and will also allow us to apply an anonymity-protection algorithm consistently to the entire data set available from the Metrics server. Entry nodes (including bridges) should record entering TCP connections and traffic volume, per day, categorized by: * Country of IP address of traffic source * ASN of IP address of traffic source, and ASN of entry node (subject to same caveat as above) * "I am a bridge" flag (I think this is a superset of the information currently collected via directory queries. If I am mistaken, please let me know.) All collected data should be passed through a differential-privacy sanitization algorithm before it leaves the Tor process's memory, and should then be uploaded to a central server (probably via a write-only hidden service API) which applies a second layer of sanitization. See below for further discussion. 3. Security implications Any collection of information about the operation of the Tor network requires careful analysis to ensure that the anonymity of its users is preserved. In this case, we are contemplating adding to what is publicly available about what the network is used _for_, and we must ensure that this does not lead to a correlation attack that would reveal _who_ did a particular thing. For instance, suppose the adversary knows that _someone_ posted a "sensitive" document on a particular site on a particular date. They also have reason to suspect that the poster hails from a particular country. If our published metrics reveal that only one person used Tor to access that site on that date, and/or that only one person from that country made use of Tor on that date, the adversary's case against their suspect has now gotten considerably stronger. The theoretical framework that deals with this class of exposure is called _differential privacy_. It rests on two observations. First, it is impossible to guarantee that _no one's_ privacy will be compromised via the release of statistics -- consider the somewhat contrived but still evocative case where the adversary already knows that Alice is 6cm shorter than the average citizen of Ruritania; publication of the Ruritanian average height reveals Alice's actual height. Note that Alice _does not_ have to be included in the average for this to work! But the adversary had to know something about Alice already. This leads to the second observation, which is that we _can_ make a reasonable privacy guarantee by changing the model. Instead of trying to protect everyone _whether or not_ they are in the database, we guarantee that, with high probability, the _released statistics_ would not be changed by the inclusion or removal of any one individual from the underlying data set. (This can be generalized to groups of N individuals, although of course the statistics get less and less accurate as N goes up.) Differential privacy prevents the example scenario described above: the adversary would only learn a _range_ of possible values for 'how many people visited site X today via Tor' and would therefore not be able to draw new conclusions about their suspect. The basic technique of differential privacy is to add "noise" to the aggregate statistics before publishing them. There are a wide variety of "mechanisms" for doing so, of which the most basic is simply to add "independent samples from a Laplace distribution" to each statistic. Research in this area focuses on finding mechanisms that minimize the amount of noise required to provide a given level of privacy (according to a security parameter, 'epsilon' as usual). [3] is a good overview of the field and [4] proposes a particular algorithm that seems well-suited for the data set we're looking at here. We must also consider the security of the reporting process. The Metrics server is already trusted, and applying "noise" there will allow us to minimize the overall amount of noise applied for the same level of differential privacy. However, if this is the only place noise is applied, a malicious node operator could capture the 'raw' statistics by reconfiguring their node to report to a server that doesn't sanitize (but perhaps forwards the report to the real server, avoiding suspicion). If each node applies some level of noise before the statistics ever leave the Tor process's memory, this class of attack is much harder, although not impossible (by modifying code or extracting raw statistics from process memory with a debugger). There is at least one proposal [5] for distributed sanitization of a data set. It is Byzantine fault-tolerant and therefore _may_ be overkill for our purposes ... but we're talking about malicious node operators here, so maybe it isn't! 4. Specification TBD 5. Compatibility should be moot 6. Implementation TBD 7. Performance and Scalability minor overhead at each exit (and entry); additional load on Metrics; if reporting is via hidden service, additional load on the network 8. References [1] "Shining Light in Dark Places: Understanding the Tor Network". McCoy, Bauer, Grunwald, Kohno, Sicker. PETS 2008. http://freehaven.net/anonbib/cache/mccoy-pet2008.pdf [2] https://www.torproject.org/eff/tor-legal-faq.html.en [3] "Differential Privacy for Statistics: What we Know and What we Want to Learn". Dwork and Smith. Journal of Privacy and Confidentiality, 2009. http://repository.cmu.edu/jpc/vol1/iss2/2/ [4] "Optimizing [Linear/Histogram] Counting Queries under Differential Privacy". Li, Hay, Rastogi, Miklau, McGregor. Principles of Database Systems, 2010. http://arxiv.org/abs/0912.4742 [5] "Our data, ourselves: Privacy via distributed noise generation". Dwork, Kenthapadi, McSherry, Mironov, Naor. EUROCRYPT 2006. http://www.truststc.org/pubs/101/ourDataOurselves.pdf

2 2

Obfs Links on dist are Suddenly Gone
by Nima 15 Mar '13

15 Mar '13

Hi, I've largely distributed mirror links to pyobfsproxy specially for Windows in Iran. Like this: http://torproject.ph3x.at/dist/torbrowser/tor-flashproxy-pyobfsproxy-browse… Simply because this is the only thing that works. But suddenly they're all gone!!! Is this on purpose or it's just an accident? People are asking for new links. what should we do now? :/ -- Nima 0x1C92A77B "I disapprove of what you say, but I will defend to the death your right to say it" --Evelyn Beatrice Hall

4 4

Interested in contributing to Tor-Dev through GSoC
by Ashish Patil 10 Mar '13

10 Mar '13

Hello! I am a newbie & very eager to contribute to Tor development. I have been using tor since a year and a half & I have been looking around the website & the source codes since past month. The upcoming GSoC has provided a very nice opportunity for me contribute to whatever extent I can. I have interest towards Stem & Obfsproxy due to their applications & would like some beginner's guidance for the same. Also, I would appreciate if anyone could provide me some online references that would help me understand networking better (in order to understand the source code fully, also is that completely necessary?) PS - I am very well acquainted in Python & have also contributed to it's development but only in Documentation aspect, hence have good knowledge of revision control, building, etc. Thanks! -Ashish Patil

3 6

Interested in contributing to tor through GSoC
by Anirudh Vemula 09 Mar '13

09 Mar '13

Hi, I use tor everyday in my college as we have a http proxy which blocks specific sites. It has been a huge help for me and I would like to improve it by contributing to it by implementing new ways and resolving bugs. I would like to do this year's GSoC with the tor organisation and I felt it would be better for me if i get acquainted with the source code well beforehand. I need suggestions on how to start going about understanding the source code. I would love if there are bugs and mini-projects that a beginner could resolve and improve my understanding of the source code. Thanks Anirudh V Anirudh Vemula | Junior Undergraduate | Computer Science & Engineering | vvanirudh.blog.com | Alumni Secretary Hostel 2 | IIT Bombay

2 1

[ooni] proposed specification for ooni backend component
by Arturo Filastò 06 Mar '13

06 Mar '13

Greetings! It would be great to get some feedback on the specification of oonib that is the backend component of ooniprobe. As later said, not everything described here is implemented, but this document details what oonib should become. Moreover if there is something that you believe we should pay special attention to in when implementing this it would be of great use to know. If you wish to modify this spec directly you can send us a pull request on github: https://github.com/TheTorProject/ooni-spec ------------------------------------- # oonib specification * version: 0.1 * date: 2013-03-04 * author: Arturo Filastò This document aims at providing a functional specification of oonib. At the time of writing this document not all parts are fully implemented, though the application interface to oonib is. # 1.0 System overview oonib is the backend component of ooni. It is responsible for: * Collecting the results of tests from ooni-probes (Collector). * Exposing a set of services that are needed for allowing ooni-probes to perform their tests (Test Helpers). # 2.0 Collector ## 2.1 System overview The oonib collector exposes a JSON RPC like HTTP interface and allows probes to submit the results of their measurements. Once probe measurement is complete the test result is published as an ooni test report in the YAML format. The oonib collector shall be exposed as a Tor Hidden Service and as a HTTPS service. ## 2.2 Theat model The collector shall provide end-to-end between the probe and the oonib collector. A malicious actor should not be able to update test results that they have not created. It is outside of the scope of the oonib collector to provide blocking resistance or to conceal to a passive network observer the fact that they are communicating to a collector. Such properties are to be provided by other software components, for example using Tor or obfsproxy. ## 2.3 Test result submission interface Unless otherwise stated all of the network operations below can be performed either via HTTPS or HTTPO (HTTP over Tor Hidden Service). Note: we will eventually want to migrate over to using YAML instead of JSON as a data exchange format. Not doing so adds unnecessary overhead in including YAML data inside of JSON data. ### 2.3.1 Create a new report When a probe starts a test they will *create* a new report with the oonib collector backend. The HTTP request it performs is: `POST /report` { 'software_name': `string` the name of the software that is creating a report (ex. "ooni-probe") 'software_version': `string` the version of the software creating the report (ex. "0.0.10-beta") 'probe_asn': `string` the Authonomous System Number of the network the test is related to prefixed by "AS" (ex. "AS1234") 'test_name': `string` the name of the test performing the network measurement. In the case of ooni-probe this is the test filename without the ".py" extension. 'test_version': `string` the version of the test peforming the network measurement. 'content': (optional) `string` it is optionally possible to create a report with already some data inside of it. 'probe_ip': (optional) `string` the IP Address of the ooniprobe client. When the test requires a test_helper the probe should inform oonib of it's IP address. We need to know this since we are not sure if the probe is accessing the report collector via Tor or not. } The server will respond with a report identifier that will then allow the probe to update the report and the version of the backend software like follows: { 'backend_version': `string` containing the version of the backend 'report_id': `string` report identifier of the format detailed below. 'test_helper_address': `string` the address of a test helper for the requested test. } The report identifier allows the probe to update the report and it will be contructed as follows: ISO 8601 timestamp + '_' + probe ASN + '_' + 50 mixed lowercase uppercase characters A report identifier can be for example: 1912-06-23T101234Z_AS1234_ihvhaeZDcBpDwYTbmdyqZSLCDNuJSQuoOdJMciiQWwAWUKJwmR Note: The report identifier should be at least 256 bits and generated by means of a CSPRNG. Client implementation notes: Probes should not expect the report identifier to be in a particular format as the report id may be changed in the future. ### 2.3.2 Update a report Once the probe has a report ID they will be able to add test related content to the report by referencing it by id: `PUT /report` { 'report_id': `string` the report identifier 'content': `string` content to be added to the report. This can be one or more report entries in the format specified in df-000-base.md } The backend should validate the request to make sure it is a valid YAML Stream. New collectors should use the following format for updating reports: `POST /report/<report_id>` { content: `string` content to be added to the report. This can be one or more report entries in the format specified in df-000-base.md } ### 2.3.3 Closing a report This request is done by a probe to tell the backend that they have finished running the test and the report can be considered done: `POST /report/<report_id>/close` To create a new report a probe will peform an HTTP POST request to the resource /report. The collector MUST implement the following HTTP JSON RPC like API: /report ## 2.4 Report lifecycle When a report is created (section 2.3.1) it should be marked as NEW and a timestamp should be recorded. Once it is updated (section 2.3.2) it should be marked as ACTIVE. An ACTIVE report can be updated with new data. The collector must keep track of the last time a certain report is updated with new and valid data. A report is considered CLOSED when either the probe instructs the collector to close the report (section 2.3.3) or when a report is in ACTIVE state and has not been updated with valid data for more than 2 hours. ## 2.5 Report publishing and cleaning Once a report is closed it should be made available to the public for download and analysis. This shall happen as soon as a report reaches the CLOSED state. Reports should be discarded and deleted if: * They have been in the NEW state for more than 4 hours * They are CLOSED, but contain only one Report Entry meaning that the report entry contains only the base test data (see df-000-base.md). Reports should be published to: `https://ooni.torproject.org/reports/` **reportFormatVersion** `/` **CC** `/` Requesting such URL may also result in a 302 to the location of reports for that specific country. Where CC is the two letter country code as specified by ISO 31666-2. For example the reports for Italy (CC is it) of the reportVersion 0.1 may be found in: https://ooni.torproject.org/reports/0.1/IT/ This directory shall contain the various reports for the test using the following convention: test name - timestamp in ISO8601 format - probe AS number - probe|backend.yamloo The timestamp is expressed using ISO 8601 including seconds and with no : to delimit hours, minutes, days. Such date is the time in which the report was created and must be set by the backend. Like so: YEAR - MONTH - DAY T HOURS MINUTES SECONDS Z The time is always expressed in UTC. If a collision is detected then an int (starting with 1) will get appended to the test. For example if two report that are created on the first of January 2012 at Noon (UTC time) sharp from MIT (AS3) will be stored here: https://ooni.torproject.org/reports/0.1/US/http_test-2012-01-01T120000Z-AS3… https://ooni.torproject.org/reports/0.1/US/http_test-2012-01-01T120000Z-AS3… Implementation notes: The task of publishing a report should be made modular so that we can replace the publishing mechanism if we discover the limitations of this system (not infinite disk space). The basic implementation shall just scp the files to a machine that is configurable by config file. ?? Question: How does this integrate into the m-lab infrastructure? How will this work when some reports results are stored on m-lab and some are not? ## 2.6 Test helper collector relationship Some tests involve adding to the report also data that is collected by means of a test helper. This is the case in the two way traceroute test, where the report should include a traceroute also from the vantage point of the ooni backend. In these circumstances the report from the vantange point of the backend should be inside of a separate file that has the same name of the probe report but "-probe" should be replaced with "-backend". For example the backend part of the report for a traceroute test called `two_way_traceroute-2012-01-01T120000Z-AS3-probe.yamloo`, shall be called `two_way_traceroute-2012-01-01T120000Z-AS3-backend.yamloo` # 3.0 Test Helpers These are services exposed to ooniprobe clients that are of assistance to performing network measurements. # 3.1 System overview Probes will always receive as address of a test helper that of the one running on the same machine as the collector. They will only point to a test helper on a different machine if the test helper for the test the user is interested in running is not available on the desired machine. This can happen, for example, if the machine does not have two network interfaces with two differnet IP addresses. In this case the HTTP Return JSON Headers test helper cannot run at the same time as the TCP echo test helper (both bind to port 80). Some communication with the collector is required. This is the case of the two way traceroute test, where a multiport multiprotocol traceroute must also be performed from the backend to the probe. Test helpers are two kinds: * Reply: are test helpers that reply to requests from probes. For example HTTP test helpers are of this kind. * Active: are test helpers that actively perform requests towards the probe idepedently from probe requests. Implementation notes: Although I am talking about the collector as two differnet software components they both run inside of the same process and are part of the same piece of software. ## 3.2 Threat model (or non-goals) Because of the nature of the services that they are exposing it is not possible to guarantee end to end confidentiality and authentication of the data transmitted to and from test helpers. Moreover we are currently not making any particular effort to make test helpers look like something that they are not (i.e. make test helper traffic not look like test helper traffic). ## 3.3 Test Helper collector mapping When a report that requires a test helper is created with the collector component of oonib the test helper should be notified of the probes IP address (the probe_ip). This will allow the test helper to know from which IP address it should either expect the probe to come from or towards what IP address it should perform an active measurement. ## 3.4 Reply Test Helpers ### 3.4.1 HTTP Return JSON Headers This test helper will bind on port 80 and expect HTTP requests from a probe. It shall respond to every HTTP request with the HTTP Headers and HTTP request line as seen from the backend point of view. The response is structured in JSON as follows: { 'request_headers': [[HTTP header1 name, HTTP header1 value], [HTTP header2 name, HTTP header2 value]] the list is ordered based on how the headers were received. 'request_line': the value of the HTTP request line. 'headers_dict': `dict` containing as keys the HTTP header name (normalized) and as value a list containing the values of such header } For example: { 'request_headers': [['User-Agent', 'IE6'], ['Content-Length', 200]] 'request_line': 'GET / HTTP/1.1' 'headers_dict': {'User-Agent': ['IE6'], 'Content-Length': [200]} } ### 3.4.2 DNS Test Helper Shall provide a DNS resolver over UDP and TCP exposed on port 53. Such DNS resolver shall not filter any DNS query. ### 3.4.3 TCP Echo Helper This shall expose a TCP echo service that is bound to port 80. ## 3.5 Active Test Helpers ### 3.5.1 Two way traceroute This shall perform the ooniprobe traceroute test and attach the result to the final report as described in section 2.5.

1 0

transparent proxy detection
by Sam Smith 06 Mar '13

06 Mar '13

Hey all, I'm looking at detection and identification of some transparent proxies (of different kinds). Has anyone already got tests that effectively but generically merge manipulation/http_host and experimental/squid.py ? Also, I'm testing on a debian (6.0) VM running in a parallels 8 on OS X. Ooni successfully runs tests once , and then requires a debian reboot in order to run again (especially http_host, it was less repeatable with others). I have no idea what's going on, and it'll be a pain to debug, but it's the sort of thing someone's probably interested in for other purposes ;) thanks for ooni Best Wishes Sam -- @smithsam

2 1

Building Tor Browser Bundle on Windows
by Tom Ritter 05 Mar '13

05 Mar '13

This weekend I tried building several of the tor projects on Windows (tor, vidalia, tbb, and vanilla firefox). I kept notes of my adventures, some of the quirks I ran into, and the final set of instructions that seemed to work for me. Perhaps most usefully, I've created an Amazon AMI that anyone will be able to boot up and use. (Yes, this has trust issues, I've addressed them in the top of the README document attached.) The intention is that people who want to test a patch or develop a patch can use this to bootstrap themselves, without having to fight long and hard to set up a Windows build environment. I've attached two documents: 1) TBB Environment README.txt 2) Adventures in Compilation.txt The first is the README for the environment. The second is a journal of my trying to figure out what to do. Specifically I'm going to call out several issues. If desired, I can submit patches to address them, and they can be reviewed. I understand that documentation is usually a lower priority than actually shipping software, please take these comments not as bitchy criticism, but instead a list of things I'd like to improve, if someone can review my suggestions and shepard me a little. 1) Firefox's official build environment is VS2010 while TBB's is VS2008 I'm not 100% sure, but it seems like it'd be relatively easy to switch TBB over to 2010. There may be some issues. But it seems like it'd be good to track FF's official build env. I ran across at least 1 Vanilla FF issue that broke in 2008, and there seemed to be a few in the bugtracker. 2) Vidalia's build environment instructions contradict TBB's https://gitweb.torproject.org/vidalia.git/blob/HEAD:/INSTALL https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/docs/buildmachine_s… Vidalia's says you need CMake in your PATH, TBB's says you don't. I think it can be in your PATH. 3) There's no explicit instructions on how to build Tor in Windows It seems like you just build it in MinGW? It worked anyway. It's a (minor) bitch to get the components built and placed in the 'correct way and place' for the MinGW toolchain. (Now documented in my README) 4) The TorBrowser build instructions are really, really bad. https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/docs/buildmachine_s… This is pretty good. But it just ends abruptly, and doesn't tell you want to do. https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/docs/HACKING This document seems to be mostly up to date, although I think the FF Extensions it lists have been changed since. https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/build-scripts/INSTA… This one is super out of date. - You don't need subversion or winrar - It talks a lot about getting FirefoxPortable (or built a custom Firefox) - I didn't try following the instructions for MinGW, msys, cmake, qt, msysGTK, or autoconf - since I had those already. It seems these sections have been subsumed by the windows buildmachine document - The OpenSSL instructions do not match the instructions IN OpenSSL - Vidalia's links are way out of date and dead - Its instructions for Vidalia don't match Vidalia's - It talks about PidginPortable? I think that's been nixed... - "Currently the Tor Browser Bundle uses FirefoxPortable binaries." - It talks about building with MSVC 7.1 (Visual Studio 2003) - It stops short of telling you how to actually build the bundle, probably because the makefiles are newer than this document https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/build-scripts/DEPLO… This doesn't seem applicable at all anymore I eventually found (by searching my email spools) the magic incantation to kick it off. The "TorBrowser" section of Adventures in Compilation.txt shows my hiccups and luck getting it going. Finally, after all this.... tbb-firefox crashes. I don't know why. I don't know if it's because I'm using VS2010, or my msvc*.dll swaps. Specifically it's crashing here: c:\build\torbrowser\build-scripts\build-alpha-windows\build\firefox-17.0.3esr\content\xul\templates\src\nsrulenetwork.h class Iterator : public ConstIterator { public: Iterator(ReteNode** aNode) : ConstIterator(aNode) {} Iterator& operator++() { ++mCurrent; return *this; } Iterator operator++(int) { Iterator result(*this); ++mCurrent; return result; } ReteNode* operator*() const { return *mCurrent; } <========================================= ReteNode* operator->() const { return *mCurrent; } bool operator==(const ConstIterator& aConstIterator) const { return mCurrent == aConstIterator.mCurrent; } bool operator!=(const ConstIterator& aConstIterator) const { return mCurrent != aConstIterator.mCurrent; } }; I'm not quite sure where to start figuring out *why* it's dereferencing a null ptr. The windbg output is in the bottom of both docs. The AMI for the EC2 instance is ami-2e36ab47 - when you start it, open ports 3389 and 8080 in the firewall for 0.0.0.0/0 (or just your IP address if you want.) If you try it, and get skeeved out by them asking for your SSH private key to get the password, check out https://github.com/tomrittervg/decrypt-windows-ec2-passwd -tom

5 5

Damian's Status Report - February 2013
by Damian Johnson 02 Mar '13

02 Mar '13

Hi all. Between being a short month and oncall for work I didn't get as much done as I'd like. What time I did have for tor mostly went into stem's descriptor functionality. In particular... ======================================== * Microdescriptor Support Stem can now read and parse microdescritpors (#8253), with controller methods coming later this weekend. I understand the desire for lightweight descriptors but they're a step backward for controllers. Their lack of fingerprints make them clunky to use (https://trac.torproject.org/8323#comment:2) and tor lacks the usual methods for retrieving them (#8323) and v3 network status information (#7953). Controllers will often need to read descriptor content from the data directory until those are fixed. * Invalid descriptor content within archives caused the reader to stop processing content from the archive. This bug is simple in retrospect but cost me around a week of hair pulling frustration to sort out. Thanks to Karsten for catching it! (#8049) * Descriptor readers can now optionally provide network status documents rather than the entries they contain. Feature request by Karsten. (#8050) * Calling str() on Descriptors choked if it contained unicode content. Caught by Sathyanarayanan. (#8265) * Thanks to Karsten Descriptors now provide hex digests. (#8036) * Discussed the new 'flag-thresholds' attribute and added support for it to stem. (#8165) * Descriptor parsers that used readline() could choke if derived from a descriptor archive. Caught by Karsten. ======================================== General tasks included... * Discussed proposal 218 with Karsten on tor-dev@... https://lists.torproject.org/pipermail/tor-dev/2013-February/004411.html * Changed our consensus-tracker script to alarm for non-exits (we had an undetected sybil attack early in the month). Also fixed an issue where the script gobbled up way too much memory. (#7831} * Made a few backward incompatible changes to improve stem's usability in anticipation of our March API freeze... * Version comparison is now done through normal comparison operators rather than a meets_requirements() method. * Renamed the keyword arguments for Controller.from_port() and others to be less verbose (for instance 'control_port' to just 'port'). * Dropped the 'path' arg from parse_file(), it was never intended for external callers. * Variety of bug fixes... * We didn't recognize a 'NEVER' date in ADDRMAP events. Caught by Desoxy. {#8162) * Patch from Abhishek so our tests avoid static /tmp usage. (#7926) * Added copyright notices throughout most of our codebase. Suggested by Juan. (#7954) * Addressed issues with get_process_name() on OSX. Caught by Sathyanarayanan. (#8266) Cheers! -Damian

1 0

The Tor git master branch is now 0.2.5.0-alpha-dev
by Nick Mathewson 01 Mar '13

01 Mar '13

Hi, folks! The Tor git master branch is now 0.2.5.0-alpha-dev. Development on 0.2.4.x continues in the branch "maint-0.2.4". I expect that the next alpha will be the last 0.2.4.x-alpha, and then we can move on to beta and rc. We now return you to your semiregularly semischeduled Tor development. yrs, -- Nick

1 0

Interest in Contributing to Tor
by kaushik s 28 Feb '13

28 Feb '13

Hi, I am a junior student at International Institute of Information Technology, Hyderabad with major in Computer Science and Security, and I am very interested in working for your organization this summer and further. I have a strong background in information security and algorithms. I strongly believe in the cause that you are fighting for - free/open Internet for all, and would like to support you by contributing to the code. I have been through the paper by Roger Dingledine and Nick Mathewson, "Design of a blocking-resistant anonymity system" and your work aroused my interest. I am trying to build up my knowledge in the field of information security with some hands-on experience, through projects. The paper by Dingledine, Mathewson had a lot of proposals for making Tor blocking-resistant, but in your list of proposed projects I only found a project for implementing a HTTPS server in the list of project ideas. What is the current status of implementation of ideas listed? I would like to contribute by into the field of algorithms and cryptography by implementing more ideas to make it even more blocking-resistant. Please let me know what are the projects and ideas having maximum priority , so that I can start working on them right away. Thanks & Regards, Kaushik Srinivasan, UG3, B.tech CSE, IIIT Hyderabad.

2 1

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 28 Feb '13

28 Feb '13

[Nick, can you merge my torspec proposal218 branch, please? Thanks!] On 2/28/13 2:08 AM, Rob Jansen wrote: > On Mon, Feb 25, 2013 at 10:28 AM, Karsten Loesing <karsten(a)torproject.org> > wrote: > >> On 2/23/13 11:20 PM, Rob Jansen wrote: >>> On Feb 23, 2013 4:22 PM, "Karsten Loesing" <karsten(a)torproject.org> >> wrote: >>>> Your understanding of n_circ_id and p_circ_id matches mine, but are you >>>> sure there's a UID for circuits other than origin circuits? I think you >>>> mean origin_circuit_t->global_identifier. But there's no such field for >>>> or_circuit_t or circuit_t. Or do you mean something else? >>>> >>> Ok. Though I thought my original patch moved the global Id to the base >>> circuit struct. Perhaps I didn't. Anyway, I'm not sure it matters... >> >> Your original patch did move the ID to circuit_t, but I thought we >> wanted to avoid numbering non-origin circuits (mostly because that >> affects relays in non-TestingTorNetwork mode and could lead to busy >> relays running out of IDs at some point), which is why I took this >> change out. >> > > Right, I remember this now. I could imagine ways to get around the 'running > out of ids' problem, like resetting the id counter after a given timeframe. > In fact, we could just let the counter overflow and loop back to 0 on its > own (assuming its a unsigned int type) since we really only need them to be > unique approximately for the life of a circuit. If we see the ID come up > again after an hour, we can assume its a new circuit. I'd rather not want to change behavior in non-simulation mode. If we really need a UID for non-origin circuits, we could add a separate uint64_t global_identifier to circuit_t and only use that in simulating mode. I'm not sure that we need it though. >> Also, even if we moved the field to circuit_t, the new CELL_STATS events >> would be the only ones using these UIDs, because all other events are >> for origin circuits only. I don't see how these IDs would help us. We >> never learn what UID the next or previous node in a circuit picks for a >> given circuit. > > Not currently, but couldn't we implement the functionality where relays log > or export their circuit UIDs and next/prev IDs over the control port? > Though I'm not sure if you'd want this in mainline Tor if its only useful > in simulation mode... We could have relays log local circuit UIDs together with p_circ_id, p_conn_id, n_circ_id, and n_conn_id in simulation mode. But I don't see how that would facilitate circuit tracking compared to the current approach using ORCONN and CELL_STATS events. Here's an example: - fileclient creates a circuit with - UID 14, - n_conn_id 15, and - n_circ_id 19403. - tokenglobal is the first hop in this circuit and reports - (new) UID 12345, - p_conn_id 32, - p_circ_id 19403, - n_conn_id 18, and - n_circ_id 6710. - tokenrelay is the second hop and reports - (new) UID 23456, - p_conn_id 17, - p_circ_id 6710, - n_conn_id 16, - n_circ_id 34402. - exit2 is the third and final hop and reports - (new) UID 34567, - p_conn_id 15, and - p_circ_id 34402. We need ORCONN events to know that fileclient's n_conn_id 15 is the same connection as tokenglobal's p_conn_id 32. How would the new UIDs make this easier? >>>> tl;dr: I _think_ it's possible to reconstruct circuits from ORCONN and >>>> CELL_STATS events as they are currently specified in proposal 218. >>>> >>> Great, but do we really expect every Tor controller parser to get this >>> right? It seems complicated enough that there should be an easier way. >>> Maybe it's just wishful thinking on my part. >> >> I agree that reconstructing circuits from ORCONN and CELL_STATS events >> is far from trivial. I don't really see how to make it simpler though. > > What about linking all of the IDs and UIDs as described above? (See above.) >> From an earlier mail in this thread: >>>> Finally, Rob, should I look into CIRC_BW events you suggested a while >>>> ago? If yes, what did you have in mind how that event would look like, >>>> and when/by whom would it be emitted? >>> >>> If we want to do this, it would likely be an aggregation of all STREAM_BW >>> events for a circuit, but only during the time when those streams >> belonged >>> to that circuit. I don't think it makes sense to emit it for every >>> STREAM_BW event though, so what if we aggregate and emit it once per >>> second? A format similar to the STREAM_BW format should work fine. >> >> Done. I specified and implemented such a CIRC_BW event. >> >> > Great! > > >> >> Here's the updated proposal 218 (Nick, please don't merge this yet): >> >> >> https://gitweb.torproject.org/user/karsten/torspec.git/blob/refs/heads/prop… > > > In section 5.3/5.4/5.5, these events are emitted in the > second_elapsed_callback(), right? I wanted to verify that a relay who > hasn't sent anything in a few seconds and then starts sending again will > emit the event at the end of the second after which it resumed sending, > rather than the first bytes after it resumed sending. Yes, all these events are emitted in second_elapsed_callback(). > The last word in 5.4 should be 'reading' instead of 'read'. Fixed. > Is the specification of 5.5 and 5.6 complex enough to warrant including > example outputs? Sure, can't hurt. Added a few examples. > In 5.6, were we planning on explaining how buckets can go negative and how > that affects the reporting of the TB_EMPTY events? I tried a better explanation: """ ReadBucketEmpty (WriteBucketEmpty) is the time in millis that the read (write) bucket was empty since the last refill. LastRefill is the time in millis since the last refill. If a bucket went negative and if refilling tokens didn't make it go positive again, there will be multiple consecutive TB_EMPTY events for each refill interval during which the bucket contained zero tokens or less. In such a case, ReadBucketEmpty or WriteBucketEmpty are capped at LastRefill in order not to report empty times more than once. """ > Here's the tor branch: >> >> >> https://gitweb.torproject.org/karsten/tor.git/shortlog/refs/heads/morestats3 >> >> Here's a Shadow log file containing all new events: >> >> https://people.torproject.org/~karsten/volatile/scallion.log.gz >> >> Here's the Java program that I used to parse the Shadow log file: >> >> https://people.torproject.org/~karsten/volatile/ParseProposal218Events.java >> >> > Ugh. I really don't look forward to writing parsers for this. I guess there > may be few projects that actually require this information, and those that > do can use your code:) I hope that some of the complexity goes away once we use a parsing library like Stem. And I think we should provide parsing code to other people looking into this. >> And finally, here's the output, which should be easier to understand now: >> >> https://people.torproject.org/~karsten/volatile/scallion.txt >> >> Search for "Circuit [fileclient-60.1.0.0]:14" to find the circuit I >> mentioned earlier in this thread. >> >> >> Can you review the proposal changes and tell me if they make sense to you? >> >> Best, >> Karsten >> >> > See comments above. Overall, it looks good. I'm still wondering about the > UIDs / IDs issue. It may be that we don't want to include that for other > reasons, in which case the current implementation is fine. Okay, in that case let's consider proposal 218 done for the moment, unless we come up with a better idea to solve circuit-tracking thing. I asked Nick above to merge my changes, so that I can put proposal 218 on the sponsor F year 3 wiki page as one result of the February 28 milestone. (That doesn't mean we cannot make it even better after February.) Thanks! Karsten

2 1

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 25 Feb '13

25 Feb '13

On 2/23/13 11:20 PM, Rob Jansen wrote: > On Feb 23, 2013 4:22 PM, "Karsten Loesing" <karsten(a)torproject.org> wrote: >> Your understanding of n_circ_id and p_circ_id matches mine, but are you >> sure there's a UID for circuits other than origin circuits? I think you >> mean origin_circuit_t->global_identifier. But there's no such field for >> or_circuit_t or circuit_t. Or do you mean something else? >> > Ok. Though I thought my original patch moved the global Id to the base > circuit struct. Perhaps I didn't. Anyway, I'm not sure it matters... Your original patch did move the ID to circuit_t, but I thought we wanted to avoid numbering non-origin circuits (mostly because that affects relays in non-TestingTorNetwork mode and could lead to busy relays running out of IDs at some point), which is why I took this change out. Also, even if we moved the field to circuit_t, the new CELL_STATS events would be the only ones using these UIDs, because all other events are for origin circuits only. I don't see how these IDs would help us. We never learn what UID the next or previous node in a circuit picks for a given circuit. >> tl;dr: I _think_ it's possible to reconstruct circuits from ORCONN and >> CELL_STATS events as they are currently specified in proposal 218. >> > Great, but do we really expect every Tor controller parser to get this > right? It seems complicated enough that there should be an easier way. > Maybe it's just wishful thinking on my part. I agree that reconstructing circuits from ORCONN and CELL_STATS events is far from trivial. I don't really see how to make it simpler though. >From an earlier mail in this thread: >> Finally, Rob, should I look into CIRC_BW events you suggested a while >> ago? If yes, what did you have in mind how that event would look like, >> and when/by whom would it be emitted? > > If we want to do this, it would likely be an aggregation of all STREAM_BW > events for a circuit, but only during the time when those streams belonged > to that circuit. I don't think it makes sense to emit it for every > STREAM_BW event though, so what if we aggregate and emit it once per > second? A format similar to the STREAM_BW format should work fine. Done. I specified and implemented such a CIRC_BW event. Here's the updated proposal 218 (Nick, please don't merge this yet): https://gitweb.torproject.org/user/karsten/torspec.git/blob/refs/heads/prop… Here's the tor branch: https://gitweb.torproject.org/karsten/tor.git/shortlog/refs/heads/morestats3 Here's a Shadow log file containing all new events: https://people.torproject.org/~karsten/volatile/scallion.log.gz Here's the Java program that I used to parse the Shadow log file: https://people.torproject.org/~karsten/volatile/ParseProposal218Events.java And finally, here's the output, which should be easier to understand now: https://people.torproject.org/~karsten/volatile/scallion.txt Search for "Circuit [fileclient-60.1.0.0]:14" to find the circuit I mentioned earlier in this thread. Can you review the proposal changes and tell me if they make sense to you? Best, Karsten

1 0

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 23 Feb '13

23 Feb '13

On 2/22/13 8:50 PM, Rob Jansen wrote: > On Fri, Feb 22, 2013 at 12:59 PM, Karsten Loesing <karsten(a)torproject.org>wrote: > [...] > >> If anything here sounds strange to you, please let me know. I'm not >> 100% certain that this is the best approach to track circuits from >> client to exit, or if it's even correct. >> >> For example, I assume here that circuit IDs are unique between two >> nodes, which I think is correct. But before working on this I also >> assumed that a circuit uses a single connection for both inbound and >> outbound directions (which is apparently not the case). >> >> > Whether or not your assumption about circuit ids is correct depends on > which circuit id you are referring to - the source relay circuit id, the > destination relay circuit id, or the id(s) written in the cells. Here is > how I understand the ids work: > > Suppose relayA and relayB are part of the same circuit, and relayA is > closer to the client than relayB. The id by which relayA refers to its > circuit_t state is only unique to relayA and is chosen when the circuit_t > struct is created. Similarly, the id by which relayB refers to its > circuit_t state is only unique to relayB and also chosen when the circuit_t > struct is created. Check this in the *circuit_new() functions in > circuitlist.c. Lets refer to these as circuit UIDs, as they are unique and > only known to individual relays. > > Now, relayA writes a circuit id into cells it sends to relayB, so that > relayB knows which circuit the cells belong to, but it does not necessarily > use its circuit UID. The id for this is computed in > get_unique_circ_id_by_conn() in circuitbuild.c and is stored in n_circ_id > in the circuit_t struct at relayA. Similarly, relayB uses p_circ_id from > the or_circuit_t struct when sending cells back to relayA. > > Upon receiving cells from relayA, relayB immediately uses the id written in > the cell (relayA's circ->n_circ_id) to look up relayB's UID for its > circuit_t state. There is a circuit id map that is kept for this purpose. > This works similarly from relayB to relayA. > > Now, the problem that prevents us from linking these is that all of the > controller events print out the UIDs, but not the n_circ_id or p_circ_id. I > tried printing out all of these various IDs but never verified that it > actually worked. > > Please let me know if my understanding is flawed in some way. Your understanding of n_circ_id and p_circ_id matches mine, but are you sure there's a UID for circuits other than origin circuits? I think you mean origin_circuit_t->global_identifier. But there's no such field for or_circuit_t or circuit_t. Or do you mean something else? Anyway, your description made me realize that my previous attempt to link CELL_STATS events was flawed for (at least) two reasons: 1. I assumed that a circuit ID (n_circ_id or p_circ_id) is unique for a given pair of nodes, but it's really only unique for a given connection between two nodes. 2. I also assumed that a circuit uses two different connections for inbound and outbound queues, but it's actually the same connection that is known under different connection IDs by the involved nodes. Maybe this gets clearer when looking again at the example: [fileclient-60.1.0.0]:[tokenconn-59.1.0.0]:34760 00:23:16 [fileclient-60.1.0.0] >>> circ=34760 conn=12 create_fast=1;relay_early=2 create_fast=1;relay_early=2 00:23:16 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1;created_fast=1 relay=1;created_fast=1 Circuit ID 34760 is what fileclient picked as n_circ_id for this circuit and what tokenconn stored as p_circ_id. There's a single OR connection carrying this circuit which fileclient identifies as 12 and tokenconn as 31. The part where this gets tricky is when we need to find out (reliably) that connection IDs 12 and 31 refer to the same connection. fileclient never tells tokenconn that it locally refers to this connection as ID 12, and vice versa. However, there can be more than one connection between the two nodes, and it's perfectly valid for those connections to each have a circuit with ID 34760 on them. I modified my Java program to parse ORCONN events to match corresponding connection IDs based on state transitions from NEW or LAUNCHED to CONNECTED. This works in 99% of cases and only fails for OR connections that were launched before the controller registered for ORCONN events. But we probably don't care about that early bootstrapping phase anyway. Here's the same circuit from my original example, this time with explanations (IDs might differ, because this example comes from another Shadow run): [fileclient-60.1.0.0]:12:34761 This queue is for circuit 34761 which runs over connection 12 at fileclient. It's an outbound queue, though that isn't explicitly stated here. The ID 34761 is what fileclient picked as n_circ_id. 00:23:16 [fileclient-60.1.0.0] >>> circ=34761 conn=12 create_fast=1;relay_early=2 create_fast=1;relay_early=2 There were three cells in outbound direction reported in this CELL_STATS event. Leaving out other CELL_STATS events here. [tokenconn-59.1.0.0]:31:34761 This queue is also for circuit 34761, but running over connection 31 at tokenconn. It's an inbound queue, so 34761 is a p_circ_id, chosen by fileclient. 00:23:16 [tokenconn-59.1.0.0] <<< circ=34761 conn=31 relay=1;created_fast=1 relay=1;created_fast=1 Two cells in inbound direction. [tokenconn-59.1.0.0]:16:26405 Queue for circuit 26405 on connection 16 at relay tokenconn. [tokenglobal-57.1.0.0]:15:26405 See above. [tokenglobal-57.1.0.0]:16:53557 See above. [exit2-62.1.0.0]:15:53557 See above. tl;dr: I _think_ it's possible to reconstruct circuits from ORCONN and CELL_STATS events as they are currently specified in proposal 218. Best, Karsten

1 0

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 22 Feb '13

22 Feb '13

On 2/22/13 5:36 PM, Rob Jansen wrote: > On Tue, Feb 19, 2013 at 10:20 AM, Karsten Loesing <karsten(a)torproject.org>wrote: > >> On 2/12/13 4:54 PM, Damian Johnson wrote: >>>> AppWard and ExitWard indicate the direction of cell flow through the >>>> circuit. Previous cell events aggregated statistics for both directions >> and >>>> therefore did not capture information which I believe may be important. >>>> These are much more informative than N and P. Do you have suggestions >> for >>>> improving this further? >>> >>> Ahhh, gotcha. Personally I'd go for 'inbound' and 'outbound' since >>> they better connotate directionality. AppWard and ExitWard sound >>> weird, but they make sense now too and would be fine if CELL_STATS got >>> a quick description saying what they mean. >> >> Sure, I'm fine with calling it 'inbound' and 'outbound'. Will change >> that the next time I touch the proposal. Thanks! >> >> >> I also thought more about Rob's idea to link circuits from client to >> exit in a testing network. I think we should add two more fields >> InboundName and OutboundName to CELL_STATS events. These fields would >> contain LongName ($CC88FB3B78599580F1EE4F6F73E26A7EC3DF2CA1~tokenconn) >> or Target (like 60.1.0.0:10002) of the previous or next node in a >> circuit. While it may be possible to extract this information from >> ORCONN events, that seems somewhat fragile to me. (When testing this, I >> had a case when we were missing ORCONN events, because the controller >> was not yet connected, so it was unclear where the CELL_STATS events >> belonged.) >> >> Here's an example of the extended CELL_STATS event (still using >> positional arguments, because I didn't change the code there yet): >> >> CELL_STATS 48957 15 $9441269F5989487F07AE824063345A0A6BCAB279 >> created=1;relay=1 created=1;relay=1 created=0;relay=0 29201 16 >> $DC284D0012ECB812C0CA2DEB080CB2F839A77E1A create=1 create=1 create=0 >> >> And here's how we'd put together relevant CELL_STATS events to learn >> what happened in a given circuit: >> >> [fileclient-60.1.0.0]:[tokenconn-59.1.0.0]:34760 >> 00:23:16 [fileclient-60.1.0.0] >>> circ=34760 conn=12 >> create_fast=1;relay_early=2 create_fast=1;relay_early=2 >> 00:23:16 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 >> relay=1;created_fast=1 relay=1;created_fast=1 >> 00:23:17 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:01 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:01 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay_early=1 >> relay_early=1 >> 00:25:02 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay_early=1 >> relay_early=1 >> 00:25:02 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=6 relay=6 >> 00:25:03 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=40 relay=40 >> 00:25:04 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay_early=1 >> relay_early=1 >> 00:25:04 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=61 relay=61 >> 00:25:05 [fileclient-60.1.0.0] >>> circ=34760 conn=12 >> relay=1;relay_early=3 relay=1;relay_early=3 >> 00:25:06 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:06 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=1 relay=1 >> 00:25:07 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=44 relay=44 >> 00:25:08 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=63 relay=63 >> 00:25:08 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=3 relay=3 >> 00:25:09 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=2 relay=2 >> 00:25:10 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=1 relay=1 >> 00:25:10 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=1 relay=1 >> 00:25:11 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=94 relay=94 >> 00:25:12 [fileclient-60.1.0.0] >>> circ=34760 conn=12 relay=4 relay=4 >> 00:25:12 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 relay=12 relay=12 >> [tokenconn-59.1.0.0]:[tokenglobal-57.1.0.0]:9272 >> 00:23:16 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 >> create=1;relay_early=1 create=1;relay_early=1 >> 00:23:16 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 >> created=1;relay=1 created=1;relay=1 >> 00:25:01 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay_early=1 >> relay_early=1 >> 00:25:01 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=1 relay=1 >> 00:25:02 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=6 relay=6 >> 00:25:02 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay_early=1 >> relay_early=1 >> 00:25:03 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=41 relay=41 >> 00:25:04 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=60 relay=60 >> 00:25:04 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay_early=1 >> relay_early=1 >> 00:25:05 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 >> relay=1;relay_early=3 relay=1;relay_early=3 >> 00:25:06 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=1 relay=1 >> 00:25:06 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=1 relay=1 >> 00:25:07 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=52 relay=52 >> 00:25:08 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=1 relay=1 >> 00:25:08 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=55 relay=55 >> 00:25:09 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=4 relay=4 >> 00:25:10 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=1 relay=1 >> 00:25:10 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=1 relay=1 >> 00:25:11 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=98 relay=98 >> 00:25:12 [tokenglobal-57.1.0.0] <<< circ=9272 conn=15 relay=8 relay=8 >> 00:25:12 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 relay=4 relay=4 >> [exit1-61.1.0.0]:[tokenglobal-57.1.0.0]:38910 >> 00:23:16 [exit1-61.1.0.0] <<< circ=38910 conn=24 created=1 created=1 >> 00:23:16 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 create=1 create=1 >> 00:25:01 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=1 relay=1 >> 00:25:01 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay_early=1 >> relay_early=1 >> 00:25:02 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=30 relay=30 >> 00:25:02 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay_early=1 >> relay_early=1 >> 00:25:03 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=77 relay=77 >> 00:25:04 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay_early=1 >> relay_early=1 >> 00:25:05 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 >> relay=1;relay_early=3 relay=1;relay_early=3 >> 00:25:06 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=1 relay=1 >> 00:25:06 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=1 relay=1 >> 00:25:07 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=80 relay=80 >> 00:25:08 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=27 relay=27 >> 00:25:08 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=1 relay=1 >> 00:25:09 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=1 relay=1 >> 00:25:09 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=4 relay=4 >> 00:25:10 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=27 relay=27 >> 00:25:10 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=1 relay=1 >> 00:25:11 [exit1-61.1.0.0] <<< circ=38910 conn=24 relay=79 relay=79 >> 00:25:12 [tokenglobal-57.1.0.0] >>> circ=38910 conn=27 relay=4 relay=4 >> >> I think this works for all circuits. Rob, what do you think about >> extending the CELL_STATS events as described above? > > > I don't think I understand the format. Is this what would appear in the log > file? Including the ">>>" and "<<<"? Those seem a little strange. Also, I'm > not seeing where e.g. the ID of the circuit as the entry relay knows it is > mapped to the ID of the circuit as the middle relay knows it. Circuit IDs > are relay-specific, right? Am I missing something here? Ah, sorry, it's the output of my little Java tool that puts together CELL_STATS events from a scallion.log file. (I can share that code if you want, but it's mostly a hack.) Here's how you'd read that format: > [fileclient-60.1.0.0]:[tokenconn-59.1.0.0]:34760 Client fileclient and relay tokenconn have one or more connections that carry circuit 34760. IIUC, fileclient picked that circuit ID. The names [fileclient-60.1.0.0] and [tokenconn-59.1.0.0] are ordered alphabetically, not in the order of whoever opened the circuit. > 00:23:16 [fileclient-60.1.0.0] >>> circ=34760 conn=12 > create_fast=1;relay_early=2 create_fast=1;relay_early=2 In that second, fileclient sent three cells over circuit 34760 using connection 12, one CREATE_FAST cell and two RELAY_EARLY cells. It's unclear who opened that connection and picked number 12 for it, but it doesn't really matter. All cells were added to and removed from the queue in the same second. The ">>>" stands for outbound. > 00:23:16 [tokenconn-59.1.0.0] <<< circ=34760 conn=31 > relay=1;created_fast=1 relay=1;created_fast=1 tokenconn sent two cells in inbound direction belonging to the same circuit 34760 but using connection 31, not 12 (which I didn't know was possible, but hey). > [...] > [tokenconn-59.1.0.0]:[tokenglobal-57.1.0.0]:9272 Relay tokenconn and relay tokenglobal use one or more connections for circuit 9772, which is the extended circuit from above. The format doesn't show this, but the program generating this format makes sure that these two circuit parts belong together. > 00:23:16 [tokenconn-59.1.0.0] >>> circ=9272 conn=18 > create=1;relay_early=1 create=1;relay_early=1 Two cells from tokenconn in outbound direction. Similar to above. > [...] > [exit1-61.1.0.0]:[tokenglobal-57.1.0.0]:38910 Relay exit1 and tokenglobal use ID 38910 for the last part of this circuit. If anything here sounds strange to you, please let me know. I'm not 100% certain that this is the best approach to track circuits from client to exit, or if it's even correct. For example, I assume here that circuit IDs are unique between two nodes, which I think is correct. But before working on this I also assumed that a circuit uses a single connection for both inbound and outbound directions (which is apparently not the case). >> Also, do we still >> need the ID= in ORCONN events if CELL_STATS events are sufficient to >> reconstruct circuits? > > Would the ORCONN IDs still be useful to track state on a per-connection > basis, like bandwidth used per connection (from the CONN_BW event)? Maybe, yes. Okay, leaving it in. >> Finally, Rob, should I look into CIRC_BW events you suggested a while >> ago? If yes, what did you have in mind how that event would look like, >> and when/by whom would it be emitted? > > If we want to do this, it would likely be an aggregation of all STREAM_BW > events for a circuit, but only during the time when those streams belonged > to that circuit. I don't think it makes sense to emit it for every > STREAM_BW event though, so what if we aggregate and emit it once per > second? A format similar to the STREAM_BW format should work fine. Sounds doable. I'll see if I can build this over the weekend. Best, Karsten

1 0

Fwd: python topics
by Ingy dot Net 22 Feb '13

22 Feb '13

Greetings, This is Ingy döt Net, a friend of ioerror's and arma's. I've been following Tor since 2009, and was asked to help out with ooni-probe a bit. I've been researching the state of the ooni, and was on a call with the primaries yesterday. I hack from a coworking space in Seattle that has a lot of various talent. During the call I learned that we need a light-weight, cross-platform gui button for using ooni-probe as simply as possible, and it would be nice to have it on the usual common platforms. I asked a co-coworker who runs the seattle python interest group (SeaPIG) for any advice and he sent me this email. Not sure it's the best stuff available, but at least we have starting points. Cheers, Ingy ---------- Forwarded message ---------- From: Ingy dot Net <ingy(a)ingy.net> Date: Wed, Feb 20, 2013 at 6:44 PM Subject: Re: python topics To: Jonathan Mark <jhmark(a)xenops.com> Thanks Jonathan! I'll let you know how it goes. On Wed, Feb 20, 2013 at 3:58 PM, Jonathan Mark <jhmark(a)xenops.com> wrote: > Tkinter is how to do a portable GUI: > http://wiki.python.org/moin/**TkInter<http://wiki.python.org/moin/TkInter> > > For building a standalone executable file: > There seems to be this new "pyinstaller" thing which sounds nice: > http://www.pyinstaller.org/ > > The old way I knew to do this is to use py2exe and py2app: > http://www.py2exe.org/ > http://wiki.python.org/moin/**MacPython/py2app<http://wiki.python.org/moin/MacPython/py2app> > > "Freeze" does it for Linux: > http://wiki.python.org/moin/**Freeze<http://wiki.python.org/moin/Freeze> > > I don't know how to do iOS or Android (unless a Linux solution covers > Android also) > > cheers, > > J >

1 0

Tor Browser Launcher
by Micah Lee 21 Feb '13

21 Feb '13

In my spare time I've been working on a way to make the Tor Browser Bundle easier to use for GNU/Linux users. I've mostly been talking about it with other people on tor-talk and in this bug: https://trac.torproject.org/projects/tor/ticket/5236 I decided to write Tor Browser Launcher, a completely separate program that's in charge or adding a "Tor Browser" .desktop file (so you can open it from your desktop environment's menu), downloads the latest version for you, verifies it's signature (making it more secure than how much GNU/Linux users get TBB), and installs it in your home directory. I think that Tor Browser Launcher can get in Debian and Ubuntu. I just finished a first version of it. The code is here: https://github.com/micahflee/torbrowser-launcher And there are screenshots here: http://imgur.com/a/Mvpwl You can read details about how it works on my latest comment on that bug: https://trac.torproject.org/projects/tor/ticket/5236#comment:32 Before trying to get it in Debian I'd like to make it so it doesn't need to be updated each time TBB is updated. There are more details in the ticket, but this would require Tor to maintain a file on https://www.torproject.org/ that has the current version number of TBB in it and a timestamp, and possibly digital signature of this file too. Do you think this is doable? I also want to get it localized into all the languages TBB is localized into. Any thoughts or suggestions? -- Micah Lee https://twitter.com/micahflee

7 27

Re: [tor-dev] Switch from Obfsproxy to Pyobfspoxy
by Nathan Freitas 21 Feb '13

21 Feb '13

(moved to tor-dev) Sorry I really haven't commented on this until now, but I have a lot of balls in the air, and sometimes minor, but very important details, like this slip by. George Kadianakis: > My current plan is to keep maintaining the C-Obfsproxy code, if it > keeps being as stable as it is today. For example, I plan to fix two > new-found bugs related to the shared-secret support of obfs2 soon > (#8217, #8223). Excellent. So we do have some time. > That said, I don't plan to port any of the new transports (like obfs3, > and some other pluggable transports currently under development) to > C-Obfsproxy any time soon (even though obfs3 wouldn't be *extremely* > hard to port, and I might consider it if there is a serious reason). Okay, I will take that into consideration. My plan then is to focus on determining how difficult supporting Pyobfsproxy on Android will be, and otherwise, try to find funding and resources to maintain the C version and potential to port obfs3. Thanks! +n

3 2

Re: [tor-dev] Proposal: Controller events to better understand connection/circuit usage
by Damian Johnson 20 Feb '13

20 Feb '13

> AppWard and ExitWard indicate the direction of cell flow through the > circuit. Previous cell events aggregated statistics for both directions and > therefore did not capture information which I believe may be important. > These are much more informative than N and P. Do you have suggestions for > improving this further? Ahhh, gotcha. Personally I'd go for 'inbound' and 'outbound' since they better connotate directionality. AppWard and ExitWard sound weird, but they make sense now too and would be fine if CELL_STATS got a quick description saying what they mean.

2 6

[Patch] or/onion_ntor.c
by Gisle Vanem 19 Feb '13

19 Feb '13

I suggest that the whole body of this file gets included inside an "#ifdef CURVE25519_ENABLED" in-case someone (like me!) inadvertently tries to compile it w/o this define set. Patch: --- Git-latest\src\or\onion_ntor.c Tue Feb 12 18:40:04 2013 +++ or\onion_ntor.c Tue Feb 12 23:46:51 2013 @@ -9,6 +9,8 @@ #include "torlog.h" #include "util.h" +#ifdef CURVE25519_ENABLED /* Rest of file */ + /** Free storage held in an ntor handshake state. */ void ntor_handshake_state_free(ntor_handshake_state_t *state) @@ -292,4 +294,4 @@ memwipe(&s, 0, sizeof(s)); return bad ? -1 : 0; } - +#endif /* CURVE25519_ENABLED */ --gv

2 1

changes to vidalia / browser launch in TBB?
by Greg Norcie 14 Feb '13

14 Feb '13

Hi all, I'm working on a followup to my usability paper from HotPETS... (http://petsymposium.org/2012/papers/hotpets12-1-usability.pdf) and doing some fact checking for the write up. I'm trying to find out if the developers tweaked Vidalia since v2.2.35-7.1. In my tests it seems like there is much less of a lag between Vidalia launching and the TBB window opening. I was hoping to find out if this was an intentional code change on the developers part, since it elliminates one of the isssues mentioned in my paper. -- -- Greg Norcie (greg(a)norcie.com) GPG key: 0x1B873635

1 0

Proposal: Controller events to better understand connection/circuit usage
by Karsten Loesing 12 Feb '13

12 Feb '13

Hi everyone, here's a proposal that defines three new controller events for the TestingTorNetwork mode that shall help us better understand connection and circuit usage in private Tor networks with the goal to make simulations more accurate. And there's also code: https://gitweb.torproject.org/karsten/tor.git/shortlog/refs/heads/morestats2 Feedback, to both proposal and code, much appreciated! Thanks, Karsten Filename: xxx-usage-controller-events.txt Title: Controller events to better understand connection/circuit usage Author: Rob Jansen, Karsten Loesing Created: 2013-02-06 Status: Open Target: 0.2.5.x 1. Overview This proposal defines three new controller events that shall help understand connection and circuit usage. These events are designed to be emitted in private Tor networks only. This proposal also defines a tweak to an existing event for the same purpose. 2. Motivation We need to better understand connection and circuit usage in order to better simulate Tor networks. Existing controller events are a fine start, but we need more detailed information about per-connection bandwidth, processed cells by circuit, and token bucket refills. This proposal defines controller events containing the desired information. Most of these usage data are too sensitive to be captured in the public network, unless aggregated sufficiently. That is why we're focusing on private Tor networks first, that is, relays that have TestingTorNetwork set. The new controller events described in this proposal shall all be restricted to private Tor networks. In the next step we might define aggregate statistics to be gathered by public relays, but that will require a new proposal. 3. Design The proposed new event types use Tor's asynchronous event mechanism where a controller registers for events by type and processes events received from the Tor process. Tor controllers can register for any of the new event types, but events will only be emitted if the Tor process is running in TestingTorNetwork mode. 4. Security implications There should be no security implications from the new event types, because they are only emitted in private Tor networks. 5. Specification 5.1. Adding an ID field to ORCONN events The new syntax for ORCONN events is: "650" SP "ORCONN" SP (LongName / Target) SP ORStatus [ SP "ID=" ConnId ] [ SP "REASON=" Reason ] [ SP "NCIRCS=" NumCircuits ] CRLF ConnID is the connection ID which is locally unique among all connection types and which is only included in TestingTorNetwork mode. The remaining specification of that event type stays unchanged. 5.2. Bandwidth used on an OR or DIR or EXIT connection The syntax is: "650" SP "CONN_BW" SP ConnID SP ConnType SP BytesRead SP BytesWritten CRLF BytesRead = 1*DIGIT BytesWritten = 1*DIGIT ConnID is the connection ID which is locally unique among all connection types. ConnType is the connection type, which can be "OR" or "DIR" or "EXIT". BytesWritten and BytesRead are the number of bytes written and read by Tor since the last CONN_BW event on this connection. These events are generated about once per second per connection; no events are generated for connections that have not read or written. These events are only generated if TestingTorNetwork is set. 5.3. Per-circuit cell stats The syntax is: "650" SP "CELL_STATS" SP PCircID SP PConnID SP PAdded SP PRemoved SP PTime SP NCircID SP NConnID SP NAdded SP NRemoved SP NTime CRLF PCircID and NCircID are the locally unique IDs of the app-ward (PCircID) and exit-ward (NCircID) circuit. PConnID and NConnID are the locally unique IDs of the app-ward (PConnID) and exit-ward (NConnID) OR connection. PAdded and NAdded are the total number of cells added to the app-ward (PAdded) and exit-ward (NAdded) queues of this circuit. PRemoved and NRemoved are the total number of cells processed from the app-ward (PRemoved) and exit-ward (NRemoved) queues of this circuit. PTime and NTime are the total waiting times in milliseconds of all processed cells in the app-ward (PTime) and exit-ward (NTime) queues of this circuit. PAdded, NAdded, PRemoved, NRemoved, PTime, and NTime are semicolon-separated key-value lists with keys being lower-case cell types and values being cell numbers or waiting times. These events are generated about once per second per circuit; no events are generated for circuits that have not added or processed any cell. These events are only generated if TestingTorNetwork is set. 5.4. Token buckets refilled The syntax is: "650" SP "TB_EMPTY" SP ["GLOBAL" || "RELAY" || "ORCONN" SP ConnID] SP ReadBucketEmpty SP WriteBucketEmpty SP LastRefill CRLF This event is generated when refilling a previously empty token bucket. The "GLOBAL" and "RELAY" keywords are used for the global or relay token buckets, the "ORCONN" keyword together with a ConnID is used for the token buckets of an OR connection. If both global and relay buckets and/or the buckets of one or more OR connections run out of tokens at the same time, multiple separate events are generated. ReadBucketEmpty (WriteBucketEmpty) is the time in millis that the read (write) bucket was empty. LastRefill is the time in millis since the last refill. ReadBucketEmpty or WriteBucketEmpty are capped at LastRefill in order not to report empty times more than once. These events are only generated if TestingTorNetwork is set. 6. Compatibility There should not be any compatibility issues with other Tor versions. 7. Implementation Most of the implementation should be straight-forward. There's one exception: we pondered adding a unique circuit ID to CELL_STATS events, but so far, only origin circuits have a unique ID. We could move that field from origin_circuit_t to circuit_t and update all references in the code. But this may have undesired side effects which we're not yet aware of. We don't have a good answer yet if we need this ID or not. 8. Performance and scalability notes Most of the new code won't be executed in normal Tor mode. Wherever we needed new fields in existing structs, we tried hard to keep them as small as possible. Still, we should make sure that memory requirements won't grow significantly on busy relays.

5 14

[Stegotorus] Dropping proxy test result
by vmonmoonshine＠gmail.com 10 Feb '13

10 Feb '13

Hi Zack, I added --drop-rate option to the tester-proxy and committed it to f--bad-proxy-ack-tester branch. I confirm that Stegotorus fails webpage-tester test when the drop-rate >= %10. Now, I'll go and look at the problem in more depth. Generally speaking, firefox is less successful in fetching web pages than webpage_tester, i.e. curl. I don't know why but I hope that problem will go away, when the problem with ack is solved. And on this: > Speaking of, vmon, I know you had trouble getting libcurl and libevent > to play nice -- did you ever look at libevhtp? First glance, I don't think it's helpful. It's an http server, isn't it? I need a http client which send the GET Request but let libevent to read the replay (only write in the socket). Hopefully I'll have more useful information soon. Cheers, Vmon

1 0

Documenting the translation process
by Runa A. Sandvik 07 Feb '13

07 Feb '13

Hi everyone, I started documenting how the translation process works for translators, developers, and those who manage it all: https://trac.torproject.org/projects/tor/wiki/doc/translation/Introduction. Any feedback will be greatly appreciated. Thanks, -- Runa A. Sandvik

1 0

Re: [tor-dev] [tor-talk] Exit Node Scanning Status
by Damian Johnson 03 Feb '13

03 Feb '13

> Actually, I might be a good fit. Feel free to ping me off-list to discuss. Unless there's a reason that you'd like to make this private it's better to have development discussions in the open (though tor-dev@ is probably more appropriate). Imho writing a SoaT counterpart based on either stem [1] or txtorcon [2] would be a great project if you're interested in improving our bad-exit detection infrastructure. Cheers! -Damian [1] https://stem.torproject.org/ [2] https://txtorcon.readthedocs.org/

4 5