- tor-dev - lists.torproject.org

[Stegotorus] Stegotorus stuff
by vmonmoonshine＠gmail.com 24 Mar '13

24 Mar '13

Hello TorDev, Here, I'm reporting my recent progress on Stegotorus as well as answering some questions from past. During past month, I mainly worked in "retransmit = false" mode trying to restore reliable browsing. I have resolved quite few bugs. Some of them where due to incomplete implementation of "retransmit = false" mode after implementing retransmission. But the major one seems to be the flush_interval implementation: if server had transmitted data to the client, the client opens a new connection, "probably" in near future, to retrieve more data. The probability follows geometric distribution to simulate user browsing behavior. This is OK most of the time. But when you transmit 1000s of packets, inevitably it happens that it samples delays such 40-60 seconds. Unfortunately, Firefox isn't patient enough to wait so long to receive the rest of the data and judge the connection as dead. I changed this now that if it real data came from the server on previous connection, the client opens a new connection 100msec which I know isn't the point but I wanted to restore reliability. The better way might be to schedule opening few connections at a time (instead of one). In this way it is very improbable that all of them sample unacceptable delays. I am going to debug retransmission in the next step. Now few questions, that I was putting off partialy cause I was tired talking and I wanted to do something practical before talking more: From: Roger Dingledine <arma(a)mit.edu> on Sun, 13 Jan 2013 13:04:23 -0500: > 3) What's the overhead of putting your Tor traffic through each of the > steg modules? Now the chopper is measuring the efficiency of the steg modules based on the following measure for each circuit: (no of byte transmitted from upstream)/(no of bytes transmitted downstream ino order to transmit upstream bytes) The summary of the results measured using speedtest.net: |-------------+----------+---------+------+------| | | Mb/s | Mb/s | Efficiency | |-------------+----------+---------+------+------| | Steg module | Download | Upload | S->C | C->S | |-------------+----------+---------+------+------| | Direct | 19.52 | 2.08 | 1 | 1 | | nosteg | 12.98 | 2.26(!) | 0.98 | 0.98 | | http | 0.28 | 0.31 | 0.03 | 0.07 | | http_apache | 1.43 | 0.47 | 0.65 | 0.52 | |-------------+----------+---------+------+------| (!) The faster upload of nosteg than the direct connection with Stegotorus is persistent and I can't figure out why. it might be a bug with speedtest.net algorithm or a their weird notion of speed. If you are downloading, the server->client efficiency is like this: |-------------+-----------------| | Steg module | S->C Efficiency | |-------------+-----------------| | Direct | 1 | | nosteg | 0.98 | | http | 0.10 | | http_apache | 0.65 | |-------------+-----------------| The efficiency of course varies depending on the cover texts. These are measured for http module using the fake cover generated by zack and for http_apcahe using random javascripts that I have downloaded plus the pdf files I had on my computer. The advantage of http_apache over http steg module is due to http_apache always chooses the most efficient cover possible verses http module which randomly chooses 10 candidates and then chooses the best one among them. The http_apach approach, however, isn't security-wise plausible. The best solution seems to be hybrid of the two with user having a say how efficient/secure wants it to be. With all my cat and mouse with curl, it still manages to steal data from libevent and I lose 1 in 100 packets or so. In presence of retransmission this is not a big deal but I have to solve it in more fundamental way. From: George Kadianakis <desnacked(a)riseup.net> on Wed, 30 Jan 2013 14:09:35 +0200: > The code monkey inside me doesn't see any code-quality-related tickets > in that URL. stegotorus boasts 12k LoC of C++ and some of that code > was written in haste. You can divide the ST code into three major parts. Network, chopper and the steg modules. The network is basically, obfsproxy. For the chopper, beside Zack's obsession with overloading functions with the same name by changing type/no of parameters, I think the rest is a pretty good standard code and well commented (I suspect it is very similar to a typical TCP implementation). It lacks is a flow chart that explain what exactly happens when data is received or sent. The ugly code are the steg modules, (partially because it is in C rather than C++), but then the steg modules are not innate part of Stegtorus and are meant to be re-implemented by interested party. Having said that, Zack has mentioned that hardening the code is number 1 priority. Just because he mentioned it later, I haven't included it in the track list yet. > Looking at this from a deployment point of view, > I feel much more relaxed and easygoing with deploying programs written > in a high-level language (like Flashproxy or pyobfsproxy) than with > deploying an unaudited C++ program with many bad code moments. I suspect that python inefficiency will break Stegotorus completely. Now when I run Stegotorus it eats all CPU time. I know this is partially due to heavy logging and partially because I compile it with zero optimization and full debug but I can only imagine how it would look like if it was written in python. There are lots of exact timing there in scope of msec. These are not important when you are running nosteg module which is basically obfsproxy with variable packet size. But when you turn to simulating http with multi-connections and constant negotiation of chopper with steg modules, I doubt if python is an option. Beside, the steg algorithms can get quite complicated. > I think that a trac ticket with a bit of research on how painful would > be to write a deployable <high-level language>-port of stegotorus > would be a good idea. I know that Zack is also interested in exploring > this avenue. I haven't looked into pyobfsproxy, but translating the chopper into python over pyobfsproxy doesn't seems to be an impossible task (you are looking at around 3000 LoC). But then you still need some steg module to get something more than pyobfsproxy (if we close our eyes on the preformance doubts I raised). Maybe, having a cython/c interface to steg module be a middle way compromise. Wow I talk too much! I'll spare everybody for now. Vmon

1 0

Does anyone need python 2.5 support in stem?
by Damian Johnson 22 Mar '13

22 Mar '13

Hi all. Will anyone be sad if I drop python 2.5 support from stem (just supporting 2.6 and higher, including the 3.x series)? If all goes well stem will be having its first release this next weekend and my thinking is this... * Python itself considers python 2.5 to be obsolete, having dropped docs for everything below 2.6. * Every OS I can think of except maybe CentOS has at least 2.6, and more likely 2.7. * I have a dozen dirty hacks throughout the codebase just for 2.5 compatability and I'd love to get rid of them. I'm presently working on a bug that needs the io module which was introduced in 2.6, so I'll need another gross hack for 2.5 support after this too... :( * Testing with 2.5 is a pita for me, so in general I don't (only when a 2.5 bug is reported or prior to a release). * I've never heard of someone actually using stem with python 2.5, so maintaining backward compatability this far back is likely a waste of time. Cheers! -Damian

2 1

Stem Tutorials
by Damian Johnson 19 Mar '13

19 Mar '13

Hi all. Stem is having its initial release later this month, and in anticipation of it I've been expanding its tutorials... https://stem.torproject.org/tutorial.html Are there any rough spots in these tutorials? Other categories that we should cover? Suggestions welcome! -Damian

3 3

Proposal: Capturing Traffic Statistics from Exit Relays
by Zack Weinberg 18 Mar '13

18 Mar '13

[This draft proposal is also available at git.torproject.org:users/zwol/torspec.git/proposals/ideas/xxx-exit-statistics.txt.] Filename: xxx-exit-statistics.txt Title: Capturing Traffic Statistics from Exit Relays Author: Zack Weinberg Created: 04-Feb-2013 Status: Draft 1. Motivation We propose to collect additional aggregate traffic information from exit nodes which will be of use to researchers, similar to how the project already collects statistics about its user population. We also propose to bring existing entry and exit statistics under the same umbrella so that anonymity protection can be applied in a principled fashion to the entire data set. There is some prior discussion in tickets #6002 and #6003. Exit nodes currently collect statistics on the destination TCP ports of exiting traffic, but this is insufficient information for research; in particular, there have been repeated requests for information about destination hosts. Individual researchers have instrumented exit nodes on an ad-hoc basis (e.g. [1]) but the project has historically been skeptical about both the value and the safety of doing so (e.g. [2] warns of running afoul of wiretapping statutes; I understand that the [1] paper was met with severe criticism, but I can't find that right now). A principled, network-wide policy and mechanism for collecting exit metrics can satisfy the demand for data for research purposes while protecting our users and minimizing additional legal risk to exit node operators. 2. Design At each exit node, we propose to measure the number of exiting TCP connections and total bytes transferred in each direction, per day, categorized three different ways: * TCP port * "Public suffix" + 1 domain component of destination (example.com, example.co.uk) * Country of IP address of destination * possibly: ASN of destination and ASN of exit (this may reveal potential traffic-correlation attackers that would not be visible any other way, but may also be too fine-grained to be safe). These are sufficiently coarse-grained that we believe they can be safely measured without damaging the network's anonymity guarantees; see below for further measures that will be taken to preserve anonymity. At the same time, they will enable interesting analyses of what the network is used _for_, in much the same way that the existing entry-side metrics enable analysis of _who_ uses the network. At the same time, we propose to rationalize entry-side data collection, which currently relies on directory queries rather than actual traffic to actual entry nodes. This will improve accuracy and will also allow us to apply an anonymity-protection algorithm consistently to the entire data set available from the Metrics server. Entry nodes (including bridges) should record entering TCP connections and traffic volume, per day, categorized by: * Country of IP address of traffic source * ASN of IP address of traffic source, and ASN of entry node (subject to same caveat as above) * "I am a bridge" flag (I think this is a superset of the information currently collected via directory queries. If I am mistaken, please let me know.) All collected data should be passed through a differential-privacy sanitization algorithm before it leaves the Tor process's memory, and should then be uploaded to a central server (probably via a write-only hidden service API) which applies a second layer of sanitization. See below for further discussion. 3. Security implications Any collection of information about the operation of the Tor network requires careful analysis to ensure that the anonymity of its users is preserved. In this case, we are contemplating adding to what is publicly available about what the network is used _for_, and we must ensure that this does not lead to a correlation attack that would reveal _who_ did a particular thing. For instance, suppose the adversary knows that _someone_ posted a "sensitive" document on a particular site on a particular date. They also have reason to suspect that the poster hails from a particular country. If our published metrics reveal that only one person used Tor to access that site on that date, and/or that only one person from that country made use of Tor on that date, the adversary's case against their suspect has now gotten considerably stronger. The theoretical framework that deals with this class of exposure is called _differential privacy_. It rests on two observations. First, it is impossible to guarantee that _no one's_ privacy will be compromised via the release of statistics -- consider the somewhat contrived but still evocative case where the adversary already knows that Alice is 6cm shorter than the average citizen of Ruritania; publication of the Ruritanian average height reveals Alice's actual height. Note that Alice _does not_ have to be included in the average for this to work! But the adversary had to know something about Alice already. This leads to the second observation, which is that we _can_ make a reasonable privacy guarantee by changing the model. Instead of trying to protect everyone _whether or not_ they are in the database, we guarantee that, with high probability, the _released statistics_ would not be changed by the inclusion or removal of any one individual from the underlying data set. (This can be generalized to groups of N individuals, although of course the statistics get less and less accurate as N goes up.) Differential privacy prevents the example scenario described above: the adversary would only learn a _range_ of possible values for 'how many people visited site X today via Tor' and would therefore not be able to draw new conclusions about their suspect. The basic technique of differential privacy is to add "noise" to the aggregate statistics before publishing them. There are a wide variety of "mechanisms" for doing so, of which the most basic is simply to add "independent samples from a Laplace distribution" to each statistic. Research in this area focuses on finding mechanisms that minimize the amount of noise required to provide a given level of privacy (according to a security parameter, 'epsilon' as usual). [3] is a good overview of the field and [4] proposes a particular algorithm that seems well-suited for the data set we're looking at here. We must also consider the security of the reporting process. The Metrics server is already trusted, and applying "noise" there will allow us to minimize the overall amount of noise applied for the same level of differential privacy. However, if this is the only place noise is applied, a malicious node operator could capture the 'raw' statistics by reconfiguring their node to report to a server that doesn't sanitize (but perhaps forwards the report to the real server, avoiding suspicion). If each node applies some level of noise before the statistics ever leave the Tor process's memory, this class of attack is much harder, although not impossible (by modifying code or extracting raw statistics from process memory with a debugger). There is at least one proposal [5] for distributed sanitization of a data set. It is Byzantine fault-tolerant and therefore _may_ be overkill for our purposes ... but we're talking about malicious node operators here, so maybe it isn't! 4. Specification TBD 5. Compatibility should be moot 6. Implementation TBD 7. Performance and Scalability minor overhead at each exit (and entry); additional load on Metrics; if reporting is via hidden service, additional load on the network 8. References [1] "Shining Light in Dark Places: Understanding the Tor Network". McCoy, Bauer, Grunwald, Kohno, Sicker. PETS 2008. http://freehaven.net/anonbib/cache/mccoy-pet2008.pdf [2] https://www.torproject.org/eff/tor-legal-faq.html.en [3] "Differential Privacy for Statistics: What we Know and What we Want to Learn". Dwork and Smith. Journal of Privacy and Confidentiality, 2009. http://repository.cmu.edu/jpc/vol1/iss2/2/ [4] "Optimizing [Linear/Histogram] Counting Queries under Differential Privacy". Li, Hay, Rastogi, Miklau, McGregor. Principles of Database Systems, 2010. http://arxiv.org/abs/0912.4742 [5] "Our data, ourselves: Privacy via distributed noise generation". Dwork, Kenthapadi, McSherry, Mironov, Naor. EUROCRYPT 2006. http://www.truststc.org/pubs/101/ourDataOurselves.pdf

2 2

Obfs Links on dist are Suddenly Gone
by Nima 15 Mar '13

15 Mar '13

Hi, I've largely distributed mirror links to pyobfsproxy specially for Windows in Iran. Like this: http://torproject.ph3x.at/dist/torbrowser/tor-flashproxy-pyobfsproxy-browse… Simply because this is the only thing that works. But suddenly they're all gone!!! Is this on purpose or it's just an accident? People are asking for new links. what should we do now? :/ -- Nima 0x1C92A77B "I disapprove of what you say, but I will defend to the death your right to say it" --Evelyn Beatrice Hall

4 4

Interested in contributing to Tor-Dev through GSoC
by Ashish Patil 10 Mar '13

10 Mar '13

Hello! I am a newbie & very eager to contribute to Tor development. I have been using tor since a year and a half & I have been looking around the website & the source codes since past month. The upcoming GSoC has provided a very nice opportunity for me contribute to whatever extent I can. I have interest towards Stem & Obfsproxy due to their applications & would like some beginner's guidance for the same. Also, I would appreciate if anyone could provide me some online references that would help me understand networking better (in order to understand the source code fully, also is that completely necessary?) PS - I am very well acquainted in Python & have also contributed to it's development but only in Documentation aspect, hence have good knowledge of revision control, building, etc. Thanks! -Ashish Patil

3 6

Interested in contributing to tor through GSoC
by Anirudh Vemula 09 Mar '13

09 Mar '13

Hi, I use tor everyday in my college as we have a http proxy which blocks specific sites. It has been a huge help for me and I would like to improve it by contributing to it by implementing new ways and resolving bugs. I would like to do this year's GSoC with the tor organisation and I felt it would be better for me if i get acquainted with the source code well beforehand. I need suggestions on how to start going about understanding the source code. I would love if there are bugs and mini-projects that a beginner could resolve and improve my understanding of the source code. Thanks Anirudh V Anirudh Vemula | Junior Undergraduate | Computer Science & Engineering | vvanirudh.blog.com | Alumni Secretary Hostel 2 | IIT Bombay

2 1

[ooni] proposed specification for ooni backend component
by Arturo Filastò 06 Mar '13

06 Mar '13

Greetings! It would be great to get some feedback on the specification of oonib that is the backend component of ooniprobe. As later said, not everything described here is implemented, but this document details what oonib should become. Moreover if there is something that you believe we should pay special attention to in when implementing this it would be of great use to know. If you wish to modify this spec directly you can send us a pull request on github: https://github.com/TheTorProject/ooni-spec ------------------------------------- # oonib specification * version: 0.1 * date: 2013-03-04 * author: Arturo Filastò This document aims at providing a functional specification of oonib. At the time of writing this document not all parts are fully implemented, though the application interface to oonib is. # 1.0 System overview oonib is the backend component of ooni. It is responsible for: * Collecting the results of tests from ooni-probes (Collector). * Exposing a set of services that are needed for allowing ooni-probes to perform their tests (Test Helpers). # 2.0 Collector ## 2.1 System overview The oonib collector exposes a JSON RPC like HTTP interface and allows probes to submit the results of their measurements. Once probe measurement is complete the test result is published as an ooni test report in the YAML format. The oonib collector shall be exposed as a Tor Hidden Service and as a HTTPS service. ## 2.2 Theat model The collector shall provide end-to-end between the probe and the oonib collector. A malicious actor should not be able to update test results that they have not created. It is outside of the scope of the oonib collector to provide blocking resistance or to conceal to a passive network observer the fact that they are communicating to a collector. Such properties are to be provided by other software components, for example using Tor or obfsproxy. ## 2.3 Test result submission interface Unless otherwise stated all of the network operations below can be performed either via HTTPS or HTTPO (HTTP over Tor Hidden Service). Note: we will eventually want to migrate over to using YAML instead of JSON as a data exchange format. Not doing so adds unnecessary overhead in including YAML data inside of JSON data. ### 2.3.1 Create a new report When a probe starts a test they will *create* a new report with the oonib collector backend. The HTTP request it performs is: `POST /report` { 'software_name': `string` the name of the software that is creating a report (ex. "ooni-probe") 'software_version': `string` the version of the software creating the report (ex. "0.0.10-beta") 'probe_asn': `string` the Authonomous System Number of the network the test is related to prefixed by "AS" (ex. "AS1234") 'test_name': `string` the name of the test performing the network measurement. In the case of ooni-probe this is the test filename without the ".py" extension. 'test_version': `string` the version of the test peforming the network measurement. 'content': (optional) `string` it is optionally possible to create a report with already some data inside of it. 'probe_ip': (optional) `string` the IP Address of the ooniprobe client. When the test requires a test_helper the probe should inform oonib of it's IP address. We need to know this since we are not sure if the probe is accessing the report collector via Tor or not. } The server will respond with a report identifier that will then allow the probe to update the report and the version of the backend software like follows: { 'backend_version': `string` containing the version of the backend 'report_id': `string` report identifier of the format detailed below. 'test_helper_address': `string` the address of a test helper for the requested test. } The report identifier allows the probe to update the report and it will be contructed as follows: ISO 8601 timestamp + '_' + probe ASN + '_' + 50 mixed lowercase uppercase characters A report identifier can be for example: 1912-06-23T101234Z_AS1234_ihvhaeZDcBpDwYTbmdyqZSLCDNuJSQuoOdJMciiQWwAWUKJwmR Note: The report identifier should be at least 256 bits and generated by means of a CSPRNG. Client implementation notes: Probes should not expect the report identifier to be in a particular format as the report id may be changed in the future. ### 2.3.2 Update a report Once the probe has a report ID they will be able to add test related content to the report by referencing it by id: `PUT /report` { 'report_id': `string` the report identifier 'content': `string` content to be added to the report. This can be one or more report entries in the format specified in df-000-base.md } The backend should validate the request to make sure it is a valid YAML Stream. New collectors should use the following format for updating reports: `POST /report/<report_id>` { content: `string` content to be added to the report. This can be one or more report entries in the format specified in df-000-base.md } ### 2.3.3 Closing a report This request is done by a probe to tell the backend that they have finished running the test and the report can be considered done: `POST /report/<report_id>/close` To create a new report a probe will peform an HTTP POST request to the resource /report. The collector MUST implement the following HTTP JSON RPC like API: /report ## 2.4 Report lifecycle When a report is created (section 2.3.1) it should be marked as NEW and a timestamp should be recorded. Once it is updated (section 2.3.2) it should be marked as ACTIVE. An ACTIVE report can be updated with new data. The collector must keep track of the last time a certain report is updated with new and valid data. A report is considered CLOSED when either the probe instructs the collector to close the report (section 2.3.3) or when a report is in ACTIVE state and has not been updated with valid data for more than 2 hours. ## 2.5 Report publishing and cleaning Once a report is closed it should be made available to the public for download and analysis. This shall happen as soon as a report reaches the CLOSED state. Reports should be discarded and deleted if: * They have been in the NEW state for more than 4 hours * They are CLOSED, but contain only one Report Entry meaning that the report entry contains only the base test data (see df-000-base.md). Reports should be published to: `https://ooni.torproject.org/reports/` **reportFormatVersion** `/` **CC** `/` Requesting such URL may also result in a 302 to the location of reports for that specific country. Where CC is the two letter country code as specified by ISO 31666-2. For example the reports for Italy (CC is it) of the reportVersion 0.1 may be found in: https://ooni.torproject.org/reports/0.1/IT/ This directory shall contain the various reports for the test using the following convention: test name - timestamp in ISO8601 format - probe AS number - probe|backend.yamloo The timestamp is expressed using ISO 8601 including seconds and with no : to delimit hours, minutes, days. Such date is the time in which the report was created and must be set by the backend. Like so: YEAR - MONTH - DAY T HOURS MINUTES SECONDS Z The time is always expressed in UTC. If a collision is detected then an int (starting with 1) will get appended to the test. For example if two report that are created on the first of January 2012 at Noon (UTC time) sharp from MIT (AS3) will be stored here: https://ooni.torproject.org/reports/0.1/US/http_test-2012-01-01T120000Z-AS3… https://ooni.torproject.org/reports/0.1/US/http_test-2012-01-01T120000Z-AS3… Implementation notes: The task of publishing a report should be made modular so that we can replace the publishing mechanism if we discover the limitations of this system (not infinite disk space). The basic implementation shall just scp the files to a machine that is configurable by config file. ?? Question: How does this integrate into the m-lab infrastructure? How will this work when some reports results are stored on m-lab and some are not? ## 2.6 Test helper collector relationship Some tests involve adding to the report also data that is collected by means of a test helper. This is the case in the two way traceroute test, where the report should include a traceroute also from the vantage point of the ooni backend. In these circumstances the report from the vantange point of the backend should be inside of a separate file that has the same name of the probe report but "-probe" should be replaced with "-backend". For example the backend part of the report for a traceroute test called `two_way_traceroute-2012-01-01T120000Z-AS3-probe.yamloo`, shall be called `two_way_traceroute-2012-01-01T120000Z-AS3-backend.yamloo` # 3.0 Test Helpers These are services exposed to ooniprobe clients that are of assistance to performing network measurements. # 3.1 System overview Probes will always receive as address of a test helper that of the one running on the same machine as the collector. They will only point to a test helper on a different machine if the test helper for the test the user is interested in running is not available on the desired machine. This can happen, for example, if the machine does not have two network interfaces with two differnet IP addresses. In this case the HTTP Return JSON Headers test helper cannot run at the same time as the TCP echo test helper (both bind to port 80). Some communication with the collector is required. This is the case of the two way traceroute test, where a multiport multiprotocol traceroute must also be performed from the backend to the probe. Test helpers are two kinds: * Reply: are test helpers that reply to requests from probes. For example HTTP test helpers are of this kind. * Active: are test helpers that actively perform requests towards the probe idepedently from probe requests. Implementation notes: Although I am talking about the collector as two differnet software components they both run inside of the same process and are part of the same piece of software. ## 3.2 Threat model (or non-goals) Because of the nature of the services that they are exposing it is not possible to guarantee end to end confidentiality and authentication of the data transmitted to and from test helpers. Moreover we are currently not making any particular effort to make test helpers look like something that they are not (i.e. make test helper traffic not look like test helper traffic). ## 3.3 Test Helper collector mapping When a report that requires a test helper is created with the collector component of oonib the test helper should be notified of the probes IP address (the probe_ip). This will allow the test helper to know from which IP address it should either expect the probe to come from or towards what IP address it should perform an active measurement. ## 3.4 Reply Test Helpers ### 3.4.1 HTTP Return JSON Headers This test helper will bind on port 80 and expect HTTP requests from a probe. It shall respond to every HTTP request with the HTTP Headers and HTTP request line as seen from the backend point of view. The response is structured in JSON as follows: { 'request_headers': [[HTTP header1 name, HTTP header1 value], [HTTP header2 name, HTTP header2 value]] the list is ordered based on how the headers were received. 'request_line': the value of the HTTP request line. 'headers_dict': `dict` containing as keys the HTTP header name (normalized) and as value a list containing the values of such header } For example: { 'request_headers': [['User-Agent', 'IE6'], ['Content-Length', 200]] 'request_line': 'GET / HTTP/1.1' 'headers_dict': {'User-Agent': ['IE6'], 'Content-Length': [200]} } ### 3.4.2 DNS Test Helper Shall provide a DNS resolver over UDP and TCP exposed on port 53. Such DNS resolver shall not filter any DNS query. ### 3.4.3 TCP Echo Helper This shall expose a TCP echo service that is bound to port 80. ## 3.5 Active Test Helpers ### 3.5.1 Two way traceroute This shall perform the ooniprobe traceroute test and attach the result to the final report as described in section 2.5.

1 0

transparent proxy detection
by Sam Smith 06 Mar '13

06 Mar '13

Hey all, I'm looking at detection and identification of some transparent proxies (of different kinds). Has anyone already got tests that effectively but generically merge manipulation/http_host and experimental/squid.py ? Also, I'm testing on a debian (6.0) VM running in a parallels 8 on OS X. Ooni successfully runs tests once , and then requires a debian reboot in order to run again (especially http_host, it was less repeatable with others). I have no idea what's going on, and it'll be a pain to debug, but it's the sort of thing someone's probably interested in for other purposes ;) thanks for ooni Best Wishes Sam -- @smithsam

2 1

Building Tor Browser Bundle on Windows
by Tom Ritter 05 Mar '13

05 Mar '13

This weekend I tried building several of the tor projects on Windows (tor, vidalia, tbb, and vanilla firefox). I kept notes of my adventures, some of the quirks I ran into, and the final set of instructions that seemed to work for me. Perhaps most usefully, I've created an Amazon AMI that anyone will be able to boot up and use. (Yes, this has trust issues, I've addressed them in the top of the README document attached.) The intention is that people who want to test a patch or develop a patch can use this to bootstrap themselves, without having to fight long and hard to set up a Windows build environment. I've attached two documents: 1) TBB Environment README.txt 2) Adventures in Compilation.txt The first is the README for the environment. The second is a journal of my trying to figure out what to do. Specifically I'm going to call out several issues. If desired, I can submit patches to address them, and they can be reviewed. I understand that documentation is usually a lower priority than actually shipping software, please take these comments not as bitchy criticism, but instead a list of things I'd like to improve, if someone can review my suggestions and shepard me a little. 1) Firefox's official build environment is VS2010 while TBB's is VS2008 I'm not 100% sure, but it seems like it'd be relatively easy to switch TBB over to 2010. There may be some issues. But it seems like it'd be good to track FF's official build env. I ran across at least 1 Vanilla FF issue that broke in 2008, and there seemed to be a few in the bugtracker. 2) Vidalia's build environment instructions contradict TBB's https://gitweb.torproject.org/vidalia.git/blob/HEAD:/INSTALL https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/docs/buildmachine_s… Vidalia's says you need CMake in your PATH, TBB's says you don't. I think it can be in your PATH. 3) There's no explicit instructions on how to build Tor in Windows It seems like you just build it in MinGW? It worked anyway. It's a (minor) bitch to get the components built and placed in the 'correct way and place' for the MinGW toolchain. (Now documented in my README) 4) The TorBrowser build instructions are really, really bad. https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/docs/buildmachine_s… This is pretty good. But it just ends abruptly, and doesn't tell you want to do. https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/docs/HACKING This document seems to be mostly up to date, although I think the FF Extensions it lists have been changed since. https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/build-scripts/INSTA… This one is super out of date. - You don't need subversion or winrar - It talks a lot about getting FirefoxPortable (or built a custom Firefox) - I didn't try following the instructions for MinGW, msys, cmake, qt, msysGTK, or autoconf - since I had those already. It seems these sections have been subsumed by the windows buildmachine document - The OpenSSL instructions do not match the instructions IN OpenSSL - Vidalia's links are way out of date and dead - Its instructions for Vidalia don't match Vidalia's - It talks about PidginPortable? I think that's been nixed... - "Currently the Tor Browser Bundle uses FirefoxPortable binaries." - It talks about building with MSVC 7.1 (Visual Studio 2003) - It stops short of telling you how to actually build the bundle, probably because the makefiles are newer than this document https://gitweb.torproject.org/torbrowser.git/blob/HEAD:/build-scripts/DEPLO… This doesn't seem applicable at all anymore I eventually found (by searching my email spools) the magic incantation to kick it off. The "TorBrowser" section of Adventures in Compilation.txt shows my hiccups and luck getting it going. Finally, after all this.... tbb-firefox crashes. I don't know why. I don't know if it's because I'm using VS2010, or my msvc*.dll swaps. Specifically it's crashing here: c:\build\torbrowser\build-scripts\build-alpha-windows\build\firefox-17.0.3esr\content\xul\templates\src\nsrulenetwork.h class Iterator : public ConstIterator { public: Iterator(ReteNode** aNode) : ConstIterator(aNode) {} Iterator& operator++() { ++mCurrent; return *this; } Iterator operator++(int) { Iterator result(*this); ++mCurrent; return result; } ReteNode* operator*() const { return *mCurrent; } <========================================= ReteNode* operator->() const { return *mCurrent; } bool operator==(const ConstIterator& aConstIterator) const { return mCurrent == aConstIterator.mCurrent; } bool operator!=(const ConstIterator& aConstIterator) const { return mCurrent != aConstIterator.mCurrent; } }; I'm not quite sure where to start figuring out *why* it's dereferencing a null ptr. The windbg output is in the bottom of both docs. The AMI for the EC2 instance is ami-2e36ab47 - when you start it, open ports 3389 and 8080 in the firewall for 0.0.0.0/0 (or just your IP address if you want.) If you try it, and get skeeved out by them asking for your SSH private key to get the password, check out https://github.com/tomrittervg/decrypt-windows-ec2-passwd -tom

5 5