- ooni-dev - lists.torproject.org

Notes from meeting today
by Meredith Whittaker 27 Jun '13

27 Jun '13

CCing all those who's emails I have, and the list. This is what I jotted down from the meeting today. I'm not sure if there's a place these should go, or if there's a better way. For now, add things I missed and correct things I got wrong. I'm assuming we're back on for the regular time, Monday, 12:00pm, EDT, featuring Tom :) Cheers, M Art 'n Aaron have merged the user interface, worked on the backend collector policy (how to allow collectors to set their own policies, etc.), and worked on code to handle downloading and caching. This work will continue into next week. There were some questions about M-Lab NS. Meredith (hi) will CC Art and Aaron on an email with the M-Lab NS developers. Nathan has been working on the threat model, now updated and expanded in the wiki: https://github.com/TheTorProject/ooni-probe/wiki/Threat-Model In a week or two, Nathan estimates he and Daira will be ready to present an overview to the team, either during the regular meeting or some other time. Once the threat model is complete, it will be the responsibility of the broader team to update and revamp as decisions are made and things change. If anyone would like to follow progress in real time, pay attention to the wiki changes. Aaron pointed out that we'll want to clearly educate the end-user about the privacy protections (etc.) they can expect from OONI. Nathan noted that he's working on a doc that describes each role (probe runner, OONIB maintainer, data analyst, etc.), and what assumptions they can make about OONI. This can be a start for clearer user-facing documentation. Review of progress toward "Fully Specified" milestone: - URL input processors: done - Triage test inclusion: done, but Meredith to update item with list of tests accepted by M-Lab Steering Committee - Backend specification: Need to add all items that need to be completed for this to be done (Art) - Test helper versioning spec: Not complete. Need to link to document (assuming it exists) that notes current specification (Art) - Threat model: In progress. ETA ~2 weeks. - Design specs: In progress, need to add all items needed to the list (not clear that everything is accounted for currently). - Spec OONI backend HTTP API: Need to review what exists currently, audit what's missing, and add all that are necessary as separate tickets. - Spec how OONI handles OONIB collector policy: Aaron currently working on this -- Meredith Whittaker Program Manager, Google Research Google NYC

3 3

Team Meeting: 2003-06-24 0900 PDT [be in #ooni 10 mins early]
by Tom Lowenthal 27 Jun '13

27 Jun '13

Hello fine folks, On Monday, we will once again meet in a hangout at 0900 to discuss the future of network measurement, and our contribution thereto. Especially if you had any difficulty joining last week, I entreat you to be in #ooni at 0850 so that we can try to get up to code and properly grounded before 0900. Here's my proposal for an agenda. If you have suggestions for additions or subtractions let's wrangle by email, but not not at 0830 on Monday. 0900 Convene: how are folks feeling, how's it going. 0910 Report: core team, what did you do this week, and what are you planning? 0940 Assess: how close are we to our nearest deliverable, what does each of us have to do to get there? 1000 Adjourn No homework this week. Enjoy your weekend! -Tom

7 9

Re: [ooni-dev] Team Meeting: 2003-06-24 (reschedule to 2013-06-26) 0900 PDT [be in #ooni 10 mins early]
by Arturo Filastò 24 Jun '13

24 Jun '13

On Jun 24, 2013, at 6:06 PM, Tom Lowenthal <me(a)tomlowenthal.com> wrote: > Arturo, > > In future, please do not try to reschedule the project meeting less > than three hours before it starts, and especially not because of a > private conversation. There is a reason that I invite schedule > wrangling in advance: it puts everyone on equal footing, and avoids > surprises. If someone would like to move the meeting, they can either > reply to the meeting-notice email on ooni-dev or -- if that's not > possible -- contact me well in advance. > > Please reply so that I know that you understand what I'm asking. Dude, seriously? On Saturday (the 22nd) I got on IRC and told that Aaron and I had an issue with the schedule. Since we did not get a reply there I then sent an email to you and all the other people in the call that we had an issue with the call and got no reply. Aaron also sent an email about this on ooni-dev (https://lists.torproject.org/pipermail/ooni-dev/2013-June/000021.html) and he also got no reply. Since more than 48 hours had passed and nobody expressed their opinion on rescheduling so I decided to reschedule it. I realize it is a bit late to reschedule, but that was the soonest I was able to get online and send an email (from the airport wifi in Dusseldorf). Next time if you have concerns please express them in time and not after we have tried to warn you of our problems with attending the meeting. Thanks! ~ Art.

1 0

ooni translations
by isis agora lovecruft 23 Jun '13

23 Jun '13

Our code, from what I understand, though I have never worked with Transifex or any of these other translation services, is not particularly well-suited to translation. First question, at what point/release is OONI going to aim for for having translations available? Going with the "usual" route of having a strings.py seems like it wouldn't work for OONI, do to the number of merge conflicts it would create (i.e. if I work on a nettest, and Aaron does some work on the director, and we both change strings.py, then the merge might not work cleanly). Are there any other opinions or ideas on the cleanest way to get ready for translations? We could have an ooni.strings submodule, and stick per file things in there, like director_str.py, etc. Thoughts? -- ♥Ⓐ isis agora lovecruft _________________________________________________________ GPG: 4096R/A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt

3 3

Communication & Meetings [action needed]
by Tom Lowenthal 22 Jun '13

22 Jun '13

I've written a initial wiki page with our meeting schedule and communication norms at <https://github.com/TheTorProject/ooni-probe/wiki/Communication>. Other notes: * All ooni-related mail should either go to the ooni-dev list or CC me. If you're sending something to the list and there's a specific ask for me, you can also CC me to get my attention. * All M-Lab-related mail should CC Meredith. * It's expected that an email directed to one of us with a specific ask will be answered within 24 hours on week-days. * Email expecting a response or action should indicate this in the subject line. # Next Actions * Subscribe to the ooni-dev mailing list at: <https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev>. * Edit that wiki page to add your public contact info, including an email address and your Github handle. If you have other communication preferences or foibles, this is a good place for them too. # Optional Actions * Look through the issues, and assign them to yourself if needed. * File new issues for things you think aren't covered. * Write wiki pages documenting other agreements or knowledge captured this week. Thanks, -Tom

2 1

Use of Chromebot Top Million URLs Instead of Alexa?
by Collin Anderson 22 Jun '13

22 Jun '13

Hello OONI, The source tree for Chromebot contains a list of "top million URLs" for stability testing [1]. The Chromebot list might present an interesting complement to Alexa for the HTTP tests, since it would be more likely resemble real-world conditions through triggering URL-specific responses from censors -- rather than exclusively measuring filtering based on hostname. [1] http://src.chromium.org/svn/trunk/tools/chromebot/top-million Cordially, Collin -- *Collin David Anderson* averysmallbird.com | @cda | Washington, D.C.

1 0

Re: [ooni-dev] Dimming the Internet: Detecting Throttling as a Mechanism of Censorship in Iran
by isis agora lovecruft 21 Jun '13

21 Jun '13

Collin Anderson transcribed 5.7K bytes: > Today, my latest paper “Dimming the Internet: Detecting Throttling as a > Mechanism of Censorship in Iran,” a documentation on three years of the use > of bandwidth throttling as a means of political censorship in Iran, was > posted on the publishing site arXiv. Congratulations, Colin! You are awesome. I put this up on the censorship wiki for Iran: https://trac.torproject.org/projects/tor/wiki/doc/OONI/censorshipwiki/Censo… -- ♥Ⓐ isis agora lovecruft _________________________________________________________ GPG: 4096R/A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt

1 0

Fwd: Dimming the Internet: Detecting Throttling as a Mechanism of Censorship in Iran
by isis agora lovecruft 21 Jun '13

21 Jun '13

----- Forwarded message from Collin Anderson <collina(a)gmail.com> ----- > From: Collin Anderson <collina(a)gmail.com> > Subject: Dimming the Internet: Detecting Throttling as a Mechanism of Censorship in Iran > Date: Thu, 20 Jun 2013 17:48:05 +0100 > Message-ID: <CA+CWxj+Xc-xa_Tr3sM4MEqtMR4Rv-rjv5hNPetG4wEG==DQEZA(a)mail.gmail.com> > To: Collin Anderson <collina(a)gmail.com> > > *I hate to do the mass BCC thing, but wanted to brag to you specifically > and hopefully get your attention for a **bit. I promise June and July will > be an interesting month of research, so please forgive my vanity. - Collin* > > Today, my latest paper “Dimming the Internet: Detecting Throttling as a > Mechanism of Censorship in Iran,” a documentation on three years of the use > of bandwidth throttling as a means of political censorship in Iran, was > posted on the publishing site arXiv. > > Blogpost (with Election data): > http://cgcsblog.asc.upenn.edu/2013/06/20/dimming-the-internet-detecting-thr… > Paper (arXiv): http://arxiv.org/abs/1306.4361 > > “Dimming the Internet" uses a three-year dataset of network measurements as > a monitoring service for political throttling, then applies the methodology > to shed light on the recent history of censorship in Iran, and finds that > Iran’s Internet has experienced prolonged and significant disruptions timed > annually near the anniversary of the Islamic Revolution and 25 Bahman, the > first anniversary of the contested elections, and protests over the > depreciation of the value of the Iranian Rial. Through the Measurement Lab > (M-Lab) platform, anti-censorship researchers gain a diverse and > non-partisan perspective on a network that is often opaque and difficult to > access from outside. The results described not only shed light on instances > of censorship, but also the manner in which public networks are subjected > to a greater degree of disruption than those of business, universities and > governments. > > Iran is not alone in this behavior, with Syria, Bahrain, Myanmar and > Vietnam reportedly utilizing similar tactics. As stories of connection > throttling timed with political instability grow and access to > international communications platforms remains a necessary means for > expression otherwise denied, “Dimming the Internet” provides a framework > for accountability on a trend that thus far has been opaque. In the coming > months, the Center for Global Communication Research at the Annenberg > School for Communication will expand on this methodology as a means to > account for historical cases and monitor this phenomenon on an ongoing > basis globally. > > *“Dimming the Internet” was supported by the Center for Global > Communication Studies at the University of Pennsylvania and Google > Research. They are awesome.* > -- > *Collin David Anderson* > averysmallbird.com | @cda | Washington, D.C. ----- End forwarded message ----- -- ♥Ⓐ isis agora lovecruft _________________________________________________________ GPG: 4096R/A3ADB67A2CDB8B35 Current Keys: https://blog.patternsinthevoid.net/isis.txt

1 0

Team Meeting: 2003-06-17 0900 PDT [be in #ooni 10 mins early]
by Tom Lowenthal 19 Jun '13

19 Jun '13

Hello hello hello fantastic Ooni folks. Get ready for a precociously productive meeting on Monday. # Time This Monday is our first team meeting since the Berlin intensive face-to-face. It's at 9am on the West coast, 10am in the mountains, noon on the East coast, 5pm in the UK, and 6pm in Europe central. If you're planning on coming and I haven't listed your time zone, let me know to include it in future reminders. The meeting is open to all who wish to observe or listen productively. # Venue I intend to meet in the pixellated video realm that is Google Hangouts. My gmail address is flamsmark(a)gmail.com, and you're totally welcome (and encouraged) to try getting in touch with me and testing your video capabilities before the call. If Hangouts will *not* work for you on Monday, reply to this email, and I'll promulgate an alternative meeting venue, likely Uberconference. I anticipate that there may be a few technical hiccups with this new and uncharted technology, so please be in #ooni on OFTC *ten minutes* before we start so that we can set up before the meeting and start on time. # Schedule My proposed tentative schedule for this meeting goes a little like this. I invite you to help me wrangle it by email, but not right before we start on Monday. 0900: Check in: how's everyone doing? 0910: Berlin takeaways: what's on your plate that isn't captured on Github? 0940: Triage upstream: collaboratively move items from Trac to Github. 1010: Week in profile: what things are you planning to get done this week? 1025: Parting thoughts. 1030: Adjourn. # Tasks In order to have a useful and productive session, here are some things that everyone should do before 9am on Monday: 1. Decompress. Travel is tough; come out of the weekend refreshed. 2. Read the agenda. Reply if you have suggestions. 3. Prepare your Hangouts. Make sure we can actually connect on Monday. 4. Review the Ooni tickets on Trac. They're all listed at <http://j.mp/153kpDN>. * * * * * That's all for now. I'll see all your gorgeous smiling faces on Monday. -Tom

3 2

test decks
by Jacob Appelbaum 19 Jun '13

19 Jun '13

Greetings from India, So I've been testing networks in Bangalore and I've noticed a few odd quirks with using a test deck. Here is my ooniprobe.conf: % cat ooniprobe.conf # This is the configuration file for OONIProbe # This file follows the YAML markup format: http://yaml.org/spec/1.2/spec.html # Keep in mind that indentation matters. basic: # Where OONIProbe should be writing it's log file logfile: ooniprobe-bangalore.log privacy: # Should we include the IP address of the probe in the report? includeip: true # Should we include the ASN of the probe in the report? includeasn: true # Should we include the country as reported by GeoIP in the report? includecountry: true # Should we include the city as reported by GeoIP in the report? includecity: true # Should we collect a full packet capture on the client? includepcap: false reports: # This is a packet capture file (.pcap) to load as a test: pcap: Null advanced: # XXX change this to point to the directory where you have stored the GeoIP # database file. This should be the directory in which OONI is installed # /path/to/ooni-probe/data/ #geoip_data_dir: /usr/share/GeoIP/ geoip_data_dir: /home/a/ooni-probe/data/ debug: true # tor_binary: '/usr/sbin/tor' # For auto detection interface: auto # Of specify a specific interface #interface: wlan0 # If you do not specify start_tor, you will have to have Tor running and # explicitly set the control port and SOCKS port start_tor: true # After how many seconds we should give up on a particular measurement measurement_timeout: 30 # After how many retries we should give up on a measurement measurement_retries: 2 # How many measurments to perform concurrently measurement_concurrency: 10 # After how may seconds we should give up reporting reporting_timeout: 30 # After how many retries to give up on reporting reporting_retries: 6 # How many reports to perform concurrently reporting_concurrency: 10 tor: socks_port: 9250 control_port: 9251 # Specify the absolute path to the Tor bridges to use for testing #bridges: bridges.list # Specify path of the tor datadirectory. # This should be set to something to avoid having Tor download each time # the descriptors and consensus data. data_dir: ~/.tor/ Here is the test deck: % cat decks/india-full.deck - options: collector: null help: 0 logfile: null pcapfile: null reportfile: null subargs: [-t, '192.168.1.1', -f, 'inputs/india-uniq-hosts-with-alexa-top-1000.txt'] test_file: nettests/blocking/dnsconsistency.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200'] test_file: nettests/manipulation/http_header_field_manipulation.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200'] test_file: nettests/manipulation/http_invalid_request_line.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200', -f, 'inputs/india-uniq-urls-with-alexa-top-1000.txt'] test_file: nettests/manipulation/http_host.py A few things happen when I attempt to use this deck. Tor fails to return my IP: 2013-06-01 00:44:15+0530 [TorControlProtocol,client] [D] 100%: Done 2013-06-01 00:44:15+0530 [TorControlProtocol,client] [D] Building a TorState 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Successfully bootstrapped Tor 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] We now have the following circuits: 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 1 BUILT [194.132.32.43 165.225.132.54 46.165.221.166] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 2 EXTENDED [194.132.32.43] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 3 EXTENDED [] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 4 EXTENDED [] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] Obtained our IP address from a Tor Relay None 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Unhandled Error Traceback (most recent call last): Failure: txtorcon.torcontrolprotocol.TorProtocolError: 551 Address unknown 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Unable to lookup the probe IP via Tor. 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] Cannot determine the probe IP address with a traceroute, becase of insufficient priviledges 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Looking up your IP address via maxmind Then things get a little strange - http_host.py is never executed. Another is that http_header_field_manipulation.py runs and the log file shows everything, the yamloo file shows only this: % cat report-http_header_field_manipulation-2013-05-31T191417Z.yamloo ########################################### # OONI Probe Report for http_header_field_manipulation (0.1.3) # Sat Jun 1 00:57:40 2013 ########################################### --- options: [-b, 'http://93.95.227.200'] probe_asn: AS24560 probe_cc: IN probe_ip: 122.167.211.176 software_name: ooniprobe software_version: 0.0.11 start_time: 1370027657.776991 test_name: http_header_field_manipulation test_version: 0.1.3 ... The debug log shows the headers being sent and the data being returned with an issue at the collector: 2013-06-01 00:57:40+0530 [SOCKS5Client,client] Creating report with OONIB Reporter. Please be patient. 2013-06-01 00:57:40+0530 [SOCKS5Client,client] This may take up to 1-2 minutes... 2013-06-01 00:57:40+0530 [SOCKS5Client,client] [D] Successfully performed report <ooni.tasks.ReportEntry object at 0x588c190> 2013-06-01 00:57:40+0530 [SOCKS5Client,client] [D] None 2013-06-01 00:57:40+0530 [Uninitialized] [!] Failed to connect to reporter backend 2013-06-01 00:57:40+0530 [Uninitialized] Traceback (most recent call last): 2013-06-01 00:57:40+0530 [Uninitialized] File "/home/io/Documents/backup/git/tor/ooni-probe/ooni/reporter.py", line 323, in createReport 2013-06-01 00:57:40+0530 [Uninitialized] bodyProducer=bodyProducer) 2013-06-01 00:57:40+0530 [Uninitialized] ConnectError: An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. 2013-06-01 00:57:40+0530 [Uninitialized] ]. 2013-06-01 00:57:40+0530 [Uninitialized] [!] Failed to open <ooni.reporter.OONIBReporter object at 0x461d710> reporter, giving up... 2013-06-01 00:57:40+0530 [Uninitialized] [!] Reporter <ooni.reporter.OONIBReporter object at 0x461d710> failed, removing from report... 2013-06-01 00:57:40+0530 [Uninitialized] [D] Starting this task <generator object generateMeasurements at 0x51906e0> 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_put 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup 2013-06-01 00:57:40+0530 [Uninitialized] [D] Performing request http://93.95.227.200 PUT {'Accept-Language': ['en-US,en;q=0.8'], 'Accept-Encoding': ['gzip,deflate,sdch'], 'Accept': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'User-Agent': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'], 'Accept-Charset': ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'], 'Host': ['XAxlpMzUMfI5Vvi.com']} 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_get_random_capitalization 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup 2013-06-01 00:57:40+0530 [Uninitialized] [D] Performing request http://93.95.227.200 gET {'accePt-lanGuAGe': ['en-US,en;q=0.8'], 'accEpT-eNcoDING': ['gzip,deflate,sdch'], 'ACCepT': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'USeR-aGEnT': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7'], 'aCcEPt-chaRseT': ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'], 'HoSt': ['l5tHomKVddWW1A4.com']} 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_post_random_capitalization 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup In the end, I didn't have any yamloo files from the nettests/manipulation/http_invalid_request_line.py test. I had three files that updated and had some data which was basically: report-dns_consistency-2013-05-31T191417Z.yamloo report-http_header_field_manipulation-2013-05-31T191417Z.yamloo ooniprobe-bangalore.log I expected a few different things - one is that each test in the deck should produce a yamloo file. If the reporting back end takes the report, I suppose I might find it alright to not have the file but in the event of a failure, I really hope the data will be logged to a local .yamloo file. When I run the following deck: % cat decks/india.deck - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: http_host_india_bangalore_justa_hotel.log pcapfile: null reportfile: http_host_india_cis.yamloo subargs: [-b, 'http://93.95.227.200', -f, 'inputs/india-uniq-urls-with-alexa-top-1000.txt'] test_file: nettests/manipulation/http_host.py I have the proper output for http_host.py: % head report-http_host-2013-05-31T193306Z.yamloo ########################################### # OONI Probe Report for http_host (0.2.3) # Sat Jun 1 01:03:06 2013 ########################################### --- options: [-b, 'http://93.95.227.200', -f, inputs/india-uniq-urls-with-alexa-top-1000.txt] probe_asn: AS24560 probe_cc: IN probe_ip: 122.167.211.176 software_name: ooniprobe % tail report-http_host-2013-05-31T193306Z.yamloo url: http://93.95.227.200 response: body: '{"headers_dict": {"Connection": ["close"], "Host": ["zustmovies.com"]}, "request_line": "\nGET / HTTP/1.1", "request_headers": [["Connection", "close"], ["Host", "zustmovies.com"]]}' code: 200 headers: [] socksproxy: null transparent_http_proxy: false ... Note that the yamloo file is created not as http_host_india_bangalore_justa_hotel.log but as report-http_host-2013-05-31T193306Z.yamloo... It seems that perhaps test decks are too experimental for actual use with these issues - or did I do something horribly wrong? Thoughts? All the best, Jacob

2 1