ooni-dev

ooni-dev@lists.torproject.org

May 2013

2 participants
2 discussions

test decks
by Jacob Appelbaum 19 Jun '13

19 Jun '13

Greetings from India, So I've been testing networks in Bangalore and I've noticed a few odd quirks with using a test deck. Here is my ooniprobe.conf: % cat ooniprobe.conf # This is the configuration file for OONIProbe # This file follows the YAML markup format: http://yaml.org/spec/1.2/spec.html # Keep in mind that indentation matters. basic: # Where OONIProbe should be writing it's log file logfile: ooniprobe-bangalore.log privacy: # Should we include the IP address of the probe in the report? includeip: true # Should we include the ASN of the probe in the report? includeasn: true # Should we include the country as reported by GeoIP in the report? includecountry: true # Should we include the city as reported by GeoIP in the report? includecity: true # Should we collect a full packet capture on the client? includepcap: false reports: # This is a packet capture file (.pcap) to load as a test: pcap: Null advanced: # XXX change this to point to the directory where you have stored the GeoIP # database file. This should be the directory in which OONI is installed # /path/to/ooni-probe/data/ #geoip_data_dir: /usr/share/GeoIP/ geoip_data_dir: /home/a/ooni-probe/data/ debug: true # tor_binary: '/usr/sbin/tor' # For auto detection interface: auto # Of specify a specific interface #interface: wlan0 # If you do not specify start_tor, you will have to have Tor running and # explicitly set the control port and SOCKS port start_tor: true # After how many seconds we should give up on a particular measurement measurement_timeout: 30 # After how many retries we should give up on a measurement measurement_retries: 2 # How many measurments to perform concurrently measurement_concurrency: 10 # After how may seconds we should give up reporting reporting_timeout: 30 # After how many retries to give up on reporting reporting_retries: 6 # How many reports to perform concurrently reporting_concurrency: 10 tor: socks_port: 9250 control_port: 9251 # Specify the absolute path to the Tor bridges to use for testing #bridges: bridges.list # Specify path of the tor datadirectory. # This should be set to something to avoid having Tor download each time # the descriptors and consensus data. data_dir: ~/.tor/ Here is the test deck: % cat decks/india-full.deck - options: collector: null help: 0 logfile: null pcapfile: null reportfile: null subargs: [-t, '192.168.1.1', -f, 'inputs/india-uniq-hosts-with-alexa-top-1000.txt'] test_file: nettests/blocking/dnsconsistency.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200'] test_file: nettests/manipulation/http_header_field_manipulation.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200'] test_file: nettests/manipulation/http_invalid_request_line.py - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: null pcapfile: null reportfile: null subargs: [-b, 'http://93.95.227.200', -f, 'inputs/india-uniq-urls-with-alexa-top-1000.txt'] test_file: nettests/manipulation/http_host.py A few things happen when I attempt to use this deck. Tor fails to return my IP: 2013-06-01 00:44:15+0530 [TorControlProtocol,client] [D] 100%: Done 2013-06-01 00:44:15+0530 [TorControlProtocol,client] [D] Building a TorState 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Successfully bootstrapped Tor 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] We now have the following circuits: 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 1 BUILT [194.132.32.43 165.225.132.54 46.165.221.166] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 2 EXTENDED [194.132.32.43] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 3 EXTENDED [] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] * <Circuit 4 EXTENDED [] for GENERAL> 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] Obtained our IP address from a Tor Relay None 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Unhandled Error Traceback (most recent call last): Failure: txtorcon.torcontrolprotocol.TorProtocolError: 551 Address unknown 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Unable to lookup the probe IP via Tor. 2013-06-01 00:44:16+0530 [TorControlProtocol,client] [D] Cannot determine the probe IP address with a traceroute, becase of insufficient priviledges 2013-06-01 00:44:16+0530 [TorControlProtocol,client] Looking up your IP address via maxmind Then things get a little strange - http_host.py is never executed. Another is that http_header_field_manipulation.py runs and the log file shows everything, the yamloo file shows only this: % cat report-http_header_field_manipulation-2013-05-31T191417Z.yamloo ########################################### # OONI Probe Report for http_header_field_manipulation (0.1.3) # Sat Jun 1 00:57:40 2013 ########################################### --- options: [-b, 'http://93.95.227.200'] probe_asn: AS24560 probe_cc: IN probe_ip: 122.167.211.176 software_name: ooniprobe software_version: 0.0.11 start_time: 1370027657.776991 test_name: http_header_field_manipulation test_version: 0.1.3 ... The debug log shows the headers being sent and the data being returned with an issue at the collector: 2013-06-01 00:57:40+0530 [SOCKS5Client,client] Creating report with OONIB Reporter. Please be patient. 2013-06-01 00:57:40+0530 [SOCKS5Client,client] This may take up to 1-2 minutes... 2013-06-01 00:57:40+0530 [SOCKS5Client,client] [D] Successfully performed report <ooni.tasks.ReportEntry object at 0x588c190> 2013-06-01 00:57:40+0530 [SOCKS5Client,client] [D] None 2013-06-01 00:57:40+0530 [Uninitialized] [!] Failed to connect to reporter backend 2013-06-01 00:57:40+0530 [Uninitialized] Traceback (most recent call last): 2013-06-01 00:57:40+0530 [Uninitialized] File "/home/io/Documents/backup/git/tor/ooni-probe/ooni/reporter.py", line 323, in createReport 2013-06-01 00:57:40+0530 [Uninitialized] bodyProducer=bodyProducer) 2013-06-01 00:57:40+0530 [Uninitialized] ConnectError: An error occurred while connecting: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost. 2013-06-01 00:57:40+0530 [Uninitialized] ]. 2013-06-01 00:57:40+0530 [Uninitialized] [!] Failed to open <ooni.reporter.OONIBReporter object at 0x461d710> reporter, giving up... 2013-06-01 00:57:40+0530 [Uninitialized] [!] Reporter <ooni.reporter.OONIBReporter object at 0x461d710> failed, removing from report... 2013-06-01 00:57:40+0530 [Uninitialized] [D] Starting this task <generator object generateMeasurements at 0x51906e0> 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_put 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup 2013-06-01 00:57:40+0530 [Uninitialized] [D] Performing request http://93.95.227.200 PUT {'Accept-Language': ['en-US,en;q=0.8'], 'Accept-Encoding': ['gzip,deflate,sdch'], 'Accept': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'User-Agent': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6'], 'Accept-Charset': ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'], 'Host': ['XAxlpMzUMfI5Vvi.com']} 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_get_random_capitalization 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup 2013-06-01 00:57:40+0530 [Uninitialized] [D] Performing request http://93.95.227.200 gET {'accePt-lanGuAGe': ['en-US,en;q=0.8'], 'accEpT-eNcoDING': ['gzip,deflate,sdch'], 'ACCepT': ['text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'], 'USeR-aGEnT': ['Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7'], 'aCcEPt-chaRseT': ['ISO-8859-1,utf-8;q=0.7,*;q=0.3'], 'HoSt': ['l5tHomKVddWW1A4.com']} 2013-06-01 00:57:40+0530 [Uninitialized] [D] Running <class 'nettests.manipulation.http_header_field_manipulation.HTTPHeaderFieldManipulation'> test_post_random_capitalization 2013-06-01 00:57:40+0530 [Uninitialized] [D] Finished test setup In the end, I didn't have any yamloo files from the nettests/manipulation/http_invalid_request_line.py test. I had three files that updated and had some data which was basically: report-dns_consistency-2013-05-31T191417Z.yamloo report-http_header_field_manipulation-2013-05-31T191417Z.yamloo ooniprobe-bangalore.log I expected a few different things - one is that each test in the deck should produce a yamloo file. If the reporting back end takes the report, I suppose I might find it alright to not have the file but in the event of a failure, I really hope the data will be logged to a local .yamloo file. When I run the following deck: % cat decks/india.deck - options: collector: httpo://nkvphnp3p6agi5qq.onion help: 0 logfile: http_host_india_bangalore_justa_hotel.log pcapfile: null reportfile: http_host_india_cis.yamloo subargs: [-b, 'http://93.95.227.200', -f, 'inputs/india-uniq-urls-with-alexa-top-1000.txt'] test_file: nettests/manipulation/http_host.py I have the proper output for http_host.py: % head report-http_host-2013-05-31T193306Z.yamloo ########################################### # OONI Probe Report for http_host (0.2.3) # Sat Jun 1 01:03:06 2013 ########################################### --- options: [-b, 'http://93.95.227.200', -f, inputs/india-uniq-urls-with-alexa-top-1000.txt] probe_asn: AS24560 probe_cc: IN probe_ip: 122.167.211.176 software_name: ooniprobe % tail report-http_host-2013-05-31T193306Z.yamloo url: http://93.95.227.200 response: body: '{"headers_dict": {"Connection": ["close"], "Host": ["zustmovies.com"]}, "request_line": "\nGET / HTTP/1.1", "request_headers": [["Connection", "close"], ["Host", "zustmovies.com"]]}' code: 200 headers: [] socksproxy: null transparent_http_proxy: false ... Note that the yamloo file is created not as http_host_india_bangalore_justa_hotel.log but as report-http_host-2013-05-31T193306Z.yamloo... It seems that perhaps test decks are too experimental for actual use with these issues - or did I do something horribly wrong? Thoughts? All the best, Jacob

2 1

Fwd: Fwd: RIPE Atlas probe source code released
by isis agora lovecruft 08 May '13

08 May '13

----- Forwarded message from Sacha van Geffen <sacha(a)greenhost.nl> ----- > From: Sacha van Geffen <sacha(a)greenhost.nl> > Subject: Fwd: RIPE Atlas probe source code released > Date: Tue, 07 May 2013 09:57:41 +0200 > Message-ID: <5188B3F5.7000404(a)greenhost.nl> > User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130329 Thunderbird/17.0.5 > To: undisclosed-recipients: ; > > Hi, > > Just in case you have not heard yet, ripe has released part of their > probe source code. > > Cheers, > > Sacha > > > ----- Forwarded message from Mirjam Kuehne <mir(a)ripe.net> ----- > Date: Mon, 06 May 2013 16:43:07 +0200 > From: Mirjam Kuehne <mir(a)ripe.net> > To: mat-wg(a)ripe.net > Subject: [mat-wg] New on RIPE Labs: Releasing RIPE Atlas Measurements > Source Code > > Dear colleagues, > > This new article on RIPE Labs introduces the RIPE Atlas measurement > source code. The measurement code is the part that runs on the RIPE > Atlas probes to schedule and execute particular measurements, and > thereby forms the core of the RIPE Atlas system. > > https://labs.ripe.net/Members/philip_homburg/ripe-atlas-measurements-source… > > Kind regards, > Mirjam Kuehne > RIPE NCC > > > ----- End forwarded message ----- ----- End forwarded message ----- -- isis agora lovecruft Email: isis(a)torproject.org Public Key: 4096R/A3ADB67A2CDB8B35 GPG & OTR Keys: https://blog.patternsinthevoid.net/isis.txt irc.oftc.net: isis(a)37lnq2veifl4kar7.onion DD89578D C0E48B93 063E15CF D1D9DE1F 5B2F567C irc.indymedia.org: isis(a)h7gf2ha3hefoj5ls.onion B83EC6C6 3B520905 44AD7F06 529CA8C9 756FEE35

1 0