[ooni-dev] Tunisian censorship lab
Aaron
aagbsn at extc.org
Wed Jun 19 14:48:25 UTC 2013
On Sat, Jun 15, 2013 at 3:03 PM, Jacob Appelbaum <jacob at appelbaum.net>wrote:
> Heya,
>
> Greetings from #404Lab in Tunis, Tunisia!
>
> I'm testing the ooniprobe in the censorship lab. I'm making a running
> list of issues and I'll outline them below.
>
> When running the following test:
>
> ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
> nettests/blocking/http_requests.py;
>
> I find that the test hangs for ~10 minutes:
>
> Log opened.
> [D] No test deck detected
> Starting Tor...
> [D] Setting control port as 9251
> [D] Setting SOCKS port as 9250
> [D] 10%: Finishing handshake with directory server
> [D] 15%: Establishing an encrypted directory connection
> [D] 20%: Asking for networkstatus consensus
> [D] 25%: Loading networkstatus consensus
> [D] 45%: Asking for relay descriptors
> [D] 80%: Connecting to the Tor network
> [D] 90%: Establishing a Tor circuit
> [D] 100%: Done
> [D] Building a TorState
> Successfully bootstrapped Tor
> [D] We now have the following circuits:
> [D] * <Circuit 1 BUILT [154.35.32.5] for GENERAL>
> [D] * <Circuit 2 BUILT [31.172.30.3] for GENERAL>
> [D] * <Circuit 3 BUILT [198.211.114.130] for GENERAL>
> [D] * <Circuit 5 BUILT [194.132.32.43] for GENERAL>
> [D] * <Circuit 6 BUILT [31.172.30.3 46.166.174.209 93.115.241.2] for
> GENERAL>
> [D] * <Circuit 7 EXTENDED [198.211.114.130 74.120.13.132] for GENERAL>
> [D] * <Circuit 8 EXTENDED [] for GENERAL>
> [D] * <Circuit 9 EXTENDED [] for GENERAL>
> [D] * <Circuit 10 EXTENDED [] for GENERAL>
> [D] * <Circuit 11 EXTENDED [] for GENERAL>
> [D] * <Circuit 12 EXTENDED [] for GENERAL>
> [D] * <Circuit 13 EXTENDED [] for GENERAL>
> [D] * <Circuit 14 EXTENDED [] for GENERAL>
> [D] * <Circuit 15 EXTENDED [] for GENERAL>
> [D] * <Circuit 16 EXTENDED [] for GENERAL>
> [D] Obtained our IP address from a Tor Relay None
> Found your IP via Tor 41.231.22.232
> We will include some geo data in the report
> Your AS number is: AS2609
> Reporting using collector: httpo://nkvphnp3p6agi5qq.onion
> [D] adding callback for startNetTest
> [D] Creating ./report-http_requests_test-2013-06-15T100736Z.yamloo
> [D] Writing report with YAML reporter
> [D] Created report with <ooni.reporter.YAMLReporter object at 0x334ca90>
> Reporting httpo://nkvphnp3p6agi5qq.onion/report
> [D] Sending {"software_name": "ooniprobe", "software_version": "0.0.11",
> "content": "---\noptions: []\nprobe_asn: AS2609\nprobe_cc: TN\nprobe_ip:
> 41.231.22.232\nsoftware_name: ooniprobe\nsoftware_version:
> 0.0.11\nstart_time: 1371290856.723735\ntest_name:
> http_requests_test\ntest_version: 0.2.3\n...\n", "test_version":
> "0.2.3", "test_name": "http_requests_test", "probe_asn": "AS2609"}
> Creating report with OONIB Reporter. Please be patient.
> This may take up to 1-2 minutes...
> [D] Created report with id
>
> 2013-06-15T100948Z_AS2609_UIsgF34hqK8qn2GaxQtaynMnlIQedKCNaH2fNVoY5voh1I4Kdw
> [D] Created report with <ooni.reporter.OONIBReporter object at 0x334cc90>
> [D] Starting this task <generator object generateMeasurements at 0x3c78870>
> [D] Running <class 'nettests.blocking.http_requests.HTTPRequestsTest'>
> test_get
> [D] Finished test setup
>
> ...
>
> I'm not clear what will happen next - it just hangs. I killed it after
> around ten minutes.
>
It looks like there is a bug where if you do not specify an input file or
host argument that ooni doesn't realize this until after test setup. On my
system, with the current git master
(514b707df4c701a1e4c58f726c0261e26fabdb32) I get the following error:
[D] Finished test setup
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File
"/home/ooni/.virtualenvs/ooni/local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 380, in callback
self._startRunCallbacks(result)
File
"/home/ooni/.virtualenvs/ooni/local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 488, in _startRunCallbacks
self._runCallbacks()
File
"/home/ooni/.virtualenvs/ooni/local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 575, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File
"/home/ooni/.virtualenvs/ooni/local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 1126, in gotResult
_inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
File
"/home/ooni/.virtualenvs/ooni/local/lib/python2.7/site-packages/twisted/internet/defer.py",
line 1070, in _inlineCallbacks
result = g.send(result)
File "/home/ooni/ooni-probe/ooni/director.py", line 227, in startNetTest
self.measurementManager.schedule(net_test.generateMeasurements())
File "/home/ooni/ooni-probe/ooni/managers.py", line 107, in schedule
self._fillSlots()
File "/home/ooni/ooni-probe/ooni/managers.py", line 58, in _fillSlots
task = self._tasks.next()
File "/home/ooni/ooni-probe/ooni/nettest.py", line 508, in
generateMeasurements
measurement = self.makeMeasurement(test_class, method, input)
File "/home/ooni/ooni-probe/ooni/nettest.py", line 483, in makeMeasurement
measurement = Measurement(test_class, test_method, test_input)
File "/home/ooni/ooni-probe/ooni/tasks.py", line 114, in __init__
self.testInstance.setUp()
File "data/nettests/blocking/http_requests.py", line 53, in setUp
raise Exception("No input specified")
exceptions.Exception: No input specified
The problem is that some nettests support multiple input arguments and
therefore cannot rely on specifying that an option is required (see: class
NetTest __init__ requiredOptions). We should either extend the NetTest API
to cleanly handle cases where NetTests can take input from either a single
host argument or an input file, or axe the single host argument and make
the input file a required option.
> I followed up with the following test:
>
> ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
> nettests/manipulation/http_invalid_request_line.py;
>
> It worked without issues.
>
> The following test goes on for a while:
>
> ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
> nettests/blocking/http_requests.py -f inputs/top-1000.txt
>
> It starts throwing the following errors:
> Performing GET request to google.com.ar
> [D] Performing request google.com.ar GET {'User-Agent': ['Mozilla/5.0
> (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6']}
> [!] Unsupported scheme: ''
> [!] Task <ooni.tasks.Measurement object at 0x3302d90> has failed 6 times
> Traceback (most recent call last):
> File "/ooni-probe/ooni/managers.py", line 59, in _fillSlots
> self._run(task)
> File "/ooni-probe/ooni/managers.py", line 70, in _run
> d = task.start()
> File "/ooni-probe/ooni/tasks.py", line 91, in start
> return BaseTask.start(self)
> File "/ooni-probe/ooni/tasks.py", line 38, in start
> self._running = defer.maybeDeferred(self.run)
> --- <exception caught here> ---
> File
> "/ooniprobe/lib/python2.7/site-packages/twisted/internet/defer.py", line
> 138, in maybeDeferred
> result = f(*args, **kw)
> File "/ooni-probe/ooni/tasks.py", line 129, in run
> d = self.netTestMethod()
> File "/ooni-probe/nettests/blocking/http_requests.py", line 117, in
> test_get
> headers=headers)
> File "/ooni-probe/ooni/templates/httpt.py", line 313, in doRequest
> body_producer)
> File "/ooni-probe/ooni/utils/txagentwithsocks.py", line 179, in request
> return client.Agent.request(self, method, uri, headers, bodyProducer)
> File "/lib/python2.7/site-packages/twisted/web/client.py", line 1218,
> in request
> parsedURI.port)
> File "/ooni-probe/ooni/utils/txagentwithsocks.py", line 199, in
> _getEndpoint
> raise SchemeNotSupported("Unsupported xxx scheme: %r" % (scheme,))
> exceptions.NameError: global name 'SchemeNotSupported' is not defined
> [!] Permanent failure for <ooni.tasks.Measurement object at 0x3302d90>
> Failed doing measurement: <ooni.tasks.Measurement object at 0x3302d90>
> [D] Starting this task <ooni.tasks.ReportEntry object at 0xbf0c950>
> [D] Starting this task <ooni.tasks.ReportEntry object at 0xbf0ca50>
>
>
Your input URL list should be a list of URLs (not just domains, but should
specify the protocol). We should also add inputProcessor's for existing
nettests so that they can take more flexible input lists. I have added
inputProcessors to a few tests, but not yet http_requests.
It then hangs like so:
>
> [D] Checking all tasks for completion 102 == 133
> [D] Successfully performed report <ooni.tasks.ReportEntry object at
> 0xc1405d0>
> [D] None
> [D] Checking all tasks for completion 103 == 133
> [D] Successfully performed report <ooni.tasks.ReportEntry object at
> 0xc1408d0>
> [D] None
>
> I killed it - I'm not sure why it hangs. The host list is just the top
> alexa hosts.
>
Probably is the case of an unhandled traceback (due to a bad input) -- it
should get caught, retried, and eventually skipped, but clearly it doesn't.
> % ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
> nettests/blocking/tcpconnect.py -f inputs/top-1000-port80.txt
>
> This worked without issue.
>
Oh good :). BTW, I added inputProcessor support to tcpconnect so you can
use it with a URL list. It doesn't currently support URLs with a specified
port #, though.
> I tried:
>
> % ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
> nettests/blocking/http_requests.py -f inputs/top-1000.txt
>
> I had similar hangs - so the tests never finished:
>
> [D] Checking all tasks for completion 100 == 133
> [D] Successfully performed report <ooni.tasks.ReportEntry object at
> 0xcc7d190>
> [D] None
> [D] Checking all tasks for completion 101 == 133
> [D] Successfully performed report <ooni.tasks.ReportEntry object at
> 0xcc7d490>
> [D] None
> [D] Checking all tasks for completion 102 == 133
> [D] Successfully performed report <ooni.tasks.ReportEntry object at
> 0xcc7d790>
> [D] None
> [D] Checking all tasks for completion 103 == 133
> [D] Successfully performed report <ooni.tasks.ReportEntry object at
> 0xcc7da90>
> [D] None
>
> Thoughts?
>
These logs aren't very descriptive, though, I'd guess it's the same URL
issue as above.
--Aaron
>
> All the best,
> Jake
> _______________________________________________
> ooni-dev mailing list
> ooni-dev at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/ooni-dev/attachments/20130619/1adfb40f/attachment-0001.html>
More information about the ooni-dev
mailing list