[ooni-dev] Tunisian censorship lab

Sat Jun 15 15:03:05 UTC 2013

Heya,

Greetings from #404Lab in Tunis, Tunisia!

I'm testing the ooniprobe in the censorship lab. I'm making a running
list of issues and I'll outline them below.

When running the following test:

 ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
nettests/blocking/http_requests.py;

I find that the test hangs for ~10 minutes:

Log opened.
[D] No test deck detected
Starting Tor...
[D] Setting control port as 9251
[D] Setting SOCKS port as 9250
[D] 10%: Finishing handshake with directory server
[D] 15%: Establishing an encrypted directory connection
[D] 20%: Asking for networkstatus consensus
[D] 25%: Loading networkstatus consensus
[D] 45%: Asking for relay descriptors
[D] 80%: Connecting to the Tor network
[D] 90%: Establishing a Tor circuit
[D] 100%: Done
[D] Building a TorState
Successfully bootstrapped Tor
[D] We now have the following circuits:
[D]  * <Circuit 1 BUILT [154.35.32.5] for GENERAL>
[D]  * <Circuit 2 BUILT [31.172.30.3] for GENERAL>
[D]  * <Circuit 3 BUILT [198.211.114.130] for GENERAL>
[D]  * <Circuit 5 BUILT [194.132.32.43] for GENERAL>
[D]  * <Circuit 6 BUILT [31.172.30.3 46.166.174.209 93.115.241.2] for
GENERAL>
[D]  * <Circuit 7 EXTENDED [198.211.114.130 74.120.13.132] for GENERAL>
[D]  * <Circuit 8 EXTENDED [] for GENERAL>
[D]  * <Circuit 9 EXTENDED [] for GENERAL>
[D]  * <Circuit 10 EXTENDED [] for GENERAL>
[D]  * <Circuit 11 EXTENDED [] for GENERAL>
[D]  * <Circuit 12 EXTENDED [] for GENERAL>
[D]  * <Circuit 13 EXTENDED [] for GENERAL>
[D]  * <Circuit 14 EXTENDED [] for GENERAL>
[D]  * <Circuit 15 EXTENDED [] for GENERAL>
[D]  * <Circuit 16 EXTENDED [] for GENERAL>
[D] Obtained our IP address from a Tor Relay None
Found your IP via Tor 41.231.22.232
We will include some geo data in the report
Your AS number is: AS2609
Reporting using collector: httpo://nkvphnp3p6agi5qq.onion
[D] adding callback for startNetTest
[D] Creating ./report-http_requests_test-2013-06-15T100736Z.yamloo
[D] Writing report with YAML reporter
[D] Created report with <ooni.reporter.YAMLReporter object at 0x334ca90>
Reporting httpo://nkvphnp3p6agi5qq.onion/report
[D] Sending {"software_name": "ooniprobe", "software_version": "0.0.11",
"content": "---\noptions: []\nprobe_asn: AS2609\nprobe_cc: TN\nprobe_ip:
41.231.22.232\nsoftware_name: ooniprobe\nsoftware_version:
0.0.11\nstart_time: 1371290856.723735\ntest_name:
http_requests_test\ntest_version: 0.2.3\n...\n", "test_version":
"0.2.3", "test_name": "http_requests_test", "probe_asn": "AS2609"}
Creating report with OONIB Reporter. Please be patient.
This may take up to 1-2 minutes...
[D] Created report with id
2013-06-15T100948Z_AS2609_UIsgF34hqK8qn2GaxQtaynMnlIQedKCNaH2fNVoY5voh1I4Kdw
[D] Created report with <ooni.reporter.OONIBReporter object at 0x334cc90>
[D] Starting this task <generator object generateMeasurements at 0x3c78870>
[D] Running <class 'nettests.blocking.http_requests.HTTPRequestsTest'>
test_get
[D] Finished test setup

...

I'm not clear what will happen next - it just hangs. I killed it after
around ten minutes.

I followed up with the following test:

./bin/ooniprobe -l tunisia-404lab-netcache-000.log
nettests/manipulation/http_invalid_request_line.py;

It worked without issues.

The following test goes on for a while:

./bin/ooniprobe -l tunisia-404lab-netcache-000.log
nettests/blocking/http_requests.py -f inputs/top-1000.txt

It starts throwing the following errors:
Performing GET request to google.com.ar
[D] Performing request google.com.ar GET {'User-Agent': ['Mozilla/5.0
(Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6']}
[!] Unsupported scheme: ''
[!] Task <ooni.tasks.Measurement object at 0x3302d90> has failed 6 times
Traceback (most recent call last):
  File "/ooni-probe/ooni/managers.py", line 59, in _fillSlots
    self._run(task)
  File "/ooni-probe/ooni/managers.py", line 70, in _run
    d = task.start()
  File "/ooni-probe/ooni/tasks.py", line 91, in start
    return BaseTask.start(self)
  File "/ooni-probe/ooni/tasks.py", line 38, in start
    self._running = defer.maybeDeferred(self.run)
--- <exception caught here> ---
  File
"/ooniprobe/lib/python2.7/site-packages/twisted/internet/defer.py", line
138, in maybeDeferred
    result = f(*args, **kw)
  File "/ooni-probe/ooni/tasks.py", line 129, in run
    d = self.netTestMethod()
  File "/ooni-probe/nettests/blocking/http_requests.py", line 117, in
test_get
    headers=headers)
  File "/ooni-probe/ooni/templates/httpt.py", line 313, in doRequest
    body_producer)
  File "/ooni-probe/ooni/utils/txagentwithsocks.py", line 179, in request
    return client.Agent.request(self, method, uri, headers, bodyProducer)
  File "/lib/python2.7/site-packages/twisted/web/client.py", line 1218,
in request
    parsedURI.port)
  File "/ooni-probe/ooni/utils/txagentwithsocks.py", line 199, in
_getEndpoint
    raise SchemeNotSupported("Unsupported xxx scheme: %r" % (scheme,))
exceptions.NameError: global name 'SchemeNotSupported' is not defined
[!] Permanent failure for <ooni.tasks.Measurement object at 0x3302d90>
Failed doing measurement: <ooni.tasks.Measurement object at 0x3302d90>
[D] Starting this task <ooni.tasks.ReportEntry object at 0xbf0c950>
[D] Starting this task <ooni.tasks.ReportEntry object at 0xbf0ca50>

It then hangs like so:

[D] Checking all tasks for completion 102 == 133
[D] Successfully performed report <ooni.tasks.ReportEntry object at
0xc1405d0>
[D] None
[D] Checking all tasks for completion 103 == 133
[D] Successfully performed report <ooni.tasks.ReportEntry object at
0xc1408d0>
[D] None

I killed it - I'm not sure why it hangs. The host list is just the top
alexa hosts.

 % ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
nettests/blocking/tcpconnect.py -f inputs/top-1000-port80.txt

This worked without issue.

I tried:

 % ./bin/ooniprobe -l tunisia-404lab-netcache-000.log
nettests/blocking/http_requests.py -f inputs/top-1000.txt

I had similar hangs - so the tests never finished:

[D] Checking all tasks for completion 100 == 133
[D] Successfully performed report <ooni.tasks.ReportEntry object at
0xcc7d190>
[D] None
[D] Checking all tasks for completion 101 == 133
[D] Successfully performed report <ooni.tasks.ReportEntry object at
0xcc7d490>
[D] None
[D] Checking all tasks for completion 102 == 133
[D] Successfully performed report <ooni.tasks.ReportEntry object at
0xcc7d790>
[D] None
[D] Checking all tasks for completion 103 == 133
[D] Successfully performed report <ooni.tasks.ReportEntry object at
0xcc7da90>
[D] None

Thoughts?

All the best,
Jake