Heya,
Greetings from #404Lab in Tunis, Tunisia!
I'm testing the ooniprobe in the censorship lab. I'm making a running list of issues and I'll outline them below.
When running the following test:
./bin/ooniprobe -l tunisia-404lab-netcache-000.log nettests/blocking/http_requests.py;
I find that the test hangs for ~10 minutes:
Log opened. [D] No test deck detected Starting Tor... [D] Setting control port as 9251 [D] Setting SOCKS port as 9250 [D] 10%: Finishing handshake with directory server [D] 15%: Establishing an encrypted directory connection [D] 20%: Asking for networkstatus consensus [D] 25%: Loading networkstatus consensus [D] 45%: Asking for relay descriptors [D] 80%: Connecting to the Tor network [D] 90%: Establishing a Tor circuit [D] 100%: Done [D] Building a TorState Successfully bootstrapped Tor [D] We now have the following circuits: [D] * <Circuit 1 BUILT [154.35.32.5] for GENERAL> [D] * <Circuit 2 BUILT [31.172.30.3] for GENERAL> [D] * <Circuit 3 BUILT [198.211.114.130] for GENERAL> [D] * <Circuit 5 BUILT [194.132.32.43] for GENERAL> [D] * <Circuit 6 BUILT [31.172.30.3 46.166.174.209 93.115.241.2] for GENERAL> [D] * <Circuit 7 EXTENDED [198.211.114.130 74.120.13.132] for GENERAL> [D] * <Circuit 8 EXTENDED [] for GENERAL> [D] * <Circuit 9 EXTENDED [] for GENERAL> [D] * <Circuit 10 EXTENDED [] for GENERAL> [D] * <Circuit 11 EXTENDED [] for GENERAL> [D] * <Circuit 12 EXTENDED [] for GENERAL> [D] * <Circuit 13 EXTENDED [] for GENERAL> [D] * <Circuit 14 EXTENDED [] for GENERAL> [D] * <Circuit 15 EXTENDED [] for GENERAL> [D] * <Circuit 16 EXTENDED [] for GENERAL> [D] Obtained our IP address from a Tor Relay None Found your IP via Tor 41.231.22.232 We will include some geo data in the report Your AS number is: AS2609 Reporting using collector: httpo://nkvphnp3p6agi5qq.onion [D] adding callback for startNetTest [D] Creating ./report-http_requests_test-2013-06-15T100736Z.yamloo [D] Writing report with YAML reporter [D] Created report with <ooni.reporter.YAMLReporter object at 0x334ca90> Reporting httpo://nkvphnp3p6agi5qq.onion/report [D] Sending {"software_name": "ooniprobe", "software_version": "0.0.11", "content": "---\noptions: []\nprobe_asn: AS2609\nprobe_cc: TN\nprobe_ip: 41.231.22.232\nsoftware_name: ooniprobe\nsoftware_version: 0.0.11\nstart_time: 1371290856.723735\ntest_name: http_requests_test\ntest_version: 0.2.3\n...\n", "test_version": "0.2.3", "test_name": "http_requests_test", "probe_asn": "AS2609"} Creating report with OONIB Reporter. Please be patient. This may take up to 1-2 minutes... [D] Created report with id 2013-06-15T100948Z_AS2609_UIsgF34hqK8qn2GaxQtaynMnlIQedKCNaH2fNVoY5voh1I4Kdw [D] Created report with <ooni.reporter.OONIBReporter object at 0x334cc90> [D] Starting this task <generator object generateMeasurements at 0x3c78870> [D] Running <class 'nettests.blocking.http_requests.HTTPRequestsTest'> test_get [D] Finished test setup
...
I'm not clear what will happen next - it just hangs. I killed it after around ten minutes.
I followed up with the following test:
./bin/ooniprobe -l tunisia-404lab-netcache-000.log nettests/manipulation/http_invalid_request_line.py;
It worked without issues.
The following test goes on for a while:
./bin/ooniprobe -l tunisia-404lab-netcache-000.log nettests/blocking/http_requests.py -f inputs/top-1000.txt
It starts throwing the following errors: Performing GET request to google.com.ar [D] Performing request google.com.ar GET {'User-Agent': ['Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2) Gecko/20100115 Firefox/3.6']} [!] Unsupported scheme: '' [!] Task <ooni.tasks.Measurement object at 0x3302d90> has failed 6 times Traceback (most recent call last): File "/ooni-probe/ooni/managers.py", line 59, in _fillSlots self._run(task) File "/ooni-probe/ooni/managers.py", line 70, in _run d = task.start() File "/ooni-probe/ooni/tasks.py", line 91, in start return BaseTask.start(self) File "/ooni-probe/ooni/tasks.py", line 38, in start self._running = defer.maybeDeferred(self.run) --- <exception caught here> --- File "/ooniprobe/lib/python2.7/site-packages/twisted/internet/defer.py", line 138, in maybeDeferred result = f(*args, **kw) File "/ooni-probe/ooni/tasks.py", line 129, in run d = self.netTestMethod() File "/ooni-probe/nettests/blocking/http_requests.py", line 117, in test_get headers=headers) File "/ooni-probe/ooni/templates/httpt.py", line 313, in doRequest body_producer) File "/ooni-probe/ooni/utils/txagentwithsocks.py", line 179, in request return client.Agent.request(self, method, uri, headers, bodyProducer) File "/lib/python2.7/site-packages/twisted/web/client.py", line 1218, in request parsedURI.port) File "/ooni-probe/ooni/utils/txagentwithsocks.py", line 199, in _getEndpoint raise SchemeNotSupported("Unsupported xxx scheme: %r" % (scheme,)) exceptions.NameError: global name 'SchemeNotSupported' is not defined [!] Permanent failure for <ooni.tasks.Measurement object at 0x3302d90> Failed doing measurement: <ooni.tasks.Measurement object at 0x3302d90> [D] Starting this task <ooni.tasks.ReportEntry object at 0xbf0c950> [D] Starting this task <ooni.tasks.ReportEntry object at 0xbf0ca50>
It then hangs like so:
[D] Checking all tasks for completion 102 == 133 [D] Successfully performed report <ooni.tasks.ReportEntry object at 0xc1405d0> [D] None [D] Checking all tasks for completion 103 == 133 [D] Successfully performed report <ooni.tasks.ReportEntry object at 0xc1408d0> [D] None
I killed it - I'm not sure why it hangs. The host list is just the top alexa hosts.
% ./bin/ooniprobe -l tunisia-404lab-netcache-000.log nettests/blocking/tcpconnect.py -f inputs/top-1000-port80.txt
This worked without issue.
I tried:
% ./bin/ooniprobe -l tunisia-404lab-netcache-000.log nettests/blocking/http_requests.py -f inputs/top-1000.txt
I had similar hangs - so the tests never finished:
[D] Checking all tasks for completion 100 == 133 [D] Successfully performed report <ooni.tasks.ReportEntry object at 0xcc7d190> [D] None [D] Checking all tasks for completion 101 == 133 [D] Successfully performed report <ooni.tasks.ReportEntry object at 0xcc7d490> [D] None [D] Checking all tasks for completion 102 == 133 [D] Successfully performed report <ooni.tasks.ReportEntry object at 0xcc7d790> [D] None [D] Checking all tasks for completion 103 == 133 [D] Successfully performed report <ooni.tasks.ReportEntry object at 0xcc7da90> [D] None
Thoughts?
All the best, Jake