[ooni-dev] Blocked.org.uk & ooniprobe

Arturo Filastò art at torproject.org
Fri Aug 28 15:58:30 UTC 2015


> On 09 Aug 2015, at 12:57, Daniel Ramsay <daniel at dretzq.org.uk> wrote:
> 
> We're already emulating a collector in the blocked.org.uk API, so
> perhaps we can go directly to peering with the pipeline.  Is there any
> reference information that I can read on how to go about getting this
> set up (protocols, hostnames, etc)?

Currently peering is achieved by me giving you an AWS shared secret and you running with a daily (or hourly) periodicity the following invoke task:
https://github.com/TheTorProject/ooni-pipeline-ng/blob/master/tasks.py#L228

The documentation of these components is basically inexistent, but if you are familiar with fabric and python software it should be quite natural it’s usage.

Basically once you have installed invoke and the required dependencies (listed in requirements.txt) you will edit the invoke.yaml file (using the .example as a template) to include the AWS shared secret.

Then you can configure a hourly cronjob to run:

invoke sync_reports /PATH/TO/YOUR/REPORTS/ARCHIVE

This will lead to you having peered with the OONI data pipeline.

>> Since we have received many requests of supporting HTTPS collectors
>> we have plans of adding support for it in the near future. Nowadays
>> it should be much easier since the twisted API for doing HTTPS has
>> improved since version 14.0.
>> 
> 
> I did add some minimal support for HTTPS collector URLs in the patch
> set.  It's still being worked on for upstream submission.  The HTTPS
> support probably doesn't go as far as you'd like though.
> 


Oh that’s great!

I would love to check out this code and provide some feedback on it.


>> Still I would like to preserve the property of having URLs be self
>> authenticating and designed a scheme to extend HTTPS URIs to support
>> something similar to certificate pinning here:
>> https://github.com/hellais/sslpin. That code is just a POC and is
>> based on an old version of twisted when it was harder to do cert
>> validation. I think supporting this in recent versions of twisted
>> should be much easier.
>> 
> 
> Newer versions of twisted and python will do certificate verification
> using the operating system's certificate store, but as you point out,
> that doesn't provide a way of ensuring that the only certificate that
> can be used is from the official CA rather than any of the others.
> 
> It may be possible to force a twisted agent to only use a bundled CA
> certificate for verification, rather than relying on the system
> installed CA list.  The python requests library supports this usage, but
> I'm not sure about twisted.

Yeah I think a bit of hacks may be needed to implement this, though I think this requirement is quite important to be met. 

~ Arturo


More information about the ooni-dev mailing list