[ooni-dev] Testing HTTPS URLs and certificate chain

[Arturo Filastò]

On Jun 22, 2015, at 8:48 AM, David Fifield <david at bamsoftware.com> wrote:
> 
> It would be good to have ongoing tests for the domains we use as fronts
> for anticensorship, e.g.:
> 	https://www.google.com/
> 	https://a0.awsstatic.com/
> 	https://ajax.aspnetcdn.com/
> I would love to have periodic checks that 1) each domain is accessible,
> and 2) the certificate chain is what we expect, to find MITM attempts.
> 
> I suppose the existing nettests/blocking/http_requests.py can handle
> simple HTTPS connectivity. Is it easy to add the URLs above to the
> standard tests?
> 

We are currently using the citizen-lab test-lists repository [1] as a source of URLs for testing.

The URLs there are categories by:

* Country
* Global
* By services

I think these domains would fix under the service category and should hence be placed inside of something like:

https://github.com/citizenlab/test-lists/tree/master/lists/services/cloudfront

or 

https://github.com/citizenlab/test-lists/tree/master/lists/services/cdn

Once they are added to the citizenlab test lists we can make a new version of ooniprobe that will include in the default test deck measurements for also the CDN service.

> I'm less sure about how to get the certificate chain. I did some
> searching and didn't find a way to get the certificate chain from the
> twisted.web.client.Agent that templates/httpt.py uses (maybe you provide
> it a twisted.internet.ssl.ContextFactory somehow?).
> nettests/experimental/tls_handshake.py doesn't seem to be quite what I
> want. What do you suggest?

Unfortunately getting the certificate chain via twisted is not an immediate task and I have been meaning to write for a while a test template for doing so properly easily.

The http_requests test does support HTTPS, but as you mentioned that doesn’t actually test for MITM since it doesn’t download the certificate.

I would say that the best bet you currently have is starting from the code in TLS Handshake and expand it to your needs.
Alternatively more minimal code (although not designed for this specific purpose) can be found in sslpin [2].

Note: Twisted has changed the SSL API between versions 13.x and 14.x and 15.x so having a solution that works with all the currently supported versions of twisted [3] is not going to be a simple feat.

Would be happy to help out in implementing this or provide help on this.

The first thing to do I think would be to write up a ticket on what exactly it is you need tested on trac: https://trac.torproject.org/projects/tor/newticket?component=Ooni

~ Arturo

[1] https://github.com/citizenlab/test-lists
[2] https://github.com/hellais/sslpin
[3] https://github.com/TheTorProject/ooni-probe/blob/master/requirements.txt#L2