[ooni-dev] Testing HTTPS URLs and certificate chain

[David Fifield]

On Mon, Jun 22, 2015 at 11:44:49AM +0200, Arturo Filastò wrote:
> On Jun 22, 2015, at 8:48 AM, David Fifield <david at bamsoftware.com> wrote:
> > 
> > It would be good to have ongoing tests for the domains we use as fronts
> > for anticensorship, e.g.:
> > 	https://www.google.com/
> > 	https://a0.awsstatic.com/
> > 	https://ajax.aspnetcdn.com/
> > I would love to have periodic checks that 1) each domain is accessible,
> > and 2) the certificate chain is what we expect, to find MITM attempts.
> > 
> > I suppose the existing nettests/blocking/http_requests.py can handle
> > simple HTTPS connectivity. Is it easy to add the URLs above to the
> > standard tests?
> 
> We are currently using the citizen-lab test-lists repository [1] as a source of URLs for testing.
> The URLs there are categories by:
> * Country
> * Global
> * By services
> I think these domains would fix under the service category and should hence be placed inside of something like:
> https://github.com/citizenlab/test-lists/tree/master/lists/services/cloudfront
> or 
> https://github.com/citizenlab/test-lists/tree/master/lists/services/cdn
> Once they are added to the citizenlab test lists we can make a new version of ooniprobe that will include in the default test deck measurements for also the CDN service.

Thanks. I made a CSV file and sent it to contributors of test-lists. I
expect it will need some changes but it's a start.

> > I'm less sure about how to get the certificate chain. I did some
> > searching and didn't find a way to get the certificate chain from the
> > twisted.web.client.Agent that templates/httpt.py uses (maybe you provide
> > it a twisted.internet.ssl.ContextFactory somehow?).
> > nettests/experimental/tls_handshake.py doesn't seem to be quite what I
> > want. What do you suggest?
> 
> Unfortunately getting the certificate chain via twisted is not an immediate task and I have been meaning to write for a while a test template for doing so properly easily.
> 
> The http_requests test does support HTTPS, but as you mentioned that doesn’t actually test for MITM since it doesn’t download the certificate.
> 
> I would say that the best bet you currently have is starting from the code in TLS Handshake and expand it to your needs.
> Alternatively more minimal code (although not designed for this specific purpose) can be found in sslpin [2].
> 
> Note: Twisted has changed the SSL API between versions 13.x and 14.x and 15.x so having a solution that works with all the currently supported versions of twisted [3] is not going to be a simple feat.
> 
> Would be happy to help out in implementing this or provide help on this.
> 
> The first thing to do I think would be to write up a ticket on what exactly it is you need tested on trac: https://trac.torproject.org/projects/tor/newticket?component=Ooni

I opened https://trac.torproject.org/projects/tor/ticket/16421.