Testing HTTPS URLs and certificate chain
It would be good to have ongoing tests for the domains we use as fronts for anticensorship, e.g.: https://www.google.com/ https://a0.awsstatic.com/ https://ajax.aspnetcdn.com/ I would love to have periodic checks that 1) each domain is accessible, and 2) the certificate chain is what we expect, to find MITM attempts. I suppose the existing nettests/blocking/http_requests.py can handle simple HTTPS connectivity. Is it easy to add the URLs above to the standard tests? I'm less sure about how to get the certificate chain. I did some searching and didn't find a way to get the certificate chain from the twisted.web.client.Agent that templates/httpt.py uses (maybe you provide it a twisted.internet.ssl.ContextFactory somehow?). nettests/experimental/tls_handshake.py doesn't seem to be quite what I want. What do you suggest?
On Jun 22, 2015, at 8:48 AM, David Fifield <david@bamsoftware.com> wrote:
It would be good to have ongoing tests for the domains we use as fronts for anticensorship, e.g.: https://www.google.com/ https://a0.awsstatic.com/ https://ajax.aspnetcdn.com/ I would love to have periodic checks that 1) each domain is accessible, and 2) the certificate chain is what we expect, to find MITM attempts.
I suppose the existing nettests/blocking/http_requests.py can handle simple HTTPS connectivity. Is it easy to add the URLs above to the standard tests?
We are currently using the citizen-lab test-lists repository [1] as a source of URLs for testing. The URLs there are categories by: * Country * Global * By services I think these domains would fix under the service category and should hence be placed inside of something like: https://github.com/citizenlab/test-lists/tree/master/lists/services/cloudfro... or https://github.com/citizenlab/test-lists/tree/master/lists/services/cdn Once they are added to the citizenlab test lists we can make a new version of ooniprobe that will include in the default test deck measurements for also the CDN service.
I'm less sure about how to get the certificate chain. I did some searching and didn't find a way to get the certificate chain from the twisted.web.client.Agent that templates/httpt.py uses (maybe you provide it a twisted.internet.ssl.ContextFactory somehow?). nettests/experimental/tls_handshake.py doesn't seem to be quite what I want. What do you suggest?
Unfortunately getting the certificate chain via twisted is not an immediate task and I have been meaning to write for a while a test template for doing so properly easily. The http_requests test does support HTTPS, but as you mentioned that doesn’t actually test for MITM since it doesn’t download the certificate. I would say that the best bet you currently have is starting from the code in TLS Handshake and expand it to your needs. Alternatively more minimal code (although not designed for this specific purpose) can be found in sslpin [2]. Note: Twisted has changed the SSL API between versions 13.x and 14.x and 15.x so having a solution that works with all the currently supported versions of twisted [3] is not going to be a simple feat. Would be happy to help out in implementing this or provide help on this. The first thing to do I think would be to write up a ticket on what exactly it is you need tested on trac: https://trac.torproject.org/projects/tor/newticket?component=Ooni ~ Arturo [1] https://github.com/citizenlab/test-lists [2] https://github.com/hellais/sslpin [3] https://github.com/TheTorProject/ooni-probe/blob/master/requirements.txt#L2
On Mon, Jun 22, 2015 at 11:44:49AM +0200, Arturo Filastò wrote:
On Jun 22, 2015, at 8:48 AM, David Fifield <david@bamsoftware.com> wrote:
It would be good to have ongoing tests for the domains we use as fronts for anticensorship, e.g.: https://www.google.com/ https://a0.awsstatic.com/ https://ajax.aspnetcdn.com/ I would love to have periodic checks that 1) each domain is accessible, and 2) the certificate chain is what we expect, to find MITM attempts.
I suppose the existing nettests/blocking/http_requests.py can handle simple HTTPS connectivity. Is it easy to add the URLs above to the standard tests?
We are currently using the citizen-lab test-lists repository [1] as a source of URLs for testing. The URLs there are categories by: * Country * Global * By services I think these domains would fix under the service category and should hence be placed inside of something like: https://github.com/citizenlab/test-lists/tree/master/lists/services/cloudfro... or https://github.com/citizenlab/test-lists/tree/master/lists/services/cdn Once they are added to the citizenlab test lists we can make a new version of ooniprobe that will include in the default test deck measurements for also the CDN service.
Thanks. I made a CSV file and sent it to contributors of test-lists. I expect it will need some changes but it's a start.
I'm less sure about how to get the certificate chain. I did some searching and didn't find a way to get the certificate chain from the twisted.web.client.Agent that templates/httpt.py uses (maybe you provide it a twisted.internet.ssl.ContextFactory somehow?). nettests/experimental/tls_handshake.py doesn't seem to be quite what I want. What do you suggest?
Unfortunately getting the certificate chain via twisted is not an immediate task and I have been meaning to write for a while a test template for doing so properly easily.
The http_requests test does support HTTPS, but as you mentioned that doesn’t actually test for MITM since it doesn’t download the certificate.
I would say that the best bet you currently have is starting from the code in TLS Handshake and expand it to your needs. Alternatively more minimal code (although not designed for this specific purpose) can be found in sslpin [2].
Note: Twisted has changed the SSL API between versions 13.x and 14.x and 15.x so having a solution that works with all the currently supported versions of twisted [3] is not going to be a simple feat.
Would be happy to help out in implementing this or provide help on this.
The first thing to do I think would be to write up a ticket on what exactly it is you need tested on trac: https://trac.torproject.org/projects/tor/newticket?component=Ooni
I opened https://trac.torproject.org/projects/tor/ticket/16421.
David Fifield <david@bamsoftware.com> writes:
I'm less sure about how to get the certificate chain. I did some searching and didn't find a way to get the certificate chain from the twisted.web.client.Agent that templates/httpt.py uses (maybe you provide it a twisted.internet.ssl.ContextFactory somehow?).
There's probably a better way, but there is some code in "carml" which does verification of certificate-chains and might give you some hints: https://github.com/meejah/carml/blob/master/carml/command/downloadbundle.py#... (As the FIXME above this says, I believe Twisted >= 14 can do that too out of the box). With the above, extracting the chain would involve registering an OpenSSL callback and recording the cert for each depth -- perhaps there is an easier way in newer Twisted releases. HTH, -- meejah
On 2015-06-22 17:45, meejah wrote:
David Fifield <david@bamsoftware.com> writes:
I'm less sure about how to get the certificate chain. I did some searching and didn't find a way to get the certificate chain from the twisted.web.client.Agent that templates/httpt.py uses (maybe you provide it a twisted.internet.ssl.ContextFactory somehow?).
The openssl python bindings are garbage and broken. Here is how I did it: https://github.com/TheTorProject/ooni-probe/blob/feature/tor_tests/ooni/nett...
There's probably a better way, but there is some code in "carml" which does verification of certificate-chains and might give you some hints:
https://github.com/meejah/carml/blob/master/carml/command/downloadbundle.py#...
(As the FIXME above this says, I believe Twisted >= 14 can do that too out of the box). With the above, extracting the chain would involve registering an OpenSSL callback and recording the cert for each depth -- perhaps there is an easier way in newer Twisted releases.
HTH,
Would really hope so, but think it's an OpenSSL thing. Also note that settings the cipher suites doesn't seem to really do anything either... :( :( :(
participants (4)
-
Aaron Gibson -
Arturo Filastò -
David Fifield -
meejah