[tor-bugs] #7642 [Ooni]: Secure download of python package dependencies

[Tor Bug Tracker & Wiki]

#7642: Secure download of python package dependencies
-------------------------+--------------------------------------------------
 Reporter:  hellais      |          Owner:  hellais
     Type:  enhancement  |         Status:  new    
 Priority:  normal       |      Milestone:         
Component:  Ooni         |        Version:         
 Keywords:  ooni_build,  |         Parent:         
   Points:               |   Actualpoints:         
-------------------------+--------------------------------------------------
 In ooniprobe we have a set of python package dependancies that must be
 downloaded.

 They are described inside of:

 https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/requirements.txt
 for ooniprobe

 and

 https://gitweb.torproject.org/ooni-
 probe.git/blob/HEAD:/oonib/requirements.txt

 Currently pip does not do SSL verification of downloaded packages even if
 the repository on which the package is stored supports SSL.
 See: https://github.com/pypa/pip/issues/425

 Moreover not all packages that are retrieved from pypi are delivered over
 SSL.

 Crate.io is actually doing it right, though we don't have cert validation
 in pip so we are back to point 0 (https://crate.io/).

 With @aagbsn we came up with the following strategy for doing this:

 * We create a mirror of all the packages we need on ooni.tpo and make a
 script that downloads the packages, checks that the gpg signature of the
 bundle of packages is good and if that is the case it will install them
 with pip.

 Any of these strategies should work with our setup.py script and it should
 be possible to install it with python setup.py install.

 The GPG key that is used to sign the python .eggs should be hardcoded
 inside of of the setup.py script.

 * We make a list of the SSL enabled sources of the the python packages and
 download them from with request (http://docs.python-
 requests.org/en/latest/) + and verify the certs with certifi
 (http://pypi.python.org/pypi/certifi) as was suggested here:
 https://github.com/pypa/pip/issues/425#issuecomment-10022808

 Notes:

 * Use distutils

 * Check if gpg is installed, if not failover to insecure mode, but warn
 the user of the security issues

 * All *must* be part of the setup.py script.


 Other useful links:

 https://github.com/pypa/pip/pull/402

 https://github.com/pypa/pip/commit/efa479c50249b00493807a325f2713c592306fcb

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/7642>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online