[tor-bugs] #7642 [Ooni]: Secure download of python package dependencies

[Tor Bug Tracker & Wiki]

#7642: Secure download of python package dependencies
-------------------------+--------------------------------------------------
 Reporter:  hellais      |          Owner:  hellais
     Type:  enhancement  |         Status:  new    
 Priority:  normal       |      Milestone:         
Component:  Ooni         |        Version:         
 Keywords:  ooni_build,  |         Parent:         
   Points:               |   Actualpoints:         
-------------------------+--------------------------------------------------
Description changed by hellais:

Old description:

> In ooniprobe we have a set of python package dependancies that must be
> downloaded.
>
> They are described inside of:
>
> https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/requirements.txt
> for ooniprobe
>
> and
>
> https://gitweb.torproject.org/ooni-
> probe.git/blob/HEAD:/oonib/requirements.txt
>
> Currently pip does not do SSL verification of downloaded packages even if
> the repository on which the package is stored supports SSL.
> See: https://github.com/pypa/pip/issues/425
>
> Moreover not all packages that are retrieved from pypi are delivered over
> SSL.
>
> Crate.io is actually doing it right, though we don't have cert validation
> in pip so we are back to point 0 (https://crate.io/).
>
> With @aagbsn we came up with the following strategy for doing this:
>
> * We create a mirror of all the packages we need on ooni.tpo and make a
> script that downloads the packages, checks that the gpg signature of the
> bundle of packages is good and if that is the case it will install them
> with pip.
>
> Any of these strategies should work with our setup.py script and it
> should be possible to install it with python setup.py install.
>
> The GPG key that is used to sign the python .eggs should be hardcoded
> inside of of the setup.py script.
>
> * We make a list of the SSL enabled sources of the the python packages
> and download them from with request (http://docs.python-
> requests.org/en/latest/) + and verify the certs with certifi
> (http://pypi.python.org/pypi/certifi) as was suggested here:
> https://github.com/pypa/pip/issues/425#issuecomment-10022808
>
> Notes:
>
> * Use distutils
>
> * Check if gpg is installed, if not failover to insecure mode, but warn
> the user of the security issues
>
> * All *must* be part of the setup.py script.
>

> Other useful links:
>
> https://github.com/pypa/pip/pull/402
>
> https://github.com/pypa/pip/commit/efa479c50249b00493807a325f2713c592306fcb

New description:

 In ooniprobe we have a set of python package dependancies that must be
 downloaded.

 They are described inside of:

 https://gitweb.torproject.org/ooni-probe.git/blob/HEAD:/requirements.txt
 for ooniprobe

 and

 https://gitweb.torproject.org/ooni-
 probe.git/blob/HEAD:/oonib/requirements.txt

 Currently pip does not do SSL verification of downloaded packages even if
 the repository on which the package is stored supports SSL.
 See: https://github.com/pypa/pip/issues/425

 Moreover not all packages that are retrieved from pypi are delivered over
 SSL.

 Crate.io is actually doing it right, though we don't have cert validation
 in pip so we are back to point 0 (https://crate.io/).

 With @aagbsn we came up with the following strategy for doing this:

 * We create a mirror of all the packages we need on ooni.tpo and make a
 script that downloads the packages, checks that the gpg signature of the
 bundle of packages is good and if that is the case it will install them
 with pip.

 Any of these strategies should work with our setup.py script and it should
 be possible to install it with python setup.py install.

 The GPG key that is used to sign the python .eggs should be hardcoded
 inside of of the setup.py script.

 * We make a list of the SSL enabled sources of the the python packages and
 download them from with request (http://docs.python-
 requests.org/en/latest/) + and verify the certs with certifi
 (http://pypi.python.org/pypi/certifi) as was suggested here:
 https://github.com/pypa/pip/issues/425#issuecomment-10022808

 * We put all the dependencies as addresses to a Tor Hidden Services and
 download them via torsocks python setup.py install. This way
 authentication and encryption is handled by Tor. (perhaps this is a
 feature that crate.io should look into supporting?)

 Notes:

 * Use distutils

 * Check if gpg is installed, if not failover to insecure mode, but warn
 the user of the security issues

 * All *must* be part of the setup.py script.


 Other useful links:

 https://github.com/pypa/pip/pull/402

 https://github.com/pypa/pip/commit/efa479c50249b00493807a325f2713c592306fcb

--

-- 
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/7642#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online