Hi ooni-dev. For your viewing pleasure, here is a forward about tickets related to deploying M-Lab on Ooni (without integration into mlab-ns). We'll send these announcements directly to ooni-dev henceforth. Enjoy.
---------- Forwarded message ---------- From: Taylor Hornby taylor@leastauthority.com Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script To: Liz Pruszko Steininger steiningerl@rfa.org, Dan Meredith meredithd@rfa.org, lynna@rfa.org, Roger Dingledine arma@mit.edu, Arturo Filastò art@torproject.org, Meredith Whittaker meredithrachel@google.com, Will Hawkins hawkinsw@opentechinstitute.org, Jordan McCarthy mccarthy@opentechinstitute.org, critzo@opentechinstitute.org Cc: "consultancy@leastauthority.com" consultancy@leastauthority.com, taylor@leastauthority.com, Zooko Wilcox-OHearn zooko@leastauthority.com, Jessica Augustus jessica@leastauthority.com, Nathan Wilcox nathan@leastauthority.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Dear OTF, Ooni, and M-Lab,
We've finished our work for Milestone C. This milestone is about writing a script for automating the process of deploying Ooni to M-Lab slices. Since such a script had already been written before we arrived, we shifted our goals for this milestone as follows:
1. Usability and reliability testing of the existing deployment automation scripts. 2. Fix any issues that we identified during that process.
Also part of Milestone C is the credential rotation deliverable, which is no longer relevant because the mechanism for distributing .ooni addresses has changed since the contract was negotiated. This is documented in the following ticket:
https://github.com/m-lab-tools/ooni-support/issues/32
As part of the first (new) goal, we ran through a deployment several times using the scripts, which is documented in this ticket:
https://github.com/m-lab-tools/ooni-support/issues/17
The issues we encountered are summarized in this umbrella ticket:
https://github.com/m-lab-tools/ooni-support/issues/21
Each issue was split out into separate tickets:
#23: Fix or document deployment gotcha of deleting $HOME https://github.com/m-lab-tools/ooni-support/issues/23
#24: Specify dependency on yum-cron for installation. https://github.com/m-lab-tools/ooni-support/issues/24
#25: Missing ``/etc/mlab/slice-functions`` https://github.com/m-lab-tools/ooni-support/issues/25
#26: Add root uid documentation and check in initialize.sh ... https://github.com/m-lab-tools/ooni-support/issues/26
#27: Fix initialize.sh to create ``/var/spool/mlab_ooni`` https://github.com/m-lab-tools/ooni-support/issues/27
#29: Ensure test_helpers can be reached from the public internet https://github.com/m-lab-tools/ooni-support/issues/29
#28: ``stop.sh`` failed to stop multiple processes. https://github.com/m-lab-tools/ooni-support/issues/28
#40: Make openssl an explicit dependency of the Ooni RPM https://github.com/m-lab-tools/ooni-support/issues/40
#12641: IStreamClientEndpointStringParser is Deprecated https://trac.torproject.org/projects/tor/ticket/12641#ticket
#41: Install service_identity https://github.com/m-lab-tools/ooni-support/issues/41
#42: prepare.sh violates ooni-backend's README instructions https://github.com/m-lab-tools/ooni-support/issues/42
#44: Is dependency installation vulnerable to MITM attacks? https://github.com/m-lab-tools/ooni-support/issues/44
All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below).
We also found a new security vulnerability in Ooni:
#12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticket
Our fixes to the issues are contained in three pull requests:
#36: Improvements to the README.md. https://github.com/m-lab-tools/ooni-support/pull/36
#37: Improvements to the initialize.sh script. https://github.com/m-lab-tools/ooni-support/pull/37
#43: Install dependencies according to ooni-backend README https://github.com/m-lab-tools/ooni-support/pull/43
Note that pull request #36 contains work from Milestone B as well.
Please let us know if you have any suggestions, questions, or concerns.
- -- Taylor Hornby Least Authoritarian
Email: taylor@leastauthority.com PGP: CE3 F8ED D999 F066 C2E2 9124 F6D4 D32C E31C 99FE Twitter: @DefuseSec
Hi Least Authoritarians,
Thanks for the report. I will be away until the 30th of July so I will not be able to resolve these issues until that date.
I will reply inline.
On 7/16/14, 11:44 PM, Nathan Wilcox wrote:
---------- Forwarded message ---------- From: Taylor Hornby taylor@leastauthority.com Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script
All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below).
I will look more into #12641 and see if it is something that can be fixed in ooni-backend, but from the looks of it it seems like a twisted bug.
We don't use the IStreamClientEndpointStringParser interface at all and I see some other projects on the internet having the same issue:
https://github.com/getsentry/raven-python/issues/466
We also found a new security vulnerability in Ooni:
#12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticket
As I commented on the ticket I believe that there is not so much we can do here except perhaps improve the documentation of ooni-backend.
I thought it was clear from the README.md that the user should verify that all the commands that are run do not fail. If the pip command fails, because it did not download a dependency, then you are correct it is possible for an attacker to serve us a tampered dependency.
This has to do with the fact that python dependency installation is quite broken.
The script in the mlab support should check for the return value of pip and make sure it's 0 and if not hard fail.
For non mlab deployment I think the best path is to start uploading ooni-backend to pip and suggest to install it only via pip without downloading the git repo.
~ Art.
On Thu, Jul 17, 2014 at 6:42 AM, Arturo Filastò art@torproject.org wrote:
Hi Least Authoritarians,
Thanks for the report. I will be away until the 30th of July so I will not be able to resolve these issues until that date.
I will reply inline.
On 7/16/14, 11:44 PM, Nathan Wilcox wrote:
---------- Forwarded message ---------- From: Taylor Hornby taylor@leastauthority.com Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script
All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below).
I will look more into #12641 and see if it is something that can be fixed in ooni-backend, but from the looks of it it seems like a twisted bug.
We don't use the IStreamClientEndpointStringParser interface at all and I see some other projects on the internet having the same issue:
For the time being I'm not aware of any functional or security problem, just an annoying warning message.
We also found a new security vulnerability in Ooni:
#12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticketAs I commented on the ticket I believe that there is not so much we can do here except perhaps improve the documentation of ooni-backend.
I thought it was clear from the README.md that the user should verify that all the commands that are run do not fail. If the pip command fails, because it did not download a dependency, then you are correct it is possible for an attacker to serve us a tampered dependency.
One thought is to hack "python ./setup.py install" to execute the pip command internally, but I'm not at all sure that's sane yet.
This has to do with the fact that python dependency installation is quite broken.
No disagreement here. ;-)
In fact, several different projects are attempting to do similar things to Ooni here, so I'm hoping some better tools emerge. I've heard peep attempts to do something better than pip for installing only verified dependencies...
The script in the mlab support should check for the return value of pip and make sure it's 0 and if not hard fail.
Agreed. Whereas this issue for ooni is kind of about enhancement or improvement to make it more robust, for ooni-support it's an outright bug:
https://github.com/m-lab-tools/ooni-support/issues/44
For non mlab deployment I think the best path is to start uploading ooni-backend to pip and suggest to install it only via pip without downloading the git repo.
+1
I am a fan of this distribution target / instructions in general for modern python. However, there are cases where end-users need to be able to securely install dependencies from tarballs, git clones, etc...
I'd love to learn why "pip install ." doesn't do the right thing, or how to prevent "python ./setup.py install" from doing the wrong thing, but neither of those tasks is on my likely-to-be-completed queue, at the moment.
~ Art. _______________________________________________ ooni-dev mailing list ooni-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
-
Arturo Filastò -
Nathan Wilcox