Fwd: Ooni / M-Lab Deployment Automation Script
Hi ooni-dev. For your viewing pleasure, here is a forward about tickets related to deploying M-Lab on Ooni (without integration into mlab-ns). We'll send these announcements directly to ooni-dev henceforth. Enjoy. ---------- Forwarded message ---------- From: Taylor Hornby <taylor@leastauthority.com> Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script To: Liz Pruszko Steininger <steiningerl@rfa.org>, Dan Meredith <meredithd@rfa.org>, lynna@rfa.org, Roger Dingledine <arma@mit.edu>, Arturo Filastò <art@torproject.org>, Meredith Whittaker <meredithrachel@google.com>, Will Hawkins <hawkinsw@opentechinstitute.org>, Jordan McCarthy <mccarthy@opentechinstitute.org>, critzo@opentechinstitute.org Cc: "consultancy@leastauthority.com" <consultancy@leastauthority.com>, taylor@leastauthority.com, Zooko Wilcox-OHearn <zooko@leastauthority.com>, Jessica Augustus <jessica@leastauthority.com>, Nathan Wilcox <nathan@leastauthority.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dear OTF, Ooni, and M-Lab, We've finished our work for Milestone C. This milestone is about writing a script for automating the process of deploying Ooni to M-Lab slices. Since such a script had already been written before we arrived, we shifted our goals for this milestone as follows: 1. Usability and reliability testing of the existing deployment automation scripts. 2. Fix any issues that we identified during that process. Also part of Milestone C is the credential rotation deliverable, which is no longer relevant because the mechanism for distributing .ooni addresses has changed since the contract was negotiated. This is documented in the following ticket: https://github.com/m-lab-tools/ooni-support/issues/32 As part of the first (new) goal, we ran through a deployment several times using the scripts, which is documented in this ticket: https://github.com/m-lab-tools/ooni-support/issues/17 The issues we encountered are summarized in this umbrella ticket: https://github.com/m-lab-tools/ooni-support/issues/21 Each issue was split out into separate tickets: #23: Fix or document deployment gotcha of deleting $HOME https://github.com/m-lab-tools/ooni-support/issues/23 #24: Specify dependency on yum-cron for installation. https://github.com/m-lab-tools/ooni-support/issues/24 #25: Missing ``/etc/mlab/slice-functions`` https://github.com/m-lab-tools/ooni-support/issues/25 #26: Add root uid documentation and check in initialize.sh ... https://github.com/m-lab-tools/ooni-support/issues/26 #27: Fix initialize.sh to create ``/var/spool/mlab_ooni`` https://github.com/m-lab-tools/ooni-support/issues/27 #29: Ensure test_helpers can be reached from the public internet https://github.com/m-lab-tools/ooni-support/issues/29 #28: ``stop.sh`` failed to stop multiple processes. https://github.com/m-lab-tools/ooni-support/issues/28 #40: Make openssl an explicit dependency of the Ooni RPM https://github.com/m-lab-tools/ooni-support/issues/40 #12641: IStreamClientEndpointStringParser is Deprecated https://trac.torproject.org/projects/tor/ticket/12641#ticket #41: Install service_identity https://github.com/m-lab-tools/ooni-support/issues/41 #42: prepare.sh violates ooni-backend's README instructions https://github.com/m-lab-tools/ooni-support/issues/42 #44: Is dependency installation vulnerable to MITM attacks? https://github.com/m-lab-tools/ooni-support/issues/44 All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below). We also found a new security vulnerability in Ooni: #12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticket Our fixes to the issues are contained in three pull requests: #36: Improvements to the README.md. https://github.com/m-lab-tools/ooni-support/pull/36 #37: Improvements to the initialize.sh script. https://github.com/m-lab-tools/ooni-support/pull/37 #43: Install dependencies according to ooni-backend README https://github.com/m-lab-tools/ooni-support/pull/43 Note that pull request #36 contains work from Milestone B as well. Please let us know if you have any suggestions, questions, or concerns. - -- Taylor Hornby Least Authoritarian Email: taylor@leastauthority.com PGP: CE3 F8ED D999 F066 C2E2 9124 F6D4 D32C E31C 99FE Twitter: @DefuseSec -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJTxvB6AAoJEPbU0yzjHJn+ccQQALHndy9a7kuz9MDifXrS+z2s uzzizfUK5EZB12G+mFaAfqF/t8pa/zcD2mZ2ycpna8AruhZPH5x9poxoZI/Agz59 gb8xlaJMwOJWFmeBHkn60Jz/zyaVZF0xTkQ8YhGKeqzXkfo1Vp+EI0ZFcanLKIvZ EaL+zHPZNyb5SQXOTiiy9OpyCXhboNOaXQru9GgxvBYJFosEeKA6aLVVyPx2ZSci irBg0KNt8jCkPQtH5YjkCrjKwjNI40niBpVU3B/jz5CvMb4f5B08ZjqL7t+Hhpul /c9dbYV7VILkq2/Q1/G5SNiosl8SUkjf3U8hDmb0pQpMeoZ/aE9V3AWDCrcABNvD dbJF9K3FD2YRrRjCBPNO0KWxXCU3X45oc58JAQbOuHbH6AVPazZB9WRgdu1pAisv Ikidl1yovoqxJkN3iEybfX3I2p1geMrDB4Q/z7FOdRP2dBNzTKR7zkTvJdXyulZf q1yI+Qav7MVQBGdCN87jX8xtt1eUXMEQXu7TVcxcNlvfgea5Uewv9s5l2/84fYa3 qu0Kp/+8BOioXIbG09PJREHzoHEeNSJvLqF7B6d5r3enBv5H0YvC194s8wjkZGTz sQBsAl4HI+7xEdeQ44vez+SV11i9NkEyHo1rwqh4T4glM8yXcdQ4buZaMwcXJ2V7 0UKWa6Sj2n563Dclb47K =RS7C -----END PGP SIGNATURE----- -- Nathan Wilcox Least Authoritarian email: nathan@leastauthority.com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993
Hi Least Authoritarians, Thanks for the report. I will be away until the 30th of July so I will not be able to resolve these issues until that date. I will reply inline. On 7/16/14, 11:44 PM, Nathan Wilcox wrote:
---------- Forwarded message ---------- From: Taylor Hornby <taylor@leastauthority.com> Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script
All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below).
I will look more into #12641 and see if it is something that can be fixed in ooni-backend, but from the looks of it it seems like a twisted bug. We don't use the IStreamClientEndpointStringParser interface at all and I see some other projects on the internet having the same issue: https://github.com/getsentry/raven-python/issues/466
We also found a new security vulnerability in Ooni:
#12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticket
As I commented on the ticket I believe that there is not so much we can do here except perhaps improve the documentation of ooni-backend. I thought it was clear from the README.md that the user should verify that all the commands that are run do not fail. If the pip command fails, because it did not download a dependency, then you are correct it is possible for an attacker to serve us a tampered dependency. This has to do with the fact that python dependency installation is quite broken. The script in the mlab support should check for the return value of pip and make sure it's 0 and if not hard fail. For non mlab deployment I think the best path is to start uploading ooni-backend to pip and suggest to install it only via pip without downloading the git repo. ~ Art.
On Thu, Jul 17, 2014 at 6:42 AM, Arturo Filastò <art@torproject.org> wrote:
Hi Least Authoritarians,
Thanks for the report. I will be away until the 30th of July so I will not be able to resolve these issues until that date.
I will reply inline.
On 7/16/14, 11:44 PM, Nathan Wilcox wrote:
---------- Forwarded message ---------- From: Taylor Hornby <taylor@leastauthority.com> Date: Wed, Jul 16, 2014 at 2:42 PM Subject: Ooni / M-Lab Deployment Automation Script
All of these tickets, with the exception of #40, #12641, #41, #42, and #44 are now closed. Ticket #40 is a minor issue, but would involve significant design decisions on M-Lab's part, so we left it open for M-Lab to close. Ticket #12641 is about the use of a deprecated function in Ooni, to be fixed by the Ooni team. Ticket #42 is about a missing dependency in Ooni for the Ooni team to fix. Ticket #44 is about a security vulnerability that requires Ooni collaboration to resolve (see below).
I will look more into #12641 and see if it is something that can be fixed in ooni-backend, but from the looks of it it seems like a twisted bug.
We don't use the IStreamClientEndpointStringParser interface at all and I see some other projects on the internet having the same issue:
For the time being I'm not aware of any functional or security problem, just an annoying warning message.
We also found a new security vulnerability in Ooni:
#12642: Can Network Attacker Downgrade Dependency Install Security? https://trac.torproject.org/projects/tor/ticket/12642#ticket
As I commented on the ticket I believe that there is not so much we can do here except perhaps improve the documentation of ooni-backend.
I thought it was clear from the README.md that the user should verify that all the commands that are run do not fail. If the pip command fails, because it did not download a dependency, then you are correct it is possible for an attacker to serve us a tampered dependency.
One thought is to hack "python ./setup.py install" to execute the pip command internally, but I'm not at all sure that's sane yet.
This has to do with the fact that python dependency installation is quite broken.
No disagreement here. ;-) In fact, several different projects are attempting to do similar things to Ooni here, so I'm hoping some better tools emerge. I've heard peep attempts to do something better than pip for installing only verified dependencies...
The script in the mlab support should check for the return value of pip and make sure it's 0 and if not hard fail.
Agreed. Whereas this issue for ooni is kind of about enhancement or improvement to make it more robust, for ooni-support it's an outright bug: https://github.com/m-lab-tools/ooni-support/issues/44
For non mlab deployment I think the best path is to start uploading ooni-backend to pip and suggest to install it only via pip without downloading the git repo.
+1 I am a fan of this distribution target / instructions in general for modern python. However, there are cases where end-users need to be able to securely install dependencies from tarballs, git clones, etc... I'd love to learn why "pip install ." doesn't do the right thing, or how to prevent "python ./setup.py install" from doing the wrong thing, but neither of those tasks is on my likely-to-be-completed queue, at the moment.
~ Art. _______________________________________________ ooni-dev mailing list ooni-dev@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/ooni-dev
-- Nathan Wilcox Least Authoritarian email: nathan@leastauthority.com twitter: @least_nathan PGP: 11169993 / AAAC 5675 E3F7 514C 67ED E9C9 3BFE 5263 1116 9993
participants (2)
-
Arturo Filastò -
Nathan Wilcox