May 2016 - tor-commits - lists.torproject.org

[oonib/master] Vagrantfile: make sure it actually works
by art＠torproject.org 09 May '16

09 May '16

commit 0c89c56fab800675223d14c4729f1c737c44337b Author: Simone Basso <bassosimone(a)gmail.com> Date: Mon Apr 4 16:13:12 2016 +0200 Vagrantfile: make sure it actually works From the works-for-me-on-vagrant-isnt-this-ironic dept.: 1) stop provision script on error 2) update url required to get latest version of pip 3) zap virtualenv: it is not needed on a customly provisoned temporary virtual machine, plus the `mkvirtualenv` command exited with nonzero exit code for a reason that I wasn't able to understand despite I tried to do so 4) fix typo 5) don't pass nonexisting argument to pip 6) add instructions explaining how to test oonib. (Related pull request: https://github.com/TheTorProject/ooni-backend/pull/64) --- Vagrantfile | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/Vagrantfile b/Vagrantfile index 4a89157..b2c97b0 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -111,26 +111,20 @@ Vagrant.configure("2") do |config| end $setup_script = <<SCRIPT +set -e apt-get update apt-get -y install curl python-setuptools python-dev libsqlite3-dev libffi-dev echo "Updating to the latest version of PIP" cd /tmp/ -curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py +curl --fail -O https://bootstrap.pypa.io/get-pip.py python ./get-pip.py ## pip (>=1.3.0) is recommended for security reasons sudo update-alternatives --install /usr/bin/pip pip /usr/local/bin/pip 0 echo "Installing virtualenv and virtualenvwrapper..." -pip install --upgrade virtualenv virtualenvwrapper -export WORKON_HOME=~/.virtualenvs && mkdir -p $WORKON_HOME - -source /usr/local/bin/virtualenvwrapper.sh - -mkvirtualenv -a $PWD --unzip-setuptools --setuptools --no-site-packages oonib - echo "Installing Tor..." echo "deb http://deb.torproject.org/torproject.org precise main" >> /etc/apt/sources.list @@ -152,12 +146,14 @@ openssl x509 -req -days 365 -in server.csr -signkey private.key -out certificate cp oonib.conf.example oonib.conf -echo "Installing oonib depedencies" -pip install -r requirements.txt --use-mirrors - -echo "Installing oonib" -python setup.py install +echo "Installing oonib dependencies" +pip install -r requirements.txt +echo "Now:" +echo "1. vagrant ssh" +echo "2. cd /data/oonib" +echo "3. vi oonib.conf # possibly" +echo "4. sudo ./bin/oonib" SCRIPT Vagrant.configure("2") do |config|

1 0

[oonib/master] Convert msg to debug message to avoid generating too much noise
by art＠torproject.org 09 May '16

09 May '16

commit 4c279e83c12d9fc3dd6f82c6479a620f38f0b71b Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Apr 22 13:29:37 2016 +0200 Convert msg to debug message to avoid generating too much noise --- oonib/report/handlers.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/oonib/report/handlers.py b/oonib/report/handlers.py index 1e1d2ba..8c4a710 100644 --- a/oonib/report/handlers.py +++ b/oonib/report/handlers.py @@ -206,7 +206,7 @@ class ReportHandler(OONIBHandler): class UpdateReportMixin(object): def updateReport(self, report_id, parsed_request): - log.msg("Got this request %s" % parsed_request) + log.debug("Got this request %s" % parsed_request) report_filename = os.path.join(self.report_dir, report_id) try:

1 0

[oonib/master] Fix unittests to use test_start_time
by art＠torproject.org 09 May '16

09 May '16

commit 370fbfbb5485edb060b159fa3517737133806a87 Author: Arturo Filastò <arturo(a)filasto.net> Date: Thu Mar 17 15:57:23 2016 +0100 Fix unittests to use test_start_time --- oonib/report/handlers.py | 2 +- oonib/test/test_report.py | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/oonib/report/handlers.py b/oonib/report/handlers.py index 56fe20b..1e1d2ba 100644 --- a/oonib/report/handlers.py +++ b/oonib/report/handlers.py @@ -311,7 +311,7 @@ class NewReportHandlerFile(ReportHandler, UpdateReportMixin): 'test_name': self.testName, 'test_version': self.testVersion, 'input_hashes': report_data.get('input_hashes', []), - 'test_start_time': report_data['test_start_time'], + 'test_start_time': str(report_data['test_start_time']), 'data_format_version': str(report_data.get('data_format_version', '0.1.0')) } data = "---\n" + yaml.dump(content) + "...\n" diff --git a/oonib/test/test_report.py b/oonib/test/test_report.py index 63865ae..5957250 100644 --- a/oonib/test/test_report.py +++ b/oonib/test/test_report.py @@ -41,7 +41,7 @@ sample_report_entry = { 'probe_ip': '127.0.0.1', 'software_name': 'ooniprobe', 'software_version': '1.1.0', - 'start_time': 0, + 'test_start_time': '2016-01-01 12:34:56', 'test_name': 'fake_test', 'test_version': '0.1.0' } @@ -59,7 +59,7 @@ probe_city: null probe_ip: 127.0.0.1 software_name: ooniprobe software_version: 1.1.0 -start_time: 0 +test_start_time: '2016-01-01 12:34:56' test_name: fake_test test_version: 0.1.0 ... @@ -72,7 +72,7 @@ dummy_data = { 'test_version': '0.1', 'probe_asn': 'AS0', 'probe_cc': 'ZZ', - 'start_time': time.time() + 'test_start_time': '2016-01-01 12:34:56' } for _, handler in reportAPI:

1 0

[oonib/master] Use wget instead of curl for downloading packages
by art＠torproject.org 09 May '16

09 May '16

commit 6633f938312fcfd229a4f3157fb0f8ba6fe2b76b Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Apr 27 19:01:55 2016 +0200 Use wget instead of curl for downloading packages --- scripts/build_tor2web_tor.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/build_tor2web_tor.sh b/scripts/build_tor2web_tor.sh index a9042ff..85397b6 100755 --- a/scripts/build_tor2web_tor.sh +++ b/scripts/build_tor2web_tor.sh @@ -55,7 +55,7 @@ if [ $? -ne 0 ]; then fi for URL in $URLS; do - curl -LO $URL + wget $URL done BUILD=$SCRIPT_ROOT/build

1 0

[oonib/master] Set date in ChangeLog
by art＠torproject.org 09 May '16

09 May '16

commit 3af8ff3b8f387edf88d7d4a8e877b7cfbe63418a Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Apr 27 13:15:04 2016 +0200 Set date in ChangeLog * Clean up requirements.txt * Clean up Vagrantfile --- ChangeLog.rst | 2 +- Vagrantfile | 103 ------------------------------------------------------- requirements.txt | 24 +++---------- 3 files changed, 5 insertions(+), 124 deletions(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index b318708..c34857c 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,7 +1,7 @@ ChangeLog ========= -1.2.0 (XXX, XXX Apr 2016) +1.2.0 (Wed, 27 Apr 2016) ------------------------- * Add support for receiving JSON based reports diff --git a/Vagrantfile b/Vagrantfile index b2c97b0..b71e4e7 100644 --- a/Vagrantfile +++ b/Vagrantfile @@ -2,112 +2,9 @@ # vi: set ft=ruby : Vagrant.configure("2") do |config| - # All Vagrant configuration is done here. The most common configuration - # options are documented and commented below. For a complete reference, - # please see the online documentation at vagrantup.com. - - # Every Vagrant virtual environment requires a box to build off of. config.vm.box = "precise32" - - # The url from where the 'config.vm.box' box will be fetched if it - # doesn't already exist on the user's system. config.vm.box_url = "http://files.vagrantup.com/precise32.box" - - # Create a forwarded port mapping which allows access to a specific port - # within the machine from a port on the host machine. In the example below, - # accessing "localhost:8080" will access port 80 on the guest machine. - # config.vm.network :forwarded_port, guest: 80, host: 8080 - - # Create a private network, which allows host-only access to the machine - # using a specific IP. - # config.vm.network :private_network, ip: "192.168.33.10" - - # Create a public network, which generally matched to bridged network. - # Bridged networks make the machine appear as another physical device on - # your network. - # config.vm.network :public_network - - # Share an additional folder to the guest VM. The first argument is - # the path on the host to the actual folder. The second argument is - # the path on the guest to mount the folder. And the optional third - # argument is a set of non-required options. config.vm.synced_folder ".", "/data/oonib" - - # Provider-specific configuration so you can fine-tune various - # backing providers for Vagrant. These expose provider-specific options. - # Example for VirtualBox: - # - # config.vm.provider :virtualbox do |vb| - # # Don't boot with headless mode - # vb.gui = true - # - # # Use VBoxManage to customize the VM. For example to change memory: - # vb.customize ["modifyvm", :id, "--memory", "1024"] - # end - # - # View the documentation for the provider you're using for more - # information on available options. - - # Enable provisioning with Puppet stand alone. Puppet manifests - # are contained in a directory path relative to this Vagrantfile. - # You will need to create the manifests directory and a manifest in - # the file precise32.pp in the manifests_path directory. - # - # An example Puppet manifest to provision the message of the day: - # - # # group { "puppet": - # # ensure => "present", - # # } - # # - # # File { owner => 0, group => 0, mode => 0644 } - # # - # # file { '/etc/motd': - # # content => "Welcome to your Vagrant-built virtual machine! - # # Managed by Puppet.\n" - # # } - # - # config.vm.provision :puppet do |puppet| - # puppet.manifests_path = "manifests" - # puppet.manifest_file = "init.pp" - # end - - # Enable provisioning with chef solo, specifying a cookbooks path, roles - # path, and data_bags path (all relative to this Vagrantfile), and adding - # some recipes and/or roles. - # - # config.vm.provision :chef_solo do |chef| - # chef.cookbooks_path = "../my-recipes/cookbooks" - # chef.roles_path = "../my-recipes/roles" - # chef.data_bags_path = "../my-recipes/data_bags" - # chef.add_recipe "mysql" - # chef.add_role "web" - # - # # You may also specify custom JSON attributes: - # chef.json = { :mysql_password => "foo" } - # end - - # Enable provisioning with chef server, specifying the chef server URL, - # and the path to the validation key (relative to this Vagrantfile). - # - # The Opscode Platform uses HTTPS. Substitute your organization for - # ORGNAME in the URL and validation key. - # - # If you have your own Chef Server, use the appropriate URL, which may be - # HTTP instead of HTTPS depending on your configuration. Also change the - # validation key to validation.pem. - # - # config.vm.provision :chef_client do |chef| - # chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME" - # chef.validation_key_path = "ORGNAME-validator.pem" - # end - # - # If you're using the Opscode platform, your validator client is - # ORGNAME-validator, replacing ORGNAME with your organization name. - # - # If you have your own Chef Server, the default validation client name is - # chef-validator, unless you changed the configuration. - # - # chef.validation_client_name = "ORGNAME-validator" end $setup_script = <<SCRIPT diff --git a/requirements.txt b/requirements.txt index 76bf399..b8c18a8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,30 +1,14 @@ +# These are sub-dependencies that we need to pin to avoid version conflicts +pyasn1>=0.1.8 +# These are dependencies of oonibackend specifically PyYaml -Twisted>=12.2.0 -#https://pypi.python.org/packages/source/T/Twisted/Twisted-13.0.0.tar.bz2#md5=68afff4e1efd3757d934e39f70c99f57#egg=Twisted +Twisted>=13.2.0 cyclone>=1.1 -#https://pypi.python.org/packages/source/c/cyclone/cyclone-1.1.tar.gz#md5=477c5ef9cf8902e37105fe450c4d8c5d#egg=cyclone ipaddr>=2.1.10 -#https://ipaddr-py.googlecode.com/files/ipaddr-2.1.10.tar.gz#sha1=c608450b077b19773d4f1b5f1ef88b26f6650ce0#egg=ipaddr pygeoip>=0.2.6 -#https://pypi.python.org/packages/source/p/pygeoip/pygeoip-0.2.6.zip#md5=b3ac1bfcd535782bc59af78e722cf5c1#egg=pygeoip -## -## Originally fetched from the hg repo on secdev.org: -## https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy -## This is a Tor Project mirror with valid SSL/TLS certs that is stable and fast: -## -transaction>=1.4.1 -#https://pypi.python.org/packages/source/s/storm/storm-0.19.tar.gz#md5=61d1ee4cd2a08639ab917e43fa2c9265#egg=storm txtorcon>=0.7 pyOpenSSL>=0.13 zope.component>=4.0.0 zope.event>=4.0.0 zope.interface>=4.0.1 -## -## pysqlite2 is required for sqlite support in Storm, see -## https://storm.canonical.com/Manual#Dependencies-2 -## -## If not installed, Storm and ooni-backend will still work, but will keep an -## in-memory database, which in some cases may slow the server. -## -pysqlite>=2.6.0 service-identity

1 0

[oonib/master] Download some with curl some with wget
by art＠torproject.org 09 May '16

09 May '16

commit 6f1eb71d1ffb1652fb86da256fbe0897973c228f Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Apr 27 19:17:38 2016 +0200 Download some with curl some with wget For some reason curl on mlab is failing to do a SSL handshake to dist.torproject.org, while wget works. Since we validate the package with gpg anyways it's not a security issue. --- scripts/build_tor2web_tor.sh | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/scripts/build_tor2web_tor.sh b/scripts/build_tor2web_tor.sh index 85397b6..080a878 100755 --- a/scripts/build_tor2web_tor.sh +++ b/scripts/build_tor2web_tor.sh @@ -10,15 +10,17 @@ ZLIB_SHA256=36658cb768a54c1d4dec43c3116c27ed893e88b02ecfcb44f2166f9c0b7f2a0d SCRIPT_ROOT=`pwd` # Package URLS -URLS="\ -https://dist.torproject.org/tor-$TOR_VERSION.tar.gz -https://dist.torproject.org/tor-$TOR_VERSION.tar.gz.asc +CURL_URLS="\ http://zlib.net/zlib-$ZLIB_VERSION.tar.gz https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz.asc https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz https://github.com/downloads/libevent/libevent/libevent-$LIBEVENT_VERSION.t… https://github.com/downloads/libevent/libevent/libevent-$LIBEVENT_VERSION.t…" +WGET_URLS="\ +https://dist.torproject.org/tor-$TOR_VERSION.tar.gz +https://dist.torproject.org/tor-$TOR_VERSION.tar.gz.asc" + if [ `command -v shasum` ]; then SHA256SUM='shasum -a 256' fi @@ -54,10 +56,15 @@ if [ $? -ne 0 ]; then if [ $? -ne 0 ]; then exit ;fi fi -for URL in $URLS; do +for URL in $WGET_URLS; do wget $URL done +for URL in $CURL_URLS; do + curl -LO $URL +done + + BUILD=$SCRIPT_ROOT/build if [ ! -e $BUILD ]; then mkdir -p $BUILD

1 0

[oonib/master] Bump to version 1.2.0
by art＠torproject.org 09 May '16

09 May '16

commit c1f2125e5d72fc376fdf67e7f1fef20bac04eea8 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Apr 22 13:30:00 2016 +0200 Bump to version 1.2.0 --- ChangeLog.rst | 6 ++++++ oonib/__init__.py | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index 499a902..b318708 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,5 +1,11 @@ ChangeLog ========= + +1.2.0 (XXX, XXX Apr 2016) +------------------------- + +* Add support for receiving JSON based reports + 1.1.4 (Wed, 1 Oct 2014) ----------------------- diff --git a/oonib/__init__.py b/oonib/__init__.py index 2593f7d..b8e196a 100644 --- a/oonib/__init__.py +++ b/oonib/__init__.py @@ -3,7 +3,7 @@ In here we shall keep track of all variables and objects that should be instantiated only once and be common to pieces of GLBackend code. """ -__version__ = '1.1.4' +__version__ = '1.2.0' __all__ = ['Storage', 'randomStr']

1 0

[oonib/master] Raise errors when report is not found
by art＠torproject.org 09 May '16

09 May '16

commit c08a825dd1fa47b8af21a524301e6014fa46819f Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Nov 7 12:11:05 2014 +0100 Raise errors when report is not found --- oonib/report/handlers.py | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/oonib/report/handlers.py b/oonib/report/handlers.py index 9aa5df9..7f7da20 100644 --- a/oonib/report/handlers.py +++ b/oonib/report/handlers.py @@ -202,14 +202,16 @@ class UpdateReportMixin(object): log.debug("Got this request %s" % parsed_request) report_filename = os.path.join(self.report_dir, report_id) - - self.reports[report_id].refresh() + try: + self.reports[report_id].refresh() + except KeyError: + raise e.OONIBError(404, "Report not found") try: with open(report_filename, 'a+') as fd: fd.write(parsed_request['content']) except IOError: - e.OONIBError(404, "Report not found") + raise e.OONIBError(404, "Report not found") self.write({'status': 'success'})

1 0

[oonib/master] update oonibackend to 1.1.4
by art＠torproject.org 09 May '16

09 May '16

commit c22b9ac4ea45a99f503b792e24d90630a16e7aae Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Oct 1 12:24:58 2014 +0200 update oonibackend to 1.1.4 --- ChangeLog.rst | 5 +++++ oonib/__init__.py | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index 9d01327..499a902 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,5 +1,10 @@ ChangeLog ========= +1.1.4 (Wed, 1 Oct 2014) +----------------------- + +* Fix bug that lead test helpers to not being started + 1.1.3 (Mon, 29 Sep 2014) ----------------------- diff --git a/oonib/__init__.py b/oonib/__init__.py index 2147635..ba6dea3 100644 --- a/oonib/__init__.py +++ b/oonib/__init__.py @@ -3,7 +3,7 @@ In here we shall keep track of all variables and objects that should be instantiated only once and be common to pieces of GLBackend code. """ -__version__ = '1.1.3' +__version__ = '1.1.4' __all__ = ['Storage', 'randomStr']

1 0

[oonib/master] Fix error handling in report update
by art＠torproject.org 09 May '16

09 May '16

commit 8f3b5fd74bbe89602312e917d3a616b78ed544b5 Author: Arturo Filastò <art(a)fuffa.org> Date: Sat Jan 24 15:15:27 2015 +0100 Fix error handling in report update --- oonib/errors.py | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/oonib/errors.py b/oonib/errors.py index c5829d2..64b179f 100644 --- a/oonib/errors.py +++ b/oonib/errors.py @@ -2,11 +2,9 @@ from cyclone.web import HTTPError class OONIBError(HTTPError): - status_code = 500 - log_message = 'oonib-error' - - def __init__(self): - pass + def __init__(self, status_code=500, log_message='oonib-error'): + self.status_code = status_code + self.log_message = log_message class InvalidRequest(OONIBError):

1 0