November 2012 - tor-commits - lists.torproject.org

[ooni-probe/master] Change required version of txtorcon to that which is stored on pypi.
by art＠torproject.org 11 Nov '12

11 Nov '12

commit d88a54c113869fecc83374d340c7ecfe5f8f1220 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 20:40:26 2012 +0100 Change required version of txtorcon to that which is stored on pypi. * Refactor README.md with easier installation of scapy. --- README.md | 37 +++++++++++-------------------------- requirements.txt | 2 +- 2 files changed, 12 insertions(+), 27 deletions(-) diff --git a/README.md b/README.md index a6e3469..98edd1e 100644 --- a/README.md +++ b/README.md @@ -37,31 +37,9 @@ The python dependencies required for running ooniprobe are: * BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ * txtorcon: https://github.com/meejah/txtorcon -### Installing scapy - -Unfortunately the version of scapy that is stored on pypi is out of date, this -means that you will have to manually download and install scapy. - -This can be done like so: - - wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz - tar xzf scapy-latest.tar.gz - cd scapy-* - python setup.py install - -If you are not inside of a virtual env the last command will have to be: - - sudo python setup.py install - -### Installing the rest of dependencies - -The other dependencies can be installed from the requirements.txt with: - - sudo pip install -r requirements.txt - -You are highly recommended to do so from inside of a virtual environment, since -pip does not download the packages via SSL and you will need to install it -system wide. +You are highly recommended to install depedencies from inside of a virtual +environment, since pip does not download the packages via SSL and you will need +to install it system wide. This will require you to have installed virtualenv. @@ -73,7 +51,14 @@ To create a new virtual environment do Then install OONI with: - pip install -r requirements.txt + pip install https://hg.secdev.org/scapy/archive/tip.zip + pip install -r requirements.txt + +If you decided not to install virtual env and want to download code via http as +run it as root, you may easily do so with: + + pip install https://hg.secdev.org/scapy/archive/tip.zip + sudo pip install -r requirements.txt ## Including your geo data in the test report diff --git a/requirements.txt b/requirements.txt index 4b7e59e..e9a7bfc 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,7 +14,7 @@ repoze.sphinx.autointerface>=0.7.1 scapy>=2.1.0 storm>=0.19 transaction>=1.3.0 -txtorcon>=0.7 +txtorcon>=0.6 wsgiref>=0.1.2 zope.component>=4.0.0 zope.event>=4.0.0

1 0

[ooni-probe/master] Update nettests/core/url_list.py
by art＠torproject.org 11 Nov '12

11 Nov '12

commit c67d69ec432e32479daca2ed65b27529260c9ba5 Author: jonmtoz <jonmtoz(a)gmail.com> Date: Sat Nov 10 22:54:34 2012 +0200 Update nettests/core/url_list.py --- nettests/core/url_list.py | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-) diff --git a/nettests/core/url_list.py b/nettests/core/url_list.py index 43f819e..9db912d 100644 --- a/nettests/core/url_list.py +++ b/nettests/core/url_list.py @@ -21,4 +21,20 @@ class URLList(httpt.HTTPTest): return self.doRequest(self.url) + def test_post(self): + if self.input: + self.url = self.input + else: + raise Exception("No input specified") + + return self.doRequest(self.url, method="POST") + + def test_put(self): + if self.input: + self.url = self.input + else: + raise Exception("No input specified") + + return self.doRequest(self.url, method="PUT") +

1 0

[ooni-probe/master] Update nettests/core/url_list.py
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 28241124e6094b224f4a3b4f6c0a8e8a69a7eeb6 Author: jonmtoz <jonmtoz(a)gmail.com> Date: Sat Nov 10 23:04:06 2012 +0200 Update nettests/core/url_list.py --- nettests/core/url_list.py | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/nettests/core/url_list.py b/nettests/core/url_list.py index 9db912d..fe678be 100644 --- a/nettests/core/url_list.py +++ b/nettests/core/url_list.py @@ -19,7 +19,7 @@ class URLList(httpt.HTTPTest): else: raise Exception("No input specified") - return self.doRequest(self.url) + return self.doRequest(self.url, method="GET") def test_post(self): if self.input:

1 0

[ooni-probe/master] Merge pull request #7 from jonmtoz/master
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 8e1b6ba589bd426dc551efa5d54eaf491ee96db4 Merge: a744bd1 2824112 Author: Arturo Filastò <hellais(a)gmail.com> Date: Sun Nov 11 10:22:33 2012 -0800 Merge pull request #7 from jonmtoz/master Added POST and PUT requests nettests/core/url_list.py | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-)

1 0

[ooni-probe/master] Merge branch 'master' of github.com:hellais/ooni-probe
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 7b8637806845a6dcb295368d7074d7580d679f7e Merge: b97ee25 8e1b6ba Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 19:39:23 2012 +0100 Merge branch 'master' of github.com:hellais/ooni-probe * 'master' of github.com:hellais/ooni-probe: Update nettests/core/url_list.py Update nettests/core/url_list.py nettests/core/url_list.py | 18 +++++++++++++++++- 1 files changed, 17 insertions(+), 1 deletions(-)

1 0

[ooni-probe/master] Remove cyclone, numpy and bs4 from the dependencies of ooniprobe.
by art＠torproject.org 11 Nov '12

11 Nov '12

commit b97ee255c462f00c37a7a0be6c2d411b13aa9e17 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 19:35:06 2012 +0100 Remove cyclone, numpy and bs4 from the dependencies of ooniprobe. * These should only be depedencies of oonib and of "advanced" users of ooniprobe --- oonib/requirements.txt | 24 ++++++++++++++++++++++++ requirements.txt | 45 +++++++++++++++++++++------------------------ 2 files changed, 45 insertions(+), 24 deletions(-) diff --git a/oonib/requirements.txt b/oonib/requirements.txt new file mode 100644 index 0000000..f9d7541 --- /dev/null +++ b/oonib/requirements.txt @@ -0,0 +1,24 @@ +BeautifulSoup==3.2.1 +Jinja2==2.6 +PyYAML==3.10 +Pygments==1.5 +Sphinx==1.1.3 +Twisted==12.2.0 +argparse==1.2.1 +cyclone==1.0-rc13 +distribute==0.6.24 +dnspython==1.10.0 +docutils==0.9.1 +ipaddr==2.1.10 +numpy==1.6.2 +pyOpenSSL==0.13 +pygeoip==0.2.5 +repoze.sphinx.autointerface==0.7.1 +scapy==2.1.0 +storm==0.19 +transaction==1.3.0 +txtorcon==0.7 +wsgiref==0.1.2 +zope.component==4.0.0 +zope.event==4.0.0 +zope.interface==4.0.1 diff --git a/requirements.txt b/requirements.txt index f9d7541..4b7e59e 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,24 +1,21 @@ -BeautifulSoup==3.2.1 -Jinja2==2.6 -PyYAML==3.10 -Pygments==1.5 -Sphinx==1.1.3 -Twisted==12.2.0 -argparse==1.2.1 -cyclone==1.0-rc13 -distribute==0.6.24 -dnspython==1.10.0 -docutils==0.9.1 -ipaddr==2.1.10 -numpy==1.6.2 -pyOpenSSL==0.13 -pygeoip==0.2.5 -repoze.sphinx.autointerface==0.7.1 -scapy==2.1.0 -storm==0.19 -transaction==1.3.0 -txtorcon==0.7 -wsgiref==0.1.2 -zope.component==4.0.0 -zope.event==4.0.0 -zope.interface==4.0.1 +Jinja2>=2.6 +PyYAML>=3.10 +Pygments>=1.5 +Sphinx>=1.1.3 +Twisted>=12.0.0 +argparse>=1.2.1 +distribute>=0.6.24 +dnspython>=1.10.0 +docutils>=0.9.1 +ipaddr>=2.1.10 +pyOpenSSL>=0.13 +pygeoip>=0.2.5 +repoze.sphinx.autointerface>=0.7.1 +scapy>=2.1.0 +storm>=0.19 +transaction>=1.3.0 +txtorcon>=0.7 +wsgiref>=0.1.2 +zope.component>=4.0.0 +zope.event>=4.0.0 +zope.interface>=4.0.1

1 0

[ooni-probe/master] Add sudo to commands that must be run as root
by art＠torproject.org 11 Nov '12

11 Nov '12

commit a744bd1b8ae1d29633bc27b4f95ce55d60cafe40 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 19:07:59 2012 +0100 Add sudo to commands that must be run as root --- README.md | 7 ++----- 1 files changed, 2 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 9d41eea..a6e3469 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ If you are not inside of a virtual env the last command will have to be: The other dependencies can be installed from the requirements.txt with: - pip install -r requirements.txt + sudo pip install -r requirements.txt You are highly recommended to do so from inside of a virtual environment, since pip does not download the packages via SSL and you will need to install it @@ -65,7 +65,7 @@ system wide. This will require you to have installed virtualenv. - apt-get install python-virtualenv + sudo apt-get install python-virtualenv To create a new virtual environment do @@ -75,7 +75,6 @@ Then install OONI with: pip install -r requirements.txt - ## Including your geo data in the test report Including geografical information on where your probe is located helps us @@ -92,7 +91,6 @@ directory is located for example: geoip_data_dir: /home/your_user/ooni-probe/data/ - ## Running some tests To see the possible command line options run: @@ -116,4 +114,3 @@ test result, nor create a pcap file. This behavior can be personalized by editing your ooniprobe.conf configuration file. -

1 0

[ooni-probe/master] Fix typo in README for installation of scapy
by art＠torproject.org 11 Nov '12

11 Nov '12

commit df94c8bbbcdd7ec46976d54c00dd312154a910c4 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 18:58:11 2012 +0100 Fix typo in README for installation of scapy * Make import of pygeoip soft --- README.md | 3 +-- ooni/utils/geodata.py | 6 +++++- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c3ef8a6..9d41eea 100644 --- a/README.md +++ b/README.md @@ -46,8 +46,7 @@ This can be done like so: wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz tar xzf scapy-latest.tar.gz - cd scapy-latest.tar.gz - cd scapy-latest + cd scapy-* python setup.py install If you are not inside of a virtual env the last command will have to be: diff --git a/ooni/utils/geodata.py b/ooni/utils/geodata.py index 5c3c481..447d2eb 100644 --- a/ooni/utils/geodata.py +++ b/ooni/utils/geodata.py @@ -10,7 +10,6 @@ import re import os -import pygeoip from twisted.web.client import Agent from twisted.internet import reactor, defer, protocol @@ -18,6 +17,11 @@ from twisted.internet import reactor, defer, protocol from ooni.utils import log, net from ooni import config +try: + import pygeoip +except ImportError: + log.err("Unable to import pygeoip. We will not be able to run geo IP related measurements") + @defer.inlineCallbacks def myIP(): target_site = 'https://check.torproject.org/'

1 0

[ooni-probe/master] Move scapy installation before all the rest
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 916adfd5bf222f486412029f023c51cc5b9e69cd Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 18:53:54 2012 +0100 Move scapy installation before all the rest --- README.md | 37 ++++++++++++++++++++----------------- 1 files changed, 20 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index b502c79..c3ef8a6 100644 --- a/README.md +++ b/README.md @@ -37,7 +37,26 @@ The python dependencies required for running ooniprobe are: * BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ * txtorcon: https://github.com/meejah/txtorcon -They can be installed from the requirements.txt with: +### Installing scapy + +Unfortunately the version of scapy that is stored on pypi is out of date, this +means that you will have to manually download and install scapy. + +This can be done like so: + + wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz + tar xzf scapy-latest.tar.gz + cd scapy-latest.tar.gz + cd scapy-latest + python setup.py install + +If you are not inside of a virtual env the last command will have to be: + + sudo python setup.py install + +### Installing the rest of dependencies + +The other dependencies can be installed from the requirements.txt with: pip install -r requirements.txt @@ -57,22 +76,6 @@ Then install OONI with: pip install -r requirements.txt -## Installing scapy - -Unfortunately the version of scapy that is stored on pypi is out of date, this -means that you will have to manually download and install scapy. - -This can be done like so: - - wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz - tar xzf scapy-latest.tar.gz - cd scapy-latest.tar.gz - cd scapy-latest - python setup.py install - -If you are not inside of a virtual env the last command will have to be: - - sudo python setup.py install ## Including your geo data in the test report

1 0

[ooni-probe/master] Fix indentation of readme file
by art＠torproject.org 11 Nov '12

11 Nov '12

commit 465c0e6a325317e17830b829c78e1a2f8be282e3 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun Nov 11 18:51:27 2012 +0100 Fix indentation of readme file --- README.md | 30 +++++++++++++++--------------- 1 files changed, 15 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index bb17cc7..b502c79 100644 --- a/README.md +++ b/README.md @@ -39,7 +39,7 @@ The python dependencies required for running ooniprobe are: They can be installed from the requirements.txt with: - pip install -r requirements.txt + pip install -r requirements.txt You are highly recommended to do so from inside of a virtual environment, since pip does not download the packages via SSL and you will need to install it @@ -47,15 +47,15 @@ system wide. This will require you to have installed virtualenv. - apt-get install python-virtualenv + apt-get install python-virtualenv To create a new virtual environment do - virtualenv env + virtualenv env Then install OONI with: - pip install -r requirements.txt + pip install -r requirements.txt ## Installing scapy @@ -64,15 +64,15 @@ means that you will have to manually download and install scapy. This can be done like so: - wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz - tar xzf scapy-latest.tar.gz - cd scapy-latest.tar.gz - cd scapy-latest - python setup.py install + wget http://www.secdev.org/projects/scapy/files/scapy-latest.tar.gz + tar xzf scapy-latest.tar.gz + cd scapy-latest.tar.gz + cd scapy-latest + python setup.py install If you are not inside of a virtual env the last command will have to be: - sudo python setup.py install + sudo python setup.py install ## Including your geo data in the test report @@ -83,29 +83,29 @@ inside of ooniprobe.conf If you wish to include geografical data in the test report, you will have to go to the data/ directory and run: - make geoip + make geoip Then edit your ooniprobe.conf to point to the absolute path of where the data/ directory is located for example: - geoip_data_dir: /home/your_user/ooni-probe/data/ + geoip_data_dir: /home/your_user/ooni-probe/data/ ## Running some tests To see the possible command line options run: - ./bin/ooniprobe --help + ./bin/ooniprobe --help For interesting tests to run look in the nettests/core/ directory. To run a test you can do so with: - ./bin/ooniprobe -o report_file_name path/to/test.py + ./bin/ooniprobe -o report_file_name path/to/test.py Normally tests take options, you can see them with: - ./bin/ooniprobe -o report_file_name path/to/test.py --help + ./bin/ooniprobe -o report_file_name path/to/test.py --help ## Configuration

1 0