August 2014 - tor-commits - lists.torproject.org

[ooni-probe/master] Add heading to changelog
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 497127ce68a80e07a3162b42942dc87dd4f0eef4 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 14:17:18 2014 +0200 Add heading to changelog --- ChangeLog.rst | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index e41924c..af4f6c8 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,3 +1,6 @@ +Changelog +========= + v1.1.0 ------ @@ -344,10 +347,10 @@ v0.0.4-alpha * Add multi protocol multi port traceroute for UDP and TCP * Implement basic HTTP request test that does capitalization variations on the -HTTP method. + HTTP method. * Bugfixing and refactoring of txscapy for sending and receiving of scapy -packets. + packets. v0.0.3-alpha ------------

1 0

[ooni-probe/master] Add dates to changelog
by art＠torproject.org 19 Aug '14

19 Aug '14

commit f2ede07db85b8d1e276797eb5c75a2874bc3fa5e Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 14:29:11 2014 +0200 Add dates to changelog --- ChangeLog.rst | 64 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index af4f6c8..578f783 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,8 +1,8 @@ Changelog ========= -v1.1.0 ------- +v1.1.0 (Tue, 19 Aug 2014) +------------------------- In this new release of ooniprobe we have added a new command line tool for listing the reports that have not been published to a collector and that allows @@ -34,6 +34,9 @@ Here is a more detailed list of what has been done: * Improvements have been made to the sniffer subsystem (thanks to Alejandro López (kudrom)) +* Fix the multi protocol traceroute test. + https://trac.torproject.org/projects/tor/ticket/12883 + Minor bug fixes: * Fix dns_spoof test (by kudrom) @@ -49,8 +52,8 @@ Minor bug fixes: * The README for ooni-probe should mention the bugtracker and repository https://trac.torproject.org/projects/tor/ticket/11980 -v1.0.2 -------- +v1.0.2 (Fri, 9 May 2014) +------------------------ * Add ooniprobe manpage. @@ -64,8 +67,8 @@ v1.0.2 * Have better test naming consistency. -v1.0.1 -------- +v1.0.1 (Fri, 14 Mar 2014) +------------------------- * Fix bugs in the traceroute test that lead to not all packets being collected. @@ -76,8 +79,8 @@ v1.0.1 * Add bridge_reachability test -v1.0.0 -------- +v1.0.0 (Thu, 20 Feb 2014) +------------------------- * Add bouncer support for discovering test helpers and collectors @@ -89,8 +92,8 @@ v1.0.0 * Add disclaimer when running ooniprobe -v0.1.0 -------- +v0.1.0 (Mon, 17 Jun 2013) +------------------------- Improvements to HTML/JS based user interface: @@ -100,8 +103,8 @@ Improvements to HTML/JS based user interface: Bugfixing and improvements to scheduler. -v0.0.12 -------- +v0.0.12 (Sat, 8 Jun 2013) +------------------------- Implement JS/HTML based user interface. @@ -111,8 +114,8 @@ Supports: * Monitoring of test progress -v0.0.11 -------- +v0.0.11 (Thu, 11 Apr 2013) +-------------------------- * Parametrize task timeout and retry count @@ -122,8 +125,8 @@ v0.0.11 * Add continuous integration with travis -v0.0.10 -------- +v0.0.10 (Wed, 26 Dec 2012) +-------------------------- ooniprobe: @@ -162,8 +165,8 @@ documentation: * Add instructions for supervisord configuration and clean up README.md -0.0.9 ------ +0.0.9 (Tue, 11 Dec 2012) +------------------------ ooniprobe: @@ -208,8 +211,8 @@ Tests: * fixup Netalyzr third party plugin -v0.0.8-alpha ------------- +v0.0.8-alpha (Sun, 2 Dec 2012) +------------------------------ ooniprobe: @@ -325,8 +328,8 @@ documentation: * Update documentation on how to setup ooniprobe. -v0.0.7.1-alpha --------------- +v0.0.7.1-alpha (Sun, 11 Nov 2012) +--------------------------------- * Add software version to the report @@ -334,15 +337,15 @@ v0.0.7.1-alpha * Improve HTTP Host test to work with the HTTP Requests test backend -v0.0.7-alpha -------------- +v0.0.7-alpha (Sat, 10 Nov 2012) +------------------------------- * Add test_name key to ooniprobe reports * Port TCP connect test to the new API -v0.0.4-alpha ------------- +v0.0.4-alpha (Sat, 10 Nov 2012) +------------------------------- * Add multi protocol multi port traceroute for UDP and TCP @@ -352,8 +355,8 @@ v0.0.4-alpha * Bugfixing and refactoring of txscapy for sending and receiving of scapy packets. -v0.0.3-alpha ------------- +v0.0.3-alpha (Fri, 9 Nov 2012) +------------------------------ * Implement logging to PCAP file support @@ -378,8 +381,7 @@ v0.0.3-alpha * Remove raw inputs and move them to a separate repository and add Makefile to fetch such lists -0.0.1-alpha ------------ +0.0.1-alpha (Tue, 6 Nov 2012) +----------------------------- First release of ooni-probe. woot! -

1 0

[ooni-probe/master] Update install instructions for debian to point to debian-backports and debian unstable repo.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 229695de65401bc9706b532b73d1ba0e6632057e Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 14:32:55 2014 +0200 Update install instructions for debian to point to debian-backports and debian unstable repo. --- README.rst | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index a8bad75..12d3fff 100644 --- a/README.rst +++ b/README.rst @@ -50,7 +50,18 @@ Installation Debian based systems -------------------- -Run this:: +If you are running debian testing or debian unstable you can install ooniprobe +simply with:: + + apt-get install ooniprobe + +If you are running debian stable you can get it from backports via:: + + sudo sh -c 'echo "deb http://http.debian.net/debian wheezy-backports main" >> /etc/apt/sources.list' + sudo apt-get update && sudo apt-get install ooniprobe + + +We also have our own debian repository. To install it from there run this:: sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list' gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4

1 0

[ooni-probe/master] Fix problem in starting Tor when no test helper is set.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit a0fe8cef28fe62f858e647188e169543f005bfea Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:03:17 2014 +0200 Fix problem in starting Tor when no test helper is set. --- ooni/oonibclient.py | 5 ++--- ooni/oonicli.py | 4 +++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/ooni/oonibclient.py b/ooni/oonibclient.py index 9e08c74..05436e0 100644 --- a/ooni/oonibclient.py +++ b/ooni/oonibclient.py @@ -30,7 +30,7 @@ class OONIBClient(object): raise e.InvalidOONIBBouncerAddress elif self.address.startswith('http://'): - log.msg("Warning using unencrypted collector") + log.msg("Warning using unencrypted backend") agent = Agent(reactor) attempts = 0 @@ -205,10 +205,9 @@ class OONIBClient(object): @defer.inlineCallbacks def lookupTestHelpers(self, test_helper_names): try: - test_helper = yield self.queryBackend('POST', '/bouncer', query={'test-helpers': test_helper_names}) - except Exception, exc: + except Exception as exc: log.exception(exc) raise e.CouldNotFindTestHelper diff --git a/ooni/oonicli.py b/ooni/oonicli.py index be242f5..f8cf849 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -168,10 +168,12 @@ def runWithDirector(logging=True, start_tor=True, check_incoherences=True): if global_options['no-collector']: log.msg("Not reporting using a collector") global_options['collector'] = None + start_tor = False + else: + start_tor = True deck = Deck(no_collector=global_options['no-collector']) deck.bouncer = global_options['bouncer'] - start_tor = False if global_options['collector']: start_tor |= True

1 0

[ooni-probe/master] Update LICENSE and expand setup.py
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 1883ede9a83e0006ef7eb9f0bbfeacb70659c0bc Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:38:25 2014 +0200 Update LICENSE and expand setup.py --- LICENSE | 4 ---- setup.py | 70 +++++++++++++++++++++++++++++++++++++++++++++++++++++++------- 2 files changed, 62 insertions(+), 12 deletions(-) diff --git a/LICENSE b/LICENSE index a45daeb..beefdd7 100644 --- a/LICENSE +++ b/LICENSE @@ -21,9 +21,5 @@ ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -The views and conclusions contained in the software and documentation are those -of the authors and should not be interpreted as representing official policies, -either expressed or implied, of the FreeBSD Project. - This product includes GeoLite data created by MaxMind, available from <a href="http://www.maxmind.com">http://www.maxmind.com</a>. diff --git a/setup.py b/setup.py index 8836220..e7fa493 100644 --- a/setup.py +++ b/setup.py @@ -69,6 +69,22 @@ for root, dirs, file_names in os.walk('data/'): files.append(pj(root, file_name)) data_files.append([pj(usr_share_path, 'ooni', root.replace('data/', '')), files]) + +packages=[ + 'ooni', + 'ooni.api', + 'ooni.kit', + 'ooni.nettests', + 'ooni.nettests.manipulation', + 'ooni.nettests.experimental', + 'ooni.nettests.scanning', + 'ooni.nettests.blocking', + 'ooni.nettests.third_party', + 'ooni.templates', + 'ooni.tests', + 'ooni.utils' +] + with open('requirements.txt') as f: for line in f: if line.startswith("#"): @@ -78,21 +94,59 @@ with open('requirements.txt') as f: continue install_requires.append(line) +with open('README.rst') as f: + readme = f.read() + +with open('Changelog.rst') as f: + changelog = f.read() + setup( name="ooniprobe", version=__version__, author=__author__, - author_email = "ooni-dev(a)torproject.org", + author_email="ooni-dev(a)torproject.org", + description="Network Interference detection tool.", + long_description=readme + '\n\n' + changelog, + license='BSD 2 clause', url="https://ooni.torproject.org/", package_dir={'ooni': 'ooni'}, data_files=data_files, - packages=['ooni', 'ooni.api', 'ooni.kit', - 'ooni.nettests', 'ooni.nettests.manipulation', - 'ooni.nettests.experimental', 'ooni.nettests.scanning', - 'ooni.nettests.blocking', - 'ooni.nettests.third_party', - 'ooni.templates', 'ooni.tests', 'ooni.utils'], + packages=packages, scripts=["bin/ooniprobe", "bin/oonireport"], dependency_links=dependency_links, - install_requires=install_requires + install_requires=install_requires, + classifiers=( + "Development Status :: 5 - Production/Stable", + "Environment :: Console", + "Framework :: Twisted", + "Intended Audience :: Developers", + "Intended Audience :: Education", + "Intended Audience :: End Users/Desktop", + "Intended Audience :: Information Technology", + "Intended Audience :: Science/Research", + "Intended Audience :: Telecommunications Industry", + "License :: OSI Approved :: BSD License" + "Programming Language :: Python", + "Programming Language :: Python :: 2", + "Programming Language :: Python :: 2 :: Only", + "Programming Language :: Python :: 2.6", + "Programming Language :: Python :: 2.7", + "Operating System :: MacOS :: MacOS X", + "Operating System :: POSIX", + "Operating System :: POSIX :: BSD", + "Operating System :: POSIX :: BSD :: BSD/OS", + "Operating System :: POSIX :: BSD :: FreeBSD", + "Operating System :: POSIX :: BSD :: NetBSD", + "Operating System :: POSIX :: BSD :: OpenBSD", + "Operating System :: POSIX :: Linux", + "Operating System :: Unix", + "Topic :: Scientific/Engineering :: Information Analysis", + "Topic :: Security", + "Topic :: Security :: Cryptography", + "Topic :: Software Development :: Libraries :: Application Frameworks", + "Topic :: Software Development :: Libraries :: Python Modules", + "Topic :: Software Development :: Testing", + "Topic :: Software Development :: Testing :: Traffic Generation", + "Topic :: System :: Networking :: Monitoring", + ) )

1 0

[ooni-probe/master] Convert README to rst
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 25be8147123fa0afdf6f363cb851c04c7680b698 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:38:17 2014 +0200 Convert README to rst --- MANIFEST.in | 2 +- README.md | 261 ----------------------------------------------------------- README.rst | 260 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 261 insertions(+), 262 deletions(-) diff --git a/MANIFEST.in b/MANIFEST.in index 2384d90..0772d8c 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,4 +1,4 @@ -include README.md ChangeLog.md requirements.txt LICENSE +include README.rst ChangeLog.md requirements.txt LICENSE recursive-include data/decks * include data/inputs/ include data/GeoIP.dat diff --git a/README.md b/README.md deleted file mode 100644 index ce7ba6e..0000000 --- a/README.md +++ /dev/null @@ -1,261 +0,0 @@ -[![Build Status](https://travis-ci.org/TheTorProject/ooni-probe.png?branch=master)](… -[![Coverage Status](https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png)](https://coveralls.io/r/TheTorProject/ooni-probe) - -![OONI](https://ooni.torproject.org/theme/img/ooni-logo.png) - -"The Net interprets censorship as damage and routes around it." - - John Gilmore; TIME magazine (6 December 1993) - -OONI, the Open Observatory of Network Interference, is a global observation -network which aims is to collect high quality data using open methodologies, -using Free and Open Source Software (FL/OSS) to share observations and data -about the various types, methods, and amounts of network tampering in the -world. - -ooniprobe is the first program that users run to probe their network and to -collect data for the OONI project. Are you interested in testing your network -for signs of surveillance and censorship? Do you want to collect data to share -with others, so that you and others may better understand your network? If so, -please read this document and we hope ooniprobe will help you to gather -network data that will assist you with your endeavors! - -Read this before running ooniprobe! ------------------------------------ - -Running ooniprobe is a potentially risky activity. This greatly depends on the -jurisdiction in which you are in and which test you are running. It is -technically possible for a person observing your internet connection to be -aware of the fact that you are running ooniprobe. This means that if running -network measurement tests is something considered to be illegal in your country -then you could be spotted. - -Futhermore, ooniprobe takes no precautions to protect the install target machine -from forensics analysis. If the fact that you have installed or used ooni -probe is a liability for you, please be aware of this risk. - -Installation -============ - -Debian based systems --------------------- - -Run this:: - - sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list' - gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4 - gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add - - sudo apt-get update && sudo apt-get install ooniprobe - -Linux ------ - -We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as -well as versions of Ubuntu such as natty and later releases. Running ooniprobe -without installing it is supported with the following commands:: - - git clone https://git.torproject.org/ooni-probe.git - cd ooni-probe - ./setup-dependencies.sh - python setup.py install - -Setting up development environment ----------------------------------- - -On debian based systems this can be done with:: - - sudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper - mkvirtualenv ooniprobe - python setup.py install - pip install -r requirements-dev.txt - -Other platforms (with Vagrant) ------------------------------- - -0) [Install Vagrant](https://www.vagrantup.com/downloads.html) and [Install Virtualbox](https://www.virtualbox.org/wiki/Downloads) - -0.1) - -On OSX: - -If you don't have it install homebrew http://mxcl.github.io/homebrew/:: - - brew install git - -On debian/ubuntu:: - - sudo apt-get install git - -1) Open a Terminal and run:: - - git clone https://git.torproject.org/ooni-probe.git - cd ooni-probe/ - vagrant up - -2) Login to the box with:: - - vagrant ssh - -ooniprobe will be installed in ``/ooni``. - -3) You can run tests with:: - - ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt - -Using ooniprobe -=============== - -**Net test** is a set of measurements to assess what kind of internet censorship is occurring. - -**Decks** are collections of ooniprobe nettests with some associated inputs. - -**Collector** is a service used to report the results of measurements. - -**Test helper** is a service used by a probe for successfully performing its measurements. - -**Bouncer** is a service used to discover the addresses of test helpers and collectors. - -### Configuring ooniprobe - -You may edit the configuration for ooniprobe by editing the configuration file -found inside of ``~/.ooni/ooniprobe.conf``. - -By default ooniprobe will not include personal identifying information in the -test result, nor create a pcap file. This behavior can be personalized. - -Running decks -------------- - -You will find all the installed decks inside of ``/usr/share/ooni/decks``. - -You may then run a deck by using the command line option ``-i``: - -As root:: - - ooniprobe -i /usr/share/ooni/decks/mlab.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck - - -Or: - -As root:: - - ooniprobe -i /usr/share/ooni/decks/complete.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck - - -The above tests will require around 20-30 minutes to complete depending on your network speed. - -If you would prefer to run some faster tests you should run: -As root:: - - ooniprobe -i /usr/share/ooni/decks/fast.deck - - -Or as a user:: - - ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck - - -Running net tests ------------------ - -You may list all the installed stable net tests with:: - - - ooniprobe -s - - -You may then run a nettest by specifying its name for example:: - - - ooniprobe manipulation/http_header_field_manipulation - - -It is also possible to specify inputs to tests as URLs:: - - - ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 - - -You can find the result of the test in your current working directory. - -By default the report result will be collected by the default ooni collector -and the addresses of test helpers will be obtained from the default bouncer. - -You may also specify your own collector or bouncer with the options ``-c`` and -``-b``. - - -Bridges and obfsproxy bridges -============================= - -ooniprobe submits reports to oonib report collectors through Tor to a hidden -service endpoint. By default, ooniprobe uses the installed system Tor, but can -also be configured to launch Tor (see the advanced.start_tor option in -ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if -obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path -to a file that should contain a set of "bridge" lines (of the same format as -used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy -bridges are to be used, the path to the obfsproxy binary must be configured. -See option advanced.obfsproxy_binary, in ooniprobe.conf. - -(Optional) Install obfsproxy ----------------------------- - -Install the latest version of obfsproxy for your platform. - -[Download Obfsproxy](https://www.torproject.org/projects/obfsproxy.html.en) - -Setting capabilities on your virtualenv python binary -===================================================== - -If your distributation supports capabilities you can avoid needing to run OONI as root:: - - - setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python - - - -Reporting bugs -============== - -You can report bugs and issues you find with ooni-probe on The Tor Projec issue -tracker [filing them under the "Ooni" component](https://trac.torproject.org/projects/tor/newticket?component=Oon…. - -You can either register an account or use the group account "cypherpunks" with -password "writecode". - -Contributing -============ - -You can download the code for ooniprobe from the following git repository:: - - - git clone https://git.torproject.org/ooni-probe.git - - -It is also viewable on the web via [https://gitweb.torproject.org/ooni-probe.git](https://git.torproject.org/oo…. - -You should then submit patches for review as pull requests to this github repository: - -https://github.com/TheTorProject/ooni-probe - -Read this article to learn how to [create a pull request on github](https://help.github.com/articles/creating-a-pull-request). - -If you prefer not to use github (or don't have an account), you may also submit -patches as attachments to tickets. - -Be sure to format the patch (given that you are working on a feature branch -that is different from master) with:: - - - git format-patch master --stdout > my_first_ooniprobe.patch - diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..d8fe11e --- /dev/null +++ b/README.rst @@ -0,0 +1,260 @@ +.. image:: https://travis-ci.org/TheTorProject/ooni-probe.png?branch=master + :target: https://travis-ci.org/TheTorProject/ooni-probe + +.. image:: https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png + :target: https://coveralls.io/r/TheTorProject/ooni-probe + + "The Net interprets censorship as damage and routes around it." + - John Gilmore; TIME magazine (6 December 1993) + +OONI, the Open Observatory of Network Interference, is a global observation +network which aims is to collect high quality data using open methodologies, +using Free and Open Source Software (FL/OSS) to share observations and data +about the various types, methods, and amounts of network tampering in the +world. + +ooniprobe is the first program that users run to probe their network and to +collect data for the OONI project. Are you interested in testing your network +for signs of surveillance and censorship? Do you want to collect data to share +with others, so that you and others may better understand your network? If so, +please read this document and we hope ooniprobe will help you to gather +network data that will assist you with your endeavors! + +Read this before running ooniprobe! +----------------------------------- + +Running ooniprobe is a potentially risky activity. This greatly depends on the +jurisdiction in which you are in and which test you are running. It is +technically possible for a person observing your internet connection to be +aware of the fact that you are running ooniprobe. This means that if running +network measurement tests is something considered to be illegal in your country +then you could be spotted. + +Futhermore, ooniprobe takes no precautions to protect the install target machine +from forensics analysis. If the fact that you have installed or used ooni +probe is a liability for you, please be aware of this risk. + +Installation +============ + +Debian based systems +-------------------- + +Run this:: + + sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list' + gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4 + gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add - + sudo apt-get update && sudo apt-get install ooniprobe + +Linux +----- + +We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as +well as versions of Ubuntu such as natty and later releases. Running ooniprobe +without installing it is supported with the following commands:: + + git clone https://git.torproject.org/ooni-probe.git + cd ooni-probe + ./setup-dependencies.sh + python setup.py install + +Setting up development environment +---------------------------------- + +On debian based systems this can be done with:: + + sudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper + mkvirtualenv ooniprobe + python setup.py install + pip install -r requirements-dev.txt + +Other platforms (with Vagrant) +------------------------------ + +1. [Install Vagrant](https://www.vagrantup.com/downloads.html) and [Install Virtualbox](https://www.virtualbox.org/wiki/Downloads) + +2. On OSX: + +If you don't have it install homebrew http://mxcl.github.io/homebrew/:: + + brew install git + +On debian/ubuntu:: + + sudo apt-get install git + +3. Open a Terminal and run:: + + git clone https://git.torproject.org/ooni-probe.git + cd ooni-probe/ + vagrant up + +4. Login to the box with:: + + vagrant ssh + +ooniprobe will be installed in ``/ooni``. + +5. You can run tests with:: + + ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt + +Using ooniprobe +=============== + +**Net test** is a set of measurements to assess what kind of internet censorship is occurring. + +**Decks** are collections of ooniprobe nettests with some associated inputs. + +**Collector** is a service used to report the results of measurements. + +**Test helper** is a service used by a probe for successfully performing its measurements. + +**Bouncer** is a service used to discover the addresses of test helpers and collectors. + +### Configuring ooniprobe + +You may edit the configuration for ooniprobe by editing the configuration file +found inside of ``~/.ooni/ooniprobe.conf``. + +By default ooniprobe will not include personal identifying information in the +test result, nor create a pcap file. This behavior can be personalized. + +Running decks +------------- + +You will find all the installed decks inside of ``/usr/share/ooni/decks``. + +You may then run a deck by using the command line option ``-i``: + +As root:: + + ooniprobe -i /usr/share/ooni/decks/mlab.deck + + +Or as a user:: + + ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck + + +Or: + +As root:: + + ooniprobe -i /usr/share/ooni/decks/complete.deck + + +Or as a user:: + + ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck + + +The above tests will require around 20-30 minutes to complete depending on your network speed. + +If you would prefer to run some faster tests you should run: +As root:: + + ooniprobe -i /usr/share/ooni/decks/fast.deck + + +Or as a user:: + + ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck + + +Running net tests +----------------- + +You may list all the installed stable net tests with:: + + + ooniprobe -s + + +You may then run a nettest by specifying its name for example:: + + + ooniprobe manipulation/http_header_field_manipulation + + +It is also possible to specify inputs to tests as URLs:: + + + ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 + + +You can find the result of the test in your current working directory. + +By default the report result will be collected by the default ooni collector +and the addresses of test helpers will be obtained from the default bouncer. + +You may also specify your own collector or bouncer with the options ``-c`` and +``-b``. + + +Bridges and obfsproxy bridges +============================= + +ooniprobe submits reports to oonib report collectors through Tor to a hidden +service endpoint. By default, ooniprobe uses the installed system Tor, but can +also be configured to launch Tor (see the advanced.start_tor option in +ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if +obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path +to a file that should contain a set of "bridge" lines (of the same format as +used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy +bridges are to be used, the path to the obfsproxy binary must be configured. +See option advanced.obfsproxy_binary, in ooniprobe.conf. + +(Optional) Install obfsproxy +---------------------------- + +Install the latest version of obfsproxy for your platform. + +[Download Obfsproxy](https://www.torproject.org/projects/obfsproxy.html.en) + +Setting capabilities on your virtualenv python binary +===================================================== + +If your distributation supports capabilities you can avoid needing to run OONI as root:: + + + setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python + + + +Reporting bugs +============== + +You can report bugs and issues you find with ooni-probe on The Tor Projec issue +tracker [filing them under the "Ooni" component](https://trac.torproject.org/projects/tor/newticket?component=Oon…. + +You can either register an account or use the group account "cypherpunks" with +password "writecode". + +Contributing +============ + +You can download the code for ooniprobe from the following git repository:: + + + git clone https://git.torproject.org/ooni-probe.git + + +It is also viewable on the web via [https://gitweb.torproject.org/ooni-probe.git](https://git.torproject.org/oo…. + +You should then submit patches for review as pull requests to this github repository: + +https://github.com/TheTorProject/ooni-probe + +Read this article to learn how to [create a pull request on github](https://help.github.com/articles/creating-a-pull-request). + +If you prefer not to use github (or don't have an account), you may also submit +patches as attachments to tickets. + +Be sure to format the patch (given that you are working on a feature branch +that is different from master) with:: + + + git format-patch master --stdout > my_first_ooniprobe.patch +

1 0

[ooni-probe/master] Reformat README.md so that it uses syntax common to markdown and rst.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 3c010b702f8ddfdcd95ff7b0cd9137376255b4f4 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:03:51 2014 +0200 Reformat README.md so that it uses syntax common to markdown and rst. --- README.md | 231 +++++++++++++++++++++++++++++++------------------------------ 1 file changed, 117 insertions(+), 114 deletions(-) diff --git a/README.md b/README.md index 28c895c..ce7ba6e 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,8 @@ with others, so that you and others may better understand your network? If so, please read this document and we hope ooniprobe will help you to gather network data that will assist you with your endeavors! -#### Read this before running ooniprobe! +Read this before running ooniprobe! +----------------------------------- Running ooniprobe is a potentially risky activity. This greatly depends on the jurisdiction in which you are in and which test you are running. It is @@ -32,41 +33,43 @@ Futhermore, ooniprobe takes no precautions to protect the install target machine from forensics analysis. If the fact that you have installed or used ooni probe is a liability for you, please be aware of this risk. -## Installation +Installation +============ -### Debian based systems +Debian based systems +-------------------- -``` -sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list' -gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4 -gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add - -sudo apt-get update && sudo apt-get install ooniprobe -``` +Run this:: -### Linux + sudo sh -c 'echo "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list' + gpg --keyserver pgp.mit.edu --recv-key 0x49B8CDF4 + gpg --export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt-key add - + sudo apt-get update && sudo apt-get install ooniprobe + +Linux +----- We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as well as versions of Ubuntu such as natty and later releases. Running ooniprobe -without installing it is supported with the following commands: +without installing it is supported with the following commands:: + + git clone https://git.torproject.org/ooni-probe.git + cd ooni-probe + ./setup-dependencies.sh + python setup.py install -``` -git clone https://git.torproject.org/ooni-probe.git -cd ooni-probe -./setup-dependencies.sh -python setup.py install -``` +Setting up development environment +---------------------------------- -### Setting up development environment +On debian based systems this can be done with:: -On debian based systems this can be done with: -``` -sudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper -mkvirtualenv ooniprobe -python setup.py install -pip install -r requirements-dev.txt -``` + sudo apt-get install libgeoip-dev python-virtualenv virtualenvwrapper + mkvirtualenv ooniprobe + python setup.py install + pip install -r requirements-dev.txt -### Other platforms (with Vagrant) +Other platforms (with Vagrant) +------------------------------ 0) [Install Vagrant](https://www.vagrantup.com/downloads.html) and [Install Virtualbox](https://www.virtualbox.org/wiki/Downloads) @@ -74,41 +77,32 @@ pip install -r requirements-dev.txt On OSX: -If you don't have it install homebrew http://mxcl.github.io/homebrew/ +If you don't have it install homebrew http://mxcl.github.io/homebrew/:: -``` -brew install git -``` + brew install git -On debian/ubuntu: +On debian/ubuntu:: -``` -sudo apt-get install git -``` + sudo apt-get install git -1) Open a Terminal and run: +1) Open a Terminal and run:: -``` -git clone https://git.torproject.org/ooni-probe.git -cd ooni-probe/ -vagrant up -``` + git clone https://git.torproject.org/ooni-probe.git + cd ooni-probe/ + vagrant up -2) Login to the box with: +2) Login to the box with:: -``` -vagrant ssh -``` + vagrant ssh -ooniprobe will be installed in `/ooni`. +ooniprobe will be installed in ``/ooni``. -3) You can run tests with: +3) You can run tests with:: -``` -ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt -``` + ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt -## Using ooniprobe +Using ooniprobe +=============== **Net test** is a set of measurements to assess what kind of internet censorship is occurring. @@ -123,87 +117,85 @@ ooniprobe blocking/http_requests -f /ooni/example_inputs/alexa-top-1k.txt ### Configuring ooniprobe You may edit the configuration for ooniprobe by editing the configuration file -found inside of `~/.ooni/ooniprobe.conf`. +found inside of ``~/.ooni/ooniprobe.conf``. By default ooniprobe will not include personal identifying information in the test result, nor create a pcap file. This behavior can be personalized. -### Running decks +Running decks +------------- + +You will find all the installed decks inside of ``/usr/share/ooni/decks``. + +You may then run a deck by using the command line option ``-i``: + +As root:: -You will find all the installed decks inside of `/usr/share/ooni/decks`. + ooniprobe -i /usr/share/ooni/decks/mlab.deck -You may then run a deck by using the command line option `-i`: -As root: -``` -ooniprobe -i /usr/share/ooni/decks/mlab.deck -``` +Or as a user:: + + ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck -Or as a user: -``` -ooniprobe -i /usr/share/ooni/decks/mlab_no_root.deck -``` Or: -As root: -``` -ooniprobe -i /usr/share/ooni/decks/complete.deck -``` +As root:: + + ooniprobe -i /usr/share/ooni/decks/complete.deck + + +Or as a user:: + + ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck -Or as a user: -``` -ooniprobe -i /usr/share/ooni/decks/complete_no_root.deck -``` The above tests will require around 20-30 minutes to complete depending on your network speed. If you would prefer to run some faster tests you should run: -As root: -``` -ooniprobe -i /usr/share/ooni/decks/fast.deck -``` +As root:: + + ooniprobe -i /usr/share/ooni/decks/fast.deck + + +Or as a user:: + + ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck + + +Running net tests +----------------- + +You may list all the installed stable net tests with:: + -Or as a user: -``` -ooniprobe -i /usr/share/ooni/decks/fast_no_root.deck -``` + ooniprobe -s -### Running net tests -You may list all the installed stable net tests with: +You may then run a nettest by specifying its name for example:: -``` -ooniprobe -s -``` -You may then run a nettest by specifying its name for example: + ooniprobe manipulation/http_header_field_manipulation -``` -ooniprobe manipulation/http_header_field_manipulation -``` -It is also possible to specify inputs to tests as URLs: +It is also possible to specify inputs to tests as URLs:: + + + ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 -``` -ooniprobe blocking/http_requests -f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1 -``` You can find the result of the test in your current working directory. By default the report result will be collected by the default ooni collector and the addresses of test helpers will be obtained from the default bouncer. -You may also specify your own collector or bouncer with the options `-c` and -`-b`. +You may also specify your own collector or bouncer with the options ``-c`` and +``-b``. -## (Optional) Install obfsproxy -Install the latest version of obfsproxy for your platform. - -[Download Obfsproxy](https://www.torproject.org/projects/obfsproxy.html.en) - -## Bridges and obfsproxy bridges +Bridges and obfsproxy bridges +============================= ooniprobe submits reports to oonib report collectors through Tor to a hidden service endpoint. By default, ooniprobe uses the installed system Tor, but can @@ -215,16 +207,25 @@ used in torrc, and as returned by https://bridges.torproject.org) If obfsproxy bridges are to be used, the path to the obfsproxy binary must be configured. See option advanced.obfsproxy_binary, in ooniprobe.conf. -## Setting capabilities on your virtualenv python binary +(Optional) Install obfsproxy +---------------------------- + +Install the latest version of obfsproxy for your platform. -If your distributation supports capabilities you can avoid needing to run OONI as root: +[Download Obfsproxy](https://www.torproject.org/projects/obfsproxy.html.en) -``` -setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python -``` +Setting capabilities on your virtualenv python binary +===================================================== +If your distributation supports capabilities you can avoid needing to run OONI as root:: -## Reporting bugs + + setcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv's/python + + + +Reporting bugs +============== You can report bugs and issues you find with ooni-probe on The Tor Projec issue tracker [filing them under the "Ooni" component](https://trac.torproject.org/projects/tor/newticket?component=Oon…. @@ -232,13 +233,14 @@ tracker [filing them under the "Ooni" component](https://trac.torproject.org/pro You can either register an account or use the group account "cypherpunks" with password "writecode". -## Contributing +Contributing +============ -You can download the code for ooniprobe from the following git repository: +You can download the code for ooniprobe from the following git repository:: + + + git clone https://git.torproject.org/ooni-probe.git -``` -git clone https://git.torproject.org/ooni-probe.git -``` It is also viewable on the web via [https://gitweb.torproject.org/ooni-probe.git](https://git.torproject.org/oo…. @@ -251,8 +253,9 @@ Read this article to learn how to [create a pull request on github](https://help If you prefer not to use github (or don't have an account), you may also submit patches as attachments to tickets. -Be sure to format the patch (given that you are working on a feature branch that is different from master) with: +Be sure to format the patch (given that you are working on a feature branch +that is different from master) with:: + + + git format-patch master --stdout > my_first_ooniprobe.patch -``` -git format-patch master --stdout > my_first_ooniprobe.patch -```

1 0

[ooni-probe/master] Convert changelog to rst.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 1e44942d3c848c4d6bb29bbd9f385d500555cc0f Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:43:06 2014 +0200 Convert changelog to rst. --- ChangeLog.md | 367 ------------------------------------------------------ ChangeLog.rst | 382 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 382 insertions(+), 367 deletions(-) diff --git a/ChangeLog.md b/ChangeLog.md deleted file mode 100644 index a349dff..0000000 --- a/ChangeLog.md +++ /dev/null @@ -1,367 +0,0 @@ -## v1.1.0 - -In this new release of ooniprobe we have added a new command line tool for -listing the reports that have not been published to a collector and that allows -the probe operator to choose which ones they would like to upload. - -We have also made some privacy improvements to the reports (we will sanitize -all things that may look like file paths) and added metadata associated with -the maxmind database being used by the probe operator. - -Here is a more detailed list of what has been done: - -* Annotate on disk which reports we have submitted and which ones we have not: - https://trac.torproject.org/projects/tor/ticket/11860 - -* Add tool called oonireport for publishing unpublished ooniprobe reports to a - collector: https://trac.torproject.org/projects/tor/ticket/11862 - -* Probe Report does not leak filepaths anymore: - https://trac.torproject.org/projects/tor/ticket/12706 - -* Reports now include version information about the maxmind database being - used: https://trac.torproject.org/projects/tor/ticket/12771 - -* We now do integrity checks on the ooniprobe.conf file so that we don't start - the tool if the config file is missing some settings or is not consistent: - https://trac.torproject.org/projects/tor/ticket/11983 - (thanks to Alejandro López (kudrom)) - -* Improvements have been made to the sniffer subsystem (thanks to Alejandro - López (kudrom)) - -Minor bug fixes: - -* Fix dns_spoof test (by kudrom) - https://trac.torproject.org/projects/tor/ticket/12486 - -* ooni might not look at requiresTor: - https://trac.torproject.org/projects/tor/ticket/11858 - -* ooni spits out gobs of tracebacks if Tor is not running and the OONI config - says it will be: - https://trac.torproject.org/projects/tor/ticket/11859 - -* The README for ooni-probe should mention the bugtracker and repository - https://trac.torproject.org/projects/tor/ticket/11980 - -## v1.0.2 - -* Add ooniprobe manpage. - -* Fix various security issues raised by the least authority audit. - -* Add a test that checks for Tor bridge reachability. - -* Record the IP address of the exit node being used in torified requests. - -* Captive portal test now uses the ooni-probe test templates. - -* Have better test naming consistency. - -## v1.0.1 - -* Fix bugs in the traceroute test that lead to not all packets being collected. - -* All values inside of http_requests test are now initialized inside of setUp. - -* Fix a bug that lead to the input value of the report not being set in some - circumstances. - -* Add bridge_reachability test - -## v1.0.0 - -* Add bouncer support for discovering test helpers and collectors - -* Fix bug that lead to HTTP tests to stall - -* Add support for connect_error and connection_lost_error error types - -* Add support for additional Tor configuration keys - -* Add disclaimer when running ooniprobe - -## v0.1.0 - -Improvements to HTML/JS based user interface: - - * XSRF protection - - * user supplied input specification - -Bugfixing and improvements to scheduler. - -## v0.0.12 - -Implement JS/HTML based user interface. - -Supports: - - * Starting and stopping of tests - - * Monitoring of test progress - -## v0.0.11 - -* Parametrize task timeout and retry count - -* Set the default collector via the command line option - -* Add option to disable the default collector - -* Add continuous integration with travis - -## v0.0.10 - -### ooniprobe - -* Fix bug that made HTTP based tests stall - -* Update DNS Test example to not import the DNS Test template If you import the - DNS Test template it will be considered a valid test case and command line - argument parsing will not work as expected. see: - #7795 for more details - -* Fix major bug in DNS test template that prevented PTR lookups from working - properly I was calling the queryUDP function with the arguments in the wrong - order. Twisted, why you API no consistent? - -* Add support for specifying the level of parallelism in tests (aka router - melt mode) - -* Do not swallow failures when a test instance fails to run fixes #7714 - -### scripts - -* Add report archival script - -Fix bug in TCP connect test that made it not properly log errors - -* Refactor failure handling code in nettest Add function that traps all the - supported failure and outputs the failure string representing it. - -### documentation - -* Add birdseye view of the ooniprobe architecture - -* Add details on the current implementation status of ooni* - -* Add draft ooniprobe API specification - -* Add instructions for supervisord configuration and clean up README.md - -## 0.0.9 - -### ooniprobe - -* Set the default ASN to 0 - -* Make Beautiful soup a soft depedency - -* Add support for sending the ASN number of the probe: - the ASN number will get sent when creating a new report - -* Add support for obtaining the probes IP address via getinfo address as per - https://trac.torproject.org/projects/tor/ticket/7447 - -* Fix bug in ooniprobe test decks - https://trac.torproject.org/projects/tor/ticket/7664 - -### oonib - -* Use twisted fdesc when writing to files - -* Add support for processing the ASN number of the probe - -* Test reports shall follow the specification detailed inside of docs/reports.rst - -* Add support for setting the tor binary path in oonib/config.py - -### scripts - -* Add a very simple example on how to securely parse the ooniprobe reports - -### documentation - -* Add documentation for the DNSSpoof test - -* Add documentation for HTTPHeaderFieldManipulation - -* Clean up writing_tests.rst - -* Properly use the power of sphinx! - -### Tests - -* fixup Netalyzr third party plugin - -## v0.0.8-alpha - -### ooniprobe - -* Allow test resolver file to have comments. - -* Autostart Tor in default configuration. - -* Add support for starting Tor via txtorcon. - -* Make the sniffer not run in a separate thread, but use a non blocking fdesc. - Do some refactoring of scapy testing, following Factory creational pattern - and a pub-sub pattern for the readers and writers. - -* Extend TrueHeaders to support calculation of difference between two HTTP headers respectful of - capitalization - -* Implement test deck system for automating the specification of command line - arguments for tests - -* Implement sr1 in txscapy - -* Include socksproxy address in HTTP based tests - -* Include the resolver IP:Port in the report - -* Changes to the report format of HTTP Test template derived tests: - Requests are now stored inside of an array to allow - the storing of multiple request/response pairs. - -* Fix bug that lead to httpt based reports to not have the url attribute set - properly. - -* twisted Headers() class edited to avoid header fix in reference to: - https://trac.torproject.org/projects/tor/ticket/7432 - -* Parametrize tor socksport for usage with modified HTTP Agent - -* Update URL List test to take as input also a single URL - -* Clean up filenames of reports generated by ooni-probe: - they now follow the format $testName_report_$timestamp.yamloo - -* Add ooniprobe prefix to logs - -* Respect the includeip = false option in ooniprobe.conf for scapyt derivate - tests: - If the option to not include the IP address of the probe is set, - change the source and destination ip address of the sent and received - packets to 127.0.0.1. - -#### Tests - -* Implement basic keyword filtering detection test. - -* Add ICMP support to multi protocol traceroute test - -* parametrize max_ttl and timeout - -* make max_ttl and timeout be included in the report - -* Port UK Mobile Network test to new API - -* Port daphn3 test - -* Randomize source port by default in traceroute test and include source port in - report - -* Test and Implement HTTP Header Field Manipulation Test (rename it to what we - had originally called it since it made most sense) - -* Implement test that detects DNS spoofing - -* Implement TCP payload sending test template: - Example test based on this test template - -* Make report IDs include the timestamp of the report - -* Add test that detects censorship in HTTP pages based on HTTP body length - -* Add socks proxy support to HTTP Test - -* Create DNS Test template: - Use such template for DNS Tamper test. - Add example usage of DNS Test Template. - -* Refactor captive portal test to run tests in threads - -### oonib - -* Implement basic collector for ooniprobe reports. - Reports can be submitted over the network via http to a remote collector. - Implement the backend component of the collector that writes submitted - reports to flat files, following the report_id naming convention. - -* Implement very simple HTTP Request backend that does only the part of HTTP we - need for testing - -* Make oonib a daemon - -* Loosen up the oonib regexp to support the timestamp report format - -* Add Tor Hidden Service support - -* Make the reporting directory of the collector configurable - -* Implement TCP Echo test helper. - -### scripts - -* Add fabfile for automatic deployment of ooni-probe to remote sites - -### documentation - -* Update documentation on how to setup ooniprobe. - -## v0.0.7.1-alpha - -* Add software version to the report - -* Implement basic oonib reporting to flat files containing the report ID. - -* Improve HTTP Host test to work with the HTTP Requests test backend - -## v0.0.7-alpha - -* Add test_name key to ooniprobe reports - -* Port TCP connect test to the new API - -## v0.0.4-alpha - -* Add multi protocol multi port traceroute for UDP and TCP - -* Implement basic HTTP request test that does capitalization variations on the -HTTP method. - -* Bugfixing and refactoring of txscapy for sending and receiving of scapy -packets. - -## v0.0.3-alpha - -* Implement logging to PCAP file support - -* Remove dependency on trial - -* Port china trigger to new API - -* Rename keyword filtering test to HTTP keyword filtering - -* Refactor install documentation. - -* Convert header of ooniprobe script to a non docstring - -* Add Makefile to fetch Maxmind geoip database files - -* Implement GeoIP lookup support - -* From configuration options it is possible to choice what level of privacy - the prober is willing to accept. Implement config file support You are able - to specify basic and advanced options in YAML format - -* Remove raw inputs and move them to a separate repository and add Makefile to - fetch such lists - -## 0.0.1-alpha - -First release of ooni-probe. woot! - diff --git a/ChangeLog.rst b/ChangeLog.rst new file mode 100644 index 0000000..e41924c --- /dev/null +++ b/ChangeLog.rst @@ -0,0 +1,382 @@ +v1.1.0 +------ + +In this new release of ooniprobe we have added a new command line tool for +listing the reports that have not been published to a collector and that allows +the probe operator to choose which ones they would like to upload. + +We have also made some privacy improvements to the reports (we will sanitize +all things that may look like file paths) and added metadata associated with +the maxmind database being used by the probe operator. + +Here is a more detailed list of what has been done: + +* Annotate on disk which reports we have submitted and which ones we have not: + https://trac.torproject.org/projects/tor/ticket/11860 + +* Add tool called oonireport for publishing unpublished ooniprobe reports to a + collector: https://trac.torproject.org/projects/tor/ticket/11862 + +* Probe Report does not leak filepaths anymore: + https://trac.torproject.org/projects/tor/ticket/12706 + +* Reports now include version information about the maxmind database being + used: https://trac.torproject.org/projects/tor/ticket/12771 + +* We now do integrity checks on the ooniprobe.conf file so that we don't start + the tool if the config file is missing some settings or is not consistent: + https://trac.torproject.org/projects/tor/ticket/11983 + (thanks to Alejandro López (kudrom)) + +* Improvements have been made to the sniffer subsystem (thanks to Alejandro + López (kudrom)) + +Minor bug fixes: + +* Fix dns_spoof test (by kudrom) + https://trac.torproject.org/projects/tor/ticket/12486 + +* ooni might not look at requiresTor: + https://trac.torproject.org/projects/tor/ticket/11858 + +* ooni spits out gobs of tracebacks if Tor is not running and the OONI config + says it will be: + https://trac.torproject.org/projects/tor/ticket/11859 + +* The README for ooni-probe should mention the bugtracker and repository + https://trac.torproject.org/projects/tor/ticket/11980 + +v1.0.2 +------- + +* Add ooniprobe manpage. + +* Fix various security issues raised by the least authority audit. + +* Add a test that checks for Tor bridge reachability. + +* Record the IP address of the exit node being used in torified requests. + +* Captive portal test now uses the ooni-probe test templates. + +* Have better test naming consistency. + +v1.0.1 +------- + +* Fix bugs in the traceroute test that lead to not all packets being collected. + +* All values inside of http_requests test are now initialized inside of setUp. + +* Fix a bug that lead to the input value of the report not being set in some + circumstances. + +* Add bridge_reachability test + +v1.0.0 +------- + +* Add bouncer support for discovering test helpers and collectors + +* Fix bug that lead to HTTP tests to stall + +* Add support for connect_error and connection_lost_error error types + +* Add support for additional Tor configuration keys + +* Add disclaimer when running ooniprobe + +v0.1.0 +------- + +Improvements to HTML/JS based user interface: + + * XSRF protection + + * user supplied input specification + +Bugfixing and improvements to scheduler. + +v0.0.12 +------- + +Implement JS/HTML based user interface. + +Supports: + + * Starting and stopping of tests + + * Monitoring of test progress + +v0.0.11 +------- + +* Parametrize task timeout and retry count + +* Set the default collector via the command line option + +* Add option to disable the default collector + +* Add continuous integration with travis + +v0.0.10 +------- + +ooniprobe: + +* Fix bug that made HTTP based tests stall + +* Update DNS Test example to not import the DNS Test template If you import the + DNS Test template it will be considered a valid test case and command line + argument parsing will not work as expected. see: + #7795 for more details + +* Fix major bug in DNS test template that prevented PTR lookups from working + properly I was calling the queryUDP function with the arguments in the wrong + order. Twisted, why you API no consistent? + +* Add support for specifying the level of parallelism in tests (aka router + melt mode) + +* Do not swallow failures when a test instance fails to run fixes #7714 + +scripts: + +* Add report archival script + +Fix bug in TCP connect test that made it not properly log errors + +* Refactor failure handling code in nettest Add function that traps all the + supported failure and outputs the failure string representing it. + +documentation: + +* Add birdseye view of the ooniprobe architecture + +* Add details on the current implementation status of ooni* + +* Add draft ooniprobe API specification + +* Add instructions for supervisord configuration and clean up README.md + +0.0.9 +----- + +ooniprobe: + +* Set the default ASN to 0 + +* Make Beautiful soup a soft depedency + +* Add support for sending the ASN number of the probe: + the ASN number will get sent when creating a new report + +* Add support for obtaining the probes IP address via getinfo address as per + https://trac.torproject.org/projects/tor/ticket/7447 + +* Fix bug in ooniprobe test decks + https://trac.torproject.org/projects/tor/ticket/7664 + +oonib: + +* Use twisted fdesc when writing to files + +* Add support for processing the ASN number of the probe + +* Test reports shall follow the specification detailed inside of docs/reports.rst + +* Add support for setting the tor binary path in oonib/config.py + +scripts: + +* Add a very simple example on how to securely parse the ooniprobe reports + +documentation: + +* Add documentation for the DNSSpoof test + +* Add documentation for HTTPHeaderFieldManipulation + +* Clean up writing_tests.rst + +* Properly use the power of sphinx! + +Tests: + +* fixup Netalyzr third party plugin + +v0.0.8-alpha +------------ + +ooniprobe: + +* Allow test resolver file to have comments. + +* Autostart Tor in default configuration. + +* Add support for starting Tor via txtorcon. + +* Make the sniffer not run in a separate thread, but use a non blocking fdesc. + Do some refactoring of scapy testing, following Factory creational pattern + and a pub-sub pattern for the readers and writers. + +* Extend TrueHeaders to support calculation of difference between two HTTP headers respectful of + capitalization + +* Implement test deck system for automating the specification of command line + arguments for tests + +* Implement sr1 in txscapy + +* Include socksproxy address in HTTP based tests + +* Include the resolver IP:Port in the report + +* Changes to the report format of HTTP Test template derived tests: + Requests are now stored inside of an array to allow + the storing of multiple request/response pairs. + +* Fix bug that lead to httpt based reports to not have the url attribute set + properly. + +* twisted Headers() class edited to avoid header fix in reference to: + https://trac.torproject.org/projects/tor/ticket/7432 + +* Parametrize tor socksport for usage with modified HTTP Agent + +* Update URL List test to take as input also a single URL + +* Clean up filenames of reports generated by ooni-probe: + they now follow the format $testName_report_$timestamp.yamloo + +* Add ooniprobe prefix to logs + +* Respect the includeip = false option in ooniprobe.conf for scapyt derivate + tests: + If the option to not include the IP address of the probe is set, + change the source and destination ip address of the sent and received + packets to 127.0.0.1. + +tests: + +* Implement basic keyword filtering detection test. + +* Add ICMP support to multi protocol traceroute test + +* parametrize max_ttl and timeout + +* make max_ttl and timeout be included in the report + +* Port UK Mobile Network test to new API + +* Port daphn3 test + +* Randomize source port by default in traceroute test and include source port in + report + +* Test and Implement HTTP Header Field Manipulation Test (rename it to what we + had originally called it since it made most sense) + +* Implement test that detects DNS spoofing + +* Implement TCP payload sending test template: + Example test based on this test template + +* Make report IDs include the timestamp of the report + +* Add test that detects censorship in HTTP pages based on HTTP body length + +* Add socks proxy support to HTTP Test + +* Create DNS Test template: + Use such template for DNS Tamper test. + Add example usage of DNS Test Template. + +* Refactor captive portal test to run tests in threads + +oonib: + +* Implement basic collector for ooniprobe reports. + Reports can be submitted over the network via http to a remote collector. + Implement the backend component of the collector that writes submitted + reports to flat files, following the report_id naming convention. + +* Implement very simple HTTP Request backend that does only the part of HTTP we + need for testing + +* Make oonib a daemon + +* Loosen up the oonib regexp to support the timestamp report format + +* Add Tor Hidden Service support + +* Make the reporting directory of the collector configurable + +* Implement TCP Echo test helper. + +scripts: + +* Add fabfile for automatic deployment of ooni-probe to remote sites + +documentation: + +* Update documentation on how to setup ooniprobe. + +v0.0.7.1-alpha +-------------- + +* Add software version to the report + +* Implement basic oonib reporting to flat files containing the report ID. + +* Improve HTTP Host test to work with the HTTP Requests test backend + +v0.0.7-alpha +------------- + +* Add test_name key to ooniprobe reports + +* Port TCP connect test to the new API + +v0.0.4-alpha +------------ + +* Add multi protocol multi port traceroute for UDP and TCP + +* Implement basic HTTP request test that does capitalization variations on the +HTTP method. + +* Bugfixing and refactoring of txscapy for sending and receiving of scapy +packets. + +v0.0.3-alpha +------------ + +* Implement logging to PCAP file support + +* Remove dependency on trial + +* Port china trigger to new API + +* Rename keyword filtering test to HTTP keyword filtering + +* Refactor install documentation. + +* Convert header of ooniprobe script to a non docstring + +* Add Makefile to fetch Maxmind geoip database files + +* Implement GeoIP lookup support + +* From configuration options it is possible to choice what level of privacy + the prober is willing to accept. Implement config file support You are able + to specify basic and advanced options in YAML format + +* Remove raw inputs and move them to a separate repository and add Makefile to + fetch such lists + +0.0.1-alpha +----------- + +First release of ooni-probe. woot! +

1 0

[ooni-probe/master] Update MANIFEST
by art＠torproject.org 19 Aug '14

19 Aug '14

commit 34b68d87f746c9b57afb4ba219ba66a66979c7f8 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:43:26 2014 +0200 Update MANIFEST --- MANIFEST.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/MANIFEST.in b/MANIFEST.in index 0772d8c..c40e0d5 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,4 +1,4 @@ -include README.rst ChangeLog.md requirements.txt LICENSE +include README.rst ChangeLog.rst requirements.txt LICENSE recursive-include data/decks * include data/inputs/ include data/GeoIP.dat

1 0

[ooni-probe/master] Improve REAMDE formatting.
by art＠torproject.org 19 Aug '14

19 Aug '14

commit d79e3d28a4cf2c55735a52940fd5879f48e63ef3 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Aug 19 11:54:01 2014 +0200 Improve REAMDE formatting. --- README.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 654acc3..b1977b1 100644 --- a/README.rst +++ b/README.rst @@ -7,6 +7,8 @@ ooniprobe: a network interference detection tool .. image:: https://coveralls.io/repos/TheTorProject/ooni-probe/badge.png :target: https://coveralls.io/r/TheTorProject/ooni-probe +___________________________________________________________________________ + .. image:: https://ooni.torproject.org/theme/img/ooni-logo.png :target: https:://ooni.torproject.org/ @@ -16,8 +18,9 @@ using Free and Open Source Software (FL/OSS) to share observations and data about the various types, methods, and amounts of network tampering in the world. + "The Net interprets censorship as damage and routes around it." - - John Gilmore; TIME magazine (6 December 1993) + - John Gilmore; TIME magazine (6 December 1993) ooniprobe is the first program that users run to probe their network and to

1 0