June 2013 - tor-commits - lists.torproject.org

[ooni-probe/develop] Implement Vagrant file for running ooniprobe on any platform
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 5782f32621369af476657ab128d1f63ad9b8e06a Author: Arturo Filastò <art(a)fuffa.org> Date: Sun May 5 17:28:31 2013 +0200 Implement Vagrant file for running ooniprobe on any platform * Update setup.py and README.md to work with Vagrant * Update inputs makefile to fetch from ooni.tpo/inputs --- README.md | 33 ++++++++---- Vagrantfile | 153 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ inputs/Makefile | 3 +- setup.py | 4 +- 4 files changed, 180 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 75095ad..eb9ab68 100644 --- a/README.md +++ b/README.md @@ -16,15 +16,30 @@ with others, so that you and others may better understand your network? If so, please read this document and we hope ooniprobe will help you to gather network data that will assist you with your endeavors! -## Getting started with ooniprobe is easy - - 0) Open your favorite terminal - - 1) Grab the source and install the dependencies - - 2) Run ooniprobe! - - 3) ... There is no step three ... +## Getting started with ooniprobe is easy (with Vagrant) + +0) [Install Vagrant](http://downloads.vagrantup.com/) + +1) Open a Terminal and run: + +``` +vagrant up +``` + +2) Login to the box with: + +``` +vagrant ssh +``` + +ooniprobe will be installed in `/data/ooniprobe`. + +3) You can run tests with: + +``` +cd /data/ooniprobe/ +./bin/ooniprobe nettests/blocking/http_requests.py -f /data/ooniprobe/inputs/input-pack/alexa-top-1k.txt +``` ## The easy way to prep your system for running ooniprobe diff --git a/Vagrantfile b/Vagrantfile new file mode 100644 index 0000000..52107e8 --- /dev/null +++ b/Vagrantfile @@ -0,0 +1,153 @@ +# -*- mode: ruby -*- +# vi: set ft=ruby : + +Vagrant.configure("2") do |config| + # All Vagrant configuration is done here. The most common configuration + # options are documented and commented below. For a complete reference, + # please see the online documentation at vagrantup.com. + + # Every Vagrant virtual environment requires a box to build off of. + config.vm.box = "precise32" + + # The url from where the 'config.vm.box' box will be fetched if it + # doesn't already exist on the user's system. + config.vm.box_url = "http://files.vagrantup.com/precise32.box" + + # Create a forwarded port mapping which allows access to a specific port + # within the machine from a port on the host machine. In the example below, + # accessing "localhost:8080" will access port 80 on the guest machine. + # config.vm.network :forwarded_port, guest: 80, host: 8080 + + # Create a private network, which allows host-only access to the machine + # using a specific IP. + # config.vm.network :private_network, ip: "192.168.33.10" + + # Create a public network, which generally matched to bridged network. + # Bridged networks make the machine appear as another physical device on + # your network. + # config.vm.network :public_network + + # Share an additional folder to the guest VM. The first argument is + # the path on the host to the actual folder. The second argument is + # the path on the guest to mount the folder. And the optional third + # argument is a set of non-required options. + config.vm.synced_folder ".", "/data/ooniprobe" + + # Provider-specific configuration so you can fine-tune various + # backing providers for Vagrant. These expose provider-specific options. + # Example for VirtualBox: + # + # config.vm.provider :virtualbox do |vb| + # # Don't boot with headless mode + # vb.gui = true + # + # # Use VBoxManage to customize the VM. For example to change memory: + # vb.customize ["modifyvm", :id, "--memory", "1024"] + # end + # + # View the documentation for the provider you're using for more + # information on available options. + + # Enable provisioning with Puppet stand alone. Puppet manifests + # are contained in a directory path relative to this Vagrantfile. + # You will need to create the manifests directory and a manifest in + # the file base.pp in the manifests_path directory. + # + # An example Puppet manifest to provision the message of the day: + # + # # group { "puppet": + # # ensure => "present", + # # } + # # + # # File { owner => 0, group => 0, mode => 0644 } + # # + # # file { '/etc/motd': + # # content => "Welcome to your Vagrant-built virtual machine! + # # Managed by Puppet.\n" + # # } + # + # config.vm.provision :puppet do |puppet| + # puppet.manifests_path = "manifests" + # puppet.manifest_file = "init.pp" + # end + + # Enable provisioning with chef solo, specifying a cookbooks path, roles + # path, and data_bags path (all relative to this Vagrantfile), and adding + # some recipes and/or roles. + # + # config.vm.provision :chef_solo do |chef| + # chef.cookbooks_path = "../my-recipes/cookbooks" + # chef.roles_path = "../my-recipes/roles" + # chef.data_bags_path = "../my-recipes/data_bags" + # chef.add_recipe "mysql" + # chef.add_role "web" + # + # # You may also specify custom JSON attributes: + # chef.json = { :mysql_password => "foo" } + # end + + # Enable provisioning with chef server, specifying the chef server URL, + # and the path to the validation key (relative to this Vagrantfile). + # + # The Opscode Platform uses HTTPS. Substitute your organization for + # ORGNAME in the URL and validation key. + # + # If you have your own Chef Server, use the appropriate URL, which may be + # HTTP instead of HTTPS depending on your configuration. Also change the + # validation key to validation.pem. + # + # config.vm.provision :chef_client do |chef| + # chef.chef_server_url = "https://api.opscode.com/organizations/ORGNAME" + # chef.validation_key_path = "ORGNAME-validator.pem" + # end + # + # If you're using the Opscode platform, your validator client is + # ORGNAME-validator, replacing ORGNAME with your organization name. + # + # If you have your own Chef Server, the default validation client name is + # chef-validator, unless you changed the configuration. + # + # chef.validation_client_name = "ORGNAME-validator" +end + +$script = <<SCRIPT +apt-get -y install curl python-setuptools python-dev + +echo "Updating to the latest version of PIP" +cd /tmp/ + +curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py +python ./get-pip.py ## pip (>=1.3.0) is recommended for security reasons + +sudo update-alternatives --install /usr/bin/pip pip /usr/local/bin/pip 0 + +echo "Installing Tor..." + +echo "deb http://deb.torproject.org/torproject.org precise main" >> /etc/apt/source.list + +gpg --keyserver keys.gnupg.net --recv 886DDD89 +gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | apt-key add - + +apt-get update +apt-get -y install deb.torproject.org-keyring tor tor-geoipdb + +apt-get -y install git-core python python-pip python-dev build-essential libdumbnet1 python-dumbnet python-libpcap python-pypcap python-dnspython python-virtualenv virtualenvwrapper tor tor-geoipdb libpcap-dev + +cd /data/ooniprobe + +echo "Installing dependencies" +pip install pyrex +pip install -r requirements.txt + +echo "Installing ooniprobe" +python setup.py install + +echo "Fetching all inputs" +cd /data/ooniprobe/inputs +make lists + +SCRIPT + +Vagrant.configure("2") do |config| + config.vm.provision :shell, :inline => $script +end diff --git a/inputs/Makefile b/inputs/Makefile index 42ae1f6..015a991 100644 --- a/inputs/Makefile +++ b/inputs/Makefile @@ -2,7 +2,8 @@ all: whatheaders lists lists: - git clone https://github.com/hellais/ooni-inputs.git ooni-inputs + curl -O https://ooni.torproject.org/inputs/input-pack.tar.gz + tar xzf input-pack.tar.gz whatheaders: wget http://s3.amazonaws.com/data.whatheaders.com/whatheaders-latest.xml.zip diff --git a/setup.py b/setup.py index 41b9050..e336e08 100644 --- a/setup.py +++ b/setup.py @@ -8,12 +8,10 @@ install_requires = [ 'scapy>=2.2.0', 'dnspython>=1.10.0', 'parsley>1.0', - 'pypcap>=1.1.1' ] dependency_links = [ - 'https://people.torproject.org/~ioerror/src/mirrors/ooniprobe', - 'https://github.com/hellais/pypcap/archive/v1.1.1.tar.gz#egg=pypcap-1.1.1' + 'https://people.torproject.org/~ioerror/src/mirrors/ooniprobe' ] with open('requirements.txt') as f:

1 0

[ooni-probe/develop] Include by default the details of the probe
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 2523773e5825069e602578353b565d1ec23950c3 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue May 7 02:51:37 2013 +0200 Include by default the details of the probe --- ooniprobe.conf.sample | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/ooniprobe.conf.sample b/ooniprobe.conf.sample index 53e7ac7..bed9da6 100644 --- a/ooniprobe.conf.sample +++ b/ooniprobe.conf.sample @@ -7,13 +7,13 @@ basic: logfile: ooniprobe.log privacy: # Should we include the IP address of the probe in the report? - includeip: false + includeip: true # Should we include the ASN of the probe in the report? - includeasn: false + includeasn: true # Should we include the country as reported by GeoIP in the report? - includecountry: false + includecountry: true # Should we include the city as reported by GeoIP in the report? - includecity: false + includecity: true # Should we collect a full packet capture on the client? includepcap: false reports:

1 0

[ooni-probe/develop] Add extra commands to step 1
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit a2a563eec5be238b165e01cbabef91b14c021426 Author: Arturo Filastò <hellais(a)gmail.com> Date: Sun May 5 18:55:11 2013 +0300 Add extra commands to step 1 --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index eb9ab68..fe89f43 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,8 @@ network data that will assist you with your endeavors! 1) Open a Terminal and run: ``` +git clone https://git.torproject.org/ooni-probe.git +cd ooni-probe/ vagrant up ```

1 0

[ooni-probe/develop] Have some slightly saner defaults for the config file
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 982cc75d654414471bd7a4e38ceb14b572442354 Author: Arturo Filastò <art(a)fuffa.org> Date: Sun May 5 16:30:34 2013 +0200 Have some slightly saner defaults for the config file --- ooni/director.py | 11 ++++++++--- ooniprobe.conf.sample | 6 +++--- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/ooni/director.py b/ooni/director.py index 7ad99a9..c796d6e 100644 --- a/ooni/director.py +++ b/ooni/director.py @@ -298,9 +298,14 @@ class Director(object): log.debug("Setting control port as %s" % tor_config.ControlPort) log.debug("Setting SOCKS port as %s" % tor_config.SocksPort) - d = launch_tor(tor_config, reactor, - tor_binary=config.advanced.tor_binary, - progress_updates=updates) + if config.advanced.tor_binary: + d = launch_tor(tor_config, reactor, + tor_binary=config.advanced.tor_binary, + progress_updates=updates) + else: + d = launch_tor(tor_config, reactor, + progress_updates=updates) + d.addCallback(setup_complete) d.addErrback(setup_failed) return d diff --git a/ooniprobe.conf.sample b/ooniprobe.conf.sample index 27a4fb3..53e7ac7 100644 --- a/ooniprobe.conf.sample +++ b/ooniprobe.conf.sample @@ -25,14 +25,14 @@ advanced: # /path/to/ooni-probe/data/ geoip_data_dir: /usr/share/GeoIP/ debug: true - tor_binary: '/usr/sbin/tor' + # tor_binary: '/usr/sbin/tor' # For auto detection interface: auto # Of specify a specific interface #interface: wlan0 # If you do not specify start_tor, you will have to have Tor running and # explicitly set the control port and SOCKS port - start_tor: true + start_tor: false # After how many seconds we should give up on a particular measurement measurement_timeout: 30 # After how many retries we should give up on a measurement @@ -46,7 +46,7 @@ advanced: # How many reports to perform concurrently reporting_concurrency: 20 tor: - #socks_port: 9050 + socks_port: 9050 #control_port: 9051 # Specify the absolute path to the Tor bridges to use for testing bridges: bridges.list

1 0

[ooni-probe/develop] Update requirements.txt to what twisted wants.
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 87c8f7b9ea43b5cee6ff93d629ae344eaeca1cb0 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed May 8 16:49:04 2013 +0200 Update requirements.txt to what twisted wants. --- requirements.txt | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/requirements.txt b/requirements.txt index bdf4cd0..f48eb88 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,13 +6,16 @@ docutils>=0.9.1 ipaddr>=2.1.10 pyOpenSSL>=0.13 pygeoip>=0.2.5 -transaction>=1.3.0 txtorcon>=0.7 txsocksx>=0.0.2 -zope.component>=4.0.0 -zope.event>=4.0.0 -zope.interface>=4.0.1 Pyrex + +# Taken from the versions required by twisted 13.0.0 (latest) +transaction>=1.1.1 +zope.component>=3.5 +zope.event>=3.5 +zope.interface>=3.6 + # # This is a Tor Project mirror with valid SSL/TLS certs that is stable and fast #

1 0

[ooni-probe/develop] Add certain deferreds into their own function so that we can do a
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 15e8e11cb9da46b78a6089f478d28a021c798a81 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 04:39:49 2013 +0000 Add certain deferreds into their own function so that we can do a defer.Deferred.addCallbacks on its return value. --- nettests/experimental/tls_handshake.py | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 0bd84bc..a21b357 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -589,8 +589,15 @@ class TLSHandshakeTest(nettest.NetTestCase): ## xxx do we need this? #return connection + @defer.inlineCallbacks + def deferMakeConnection(host): + connection = yield makeConnection(host) + if isinstance(connection, Failure) \ + or isinstance(connection, Exception): + failed = connectionFailed(connection, host) + defer.returnValue(failed) else: - return handshakeSucceeded(connection) + defer.returnValue(connection) addr, port = self.input connection = defer.maybeDeferred(makeConnection, addr, port)

1 0

[ooni-probe/develop] Add chained callbacks to handshake checking methods, and remove some unused
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 1fa06bffa93632fb031ff690e69f50b0c2f65436 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 04:41:01 2013 +0000 Add chained callbacks to handshake checking methods, and remove some unused code from the end of the file. --- nettests/experimental/tls_handshake.py | 33 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 19 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index a21b357..d990a5b 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -599,24 +599,19 @@ class TLSHandshakeTest(nettest.NetTestCase): else: defer.returnValue(connection) - addr, port = self.input - connection = defer.maybeDeferred(makeConnection, addr, port) - connection.addCallback(doHandshake) - connection.addErrback(log.err) - connection.addCallback(handshakeSucceeded) - connection.addErrback(handshakeFailed) + connection = deferMakeConnection(self.input) + connection.addCallbacks(connectionSucceeded, connectionFailed, + callbackArgs=[self.input, self.timeout], + errbackArgs=[self.input]) - return connection + handshake = defer.Deferred() + handshake.addCallback(doHandshake) + handshake.addCallbacks(handshakeSucceeded, handshakeFailed, + errbackArgs=[self.input]) -## XXX clean me up -## old function from anonymous contribution: (saved mostly for reference of the -## success string) -## -#def checkBridgeConnection(host, port) -# cipher_arg = ":".join(ciphers) -# cmd = ["openssl", "s_client", "-connect", "%s:%s" % (host,port)] -# cmd += ["-cipher", cipher_arg] -# proc = subprocess.Popen(cmd, stdout=PIPE, stderr=PIPE,stdin=PIPE) -# out, error = proc.communicate() -# success = "Cipher is DHE-RSA-AES256-SHA" in out -# return success + connection.chainDeferred(handshake) + connection.addCallbacks(connectionShutdown, defer.passthru, + callbackArgs=[self.input]) + connection.addBoth(log.exception) + + return connection

1 0

[ooni-probe/develop] Fix #8358. Add correct formatting and parsing of OpenSSL.crypto.X509Name
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 2a474c23522bc3accc76754dd4f5f7c3a7f4f4a8 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 13:08:05 2013 +0000 Fix #8358. Add correct formatting and parsing of OpenSSL.crypto.X509Name objects and their fields to reports. --- nettests/experimental/tls_handshake.py | 37 ++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index d990a5b..5dcc107 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -200,6 +200,37 @@ class TLSHandshakeTest(nettest.NetTestCase): cert_chain.append(pem_cert) return cert_chain + @staticmethod + def getX509Name(certificate, get_components=False): + """ + Get the DER encoded form of the Name portions of and X509 certificate. + + @param certificate: A :class:`OpenSSL.crypto.X509Name` object. + @param get_components: A boolean. If True, returns a list of tuples of + the (name, value)s of each Name field in the + :param:`certificate`. If False, returns the DER + encoded form of the Name fields of the + :param:`certificate`. + """ + x509_name = None + + try: + assert isinstance(certificate, crypto.X509Name), \ + "getX509Name takes OpenSSL.crypto.X509Name as first argument!" + x509_name = crypto.X509Name(certificate) + except AssertionError as ae: + log.err(ae) + except Exception as exc: + log.exception(exc) + + if not x509_name is None: + if not get_components: + return x509_name.der() + else: + return x509_name.get_components() + else: + log.debug("getX509Name: got None for ivar x509_name") + def test_tlsv1_handshake(self): """xxx fill me in""" @@ -538,9 +569,11 @@ class TLSHandshakeTest(nettest.NetTestCase): server_cert_chain = self.getPeerCert(connection, get_chain=True) s_cert = connection.get_peer_certificate() - cert_subject = s_cert.get_subject() + cert_subject = getX509Name(s_cert.get_subject(), + get_components=True) cert_subj_hash = s_cert.subject_name_hash() - cert_issuer = s_cert.get_issuer() + cert_issuer = getX509Name(s_cert.get_issuer(), + get_components=True) cert_public_key = s_cert.get_pubkey() cert_serial_no = s_cert.get_serial_number() cert_sig_algo = s_cert.get_signature_algorithm()

1 0

[ooni-probe/develop] Fix import statements to include X509Name class.
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit ebdf72719f82a575717a2713f52d5fa2665bf835 Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 13:23:45 2013 +0000 Fix import statements to include X509Name class. --- nettests/experimental/tls_handshake.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 5dcc107..34a917c 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -33,6 +33,7 @@ import types from ipaddr import IPAddress from OpenSSL import SSL from OpenSSL.crypto import dump_certificate, FILETYPE_PEM +from OpenSSL.crypto import X509Name from twisted.internet import defer from twisted.python import usage from twisted.python.failure import Failure @@ -215,9 +216,9 @@ class TLSHandshakeTest(nettest.NetTestCase): x509_name = None try: - assert isinstance(certificate, crypto.X509Name), \ + assert isinstance(certificate, X509Name), \ "getX509Name takes OpenSSL.crypto.X509Name as first argument!" - x509_name = crypto.X509Name(certificate) + x509_name = X509Name(certificate) except AssertionError as ae: log.err(ae) except Exception as exc:

1 0

[ooni-probe/develop] Fix method connectionFailed to handle socket timeouts.
by isis＠torproject.org 07 Jun '13

07 Jun '13

commit 61cea5561d9a22dd9009331ca16b7506a3c24d7f Author: Isis Lovecruft <isis(a)torproject.org> Date: Thu Feb 28 17:42:31 2013 +0000 Fix method connectionFailed to handle socket timeouts. --- nettests/experimental/tls_handshake.py | 33 +++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/nettests/experimental/tls_handshake.py b/nettests/experimental/tls_handshake.py index 4c42017..610cdff 100644 --- a/nettests/experimental/tls_handshake.py +++ b/nettests/experimental/tls_handshake.py @@ -292,22 +292,29 @@ class TLSHandshakeTest(nettest.NetTestCase): specifying the host port, i.e. ('1.1.1.1', 443) """ addr, port = host - if isinstance(connection, IOError): - ## On some *nix distros, /dev/random is 0600 root:root and we get - ## a permissions error when trying to read - if connection.message.find("[Errno 13]"): - raise NotRootError( - "%s" % connection.message.split("[Errno 13]", 1)[1]) - - if isinstance(connection, socket_error): - if connection.message.find("[Errno 101]"): - raise HostUnreachableError( - "Host unreachable: %s:%s" % (addr, port)) - - log.err(connection) + + if not isinstance(connection, SSL.Connection): + if isinstance(connection, IOError): + ## On some *nix distros, /dev/random is 0600 root:root and + ## we get a permissions error when trying to read + if connection.message.find("[Errno 13]"): + raise NotRootError( + "%s" % connection.message.split("[Errno 13]", 1)[1]) + elif isinstance(connection, socket_error): + if connection.message.find("[Errno 101]"): + raise HostUnreachableError( + "Host unreachable: %s:%s" % (addr, port)) + elif isinstance(connection, Exception): + log.debug("connectionFailed: got Exception:") + log.err("Connection failed with reason: %s" + % connection.message) + else: + log.err("Connection failed with reason: %s" % str(connection)) + self.report['host'] = addr self.report['port'] = port self.report['state'] = 'CONNECTION_FAILED' + return connection def connectionSucceeded(connection, host, timeout):

1 0