October 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] Ensure annotations are always dicts
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 528ace6a2eec664d86bca38214462d9874b370e2 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 12:38:07 2016 +0200 Ensure annotations are always dicts --- ooni/nettest.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ooni/nettest.py b/ooni/nettest.py index 5d33342..b81678b 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -171,6 +171,9 @@ class NetTestLoader(object): def __init__(self, options, test_file=None, test_string=None, annotations={}): self.options = options + # Ensure annotations are always dicts + if not isinstance(annotations, dict): + annotations = {} self.annotations = annotations self.annotations['platform'] = self.annotations.get('platform', config.platform)

1 0

[ooni-probe/master] Add support for generating manpages from rst files.
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 41f80dbc1cbc706ff4d08e35cc82d1e05fa95f21 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 19:35:32 2016 +0200 Add support for generating manpages from rst files. Add manpages for ooni* --- docs/source/conf.py | 7 +- docs/source/manual/oonideckgen.rst | 17 ++++ docs/source/manual/ooniprobe-agent.rst | 103 ++++++++++++++++++++++++ docs/source/manual/ooniprobe.rst | 140 +++++++++++++++++++++++++++++++++ docs/source/manual/oonireport.rst | 85 ++++++++++++++++++++ docs/source/manual/ooniresources.rst | 15 ++++ 6 files changed, 365 insertions(+), 2 deletions(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 8600641..e3c6544 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -218,8 +218,11 @@ latex_documents = [ # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ - ('index', 'ooniprobe', u'an internet censorship measurement tool', - [u'The Tor Project'], 1) + ('manual/oonideckgen', 'oonideckgen', u'DEPRECATED ooni tool', [u'The Tor Project'], 1), + ('manual/ooniprobe-agent', 'ooniprobe-agent', u'an internet censorship measurement tool, system daemon.', [u'The Tor Project'], 1), + ('manual/ooniprobe', 'ooniprobe', u'an internet censorship measurement tool', [u'The Tor Project'], 1), + ('manual/oonireport', 'oonireport', u'upload ooniprobe measurements', [u'The Tor Project'], 1), + ('manual/ooniresources', 'ooniresources', u'DEPRECATED ooni tool', [u'The Tor Project'], 1), ] # If true, show URL addresses after external links. diff --git a/docs/source/manual/oonideckgen.rst b/docs/source/manual/oonideckgen.rst new file mode 100644 index 0000000..cbdddc4 --- /dev/null +++ b/docs/source/manual/oonideckgen.rst @@ -0,0 +1,17 @@ +oonideckgen - DEPRECATED +======================== + +Synopsis +-------- + +**oonideckgen** + +Description +----------- + +:program:`oonideckgen`, has now been superseded by the new test +deck format. +You will will find all the decks you need to run inside of +/usr/share/ooni/decks-available/ and these can be run with: + + :program:`ooniprobe -i /usr/share/ooni/decks-available/web.yaml` diff --git a/docs/source/manual/ooniprobe-agent.rst b/docs/source/manual/ooniprobe-agent.rst new file mode 100644 index 0000000..1958ab5 --- /dev/null +++ b/docs/source/manual/ooniprobe-agent.rst @@ -0,0 +1,103 @@ +ooniprobe-agent +=============== + +Synopsis +-------- + +**ooniprobe-agent** start | stop | run | status + +Description +----------- + +:program:`ooniprobe-agent`, is a system daemon responsible for automatically +running OONI measurements and exposing a web user interface. The web user +interface is accessible by default on http://127.0.0.1:8842/ + +Options +------- + +--help + Display help and exit. + +--version + Display the ooniprobe version and exit. + +Commands +-------- + +**start** +Start the ooniprobe-agent in the background + +**stop** +Stop the ooniprobe-agent + +**status** +Show status of the ooniprobe-agent + +**run** +Run the ooniprobe-agent in the foreground + + +ooniprobe +--------- + +Read this before running ooniprobe! +................................... +Running ooniprobe is a potentially risky activity. This greatly depends on the +jurisdiction in which you are in and which test you are running. It is +technically possible for a person observing your internet connection to be +aware of the fact that you are running ooniprobe. This means that if running +network measurement tests is something considered to be illegal in your country +then you could be spotted. + +Furthermore, ooniprobe takes no precautions to protect the install target machine +from forensics analysis. If the fact that you have installed or used ooni +probe is a liability for you, please be aware of this risk. + +What is this? +............. + +ooniprobe is the command line tool that volunteers and researches interested in +contributing data to the project should be running. + +If you are interested in using ooniprobe from a graphical user interface +refer to :program:`ooniprobe-agent` and see how to run that. + +ooniprobe allows the user to select what test should be run and what backend +should be used for storing the test report and/or assisting them in the running +of the test. + +ooniprobe tests are divided into two categories: **Traffic Manipulation** and +**Content Blocking**. + +**Traffic Manipulation** tests aim to detect the presence of some sort of +tampering with the internet traffic between the probe and a remote test helper +backend. As such they usually require the selection of a oonib backend +component for running the test. + +**Content Blocking** are aimed at enumerating the kind of content that is +blocked from the probes network point of view. As such they usually require to +have specified an input list for running the test. + +Examples +-------- + +Run the web_connectivity test on http://torproject.org: + + :program:`ooniprobe web_connectivity --url http://torproject.org/` + +Run the http_invalid_request_line test to detect middleboxes: + + :program:`ooniprobe http_invalid_request_line` + +Run the http_header_field_manipulation test to detect middleboxes: + + :program:`ooniprobe http_header_field_manipulation` + +List all the available tests: + + :program:`ooniprobe -s` + +Start the web user interface: + + :program:`ooniprobe -w` diff --git a/docs/source/manual/ooniprobe.rst b/docs/source/manual/ooniprobe.rst new file mode 100644 index 0000000..7d71795 --- /dev/null +++ b/docs/source/manual/ooniprobe.rst @@ -0,0 +1,140 @@ +ooniprobe +========= + +Synopsis +-------- + +**ooniprobe** [*options*] ([*test name*] | [*path to nettest*.py]) + +Description +----------- + +:program:`ooniprobe`, is a tool for performing internet censorship +measurements. Our goal is to achieve a common data format and set of +methodologies for conducting censorship related research. + +Options +------- + +-h, --help + Display help and exit. + +-n, --no-collector + Disable writing to collector + +-N, --no-njson + Disable writing to disk + +-g, --no-geoip + Disable geoip lookup on start + +-s, --list + List the currently installed ooniprobe nettests + +-v, --verbose + Show more verbose information + +-w, --web-ui + Start the web UI + +-z, --initialize + Initialize ooniprobe to begin running it + +-o, --reportfile PATH_TO_FILE + Specify the report file name to write to. + +-i, --testdeck PATH_TO_DECK + Specify as input a test deck: a yaml file containing the tests to run and their arguments. + +-c, --collector COLLECTOR_ADDRESS + Specify the address of the collector for test results. In most cases a user + will prefer to specify a bouncer over this. + +-b, --bouncer BOUNCER_ADDRESS + Specify the bouncer used to obtain the address of the collector and test helpers. + +-l, --logfile PATH_TO_LOGFILE + Write to this logs to this filename. + +-O, --pcapfile PATH_TO_PCAPFILE + Write a PCAP of the ooniprobe session to this filename. + +-f, --configfile PATH_TO_CONFIG + Specify a path to the ooniprobe configuration file. + +-d, --datadir + Specify a path to the ooniprobe data directory. + +-a, --annotations key:value[,key2:value2] + Annotate the report with a key:value[, key:value] format. + +-P, --preferred-backend onion|https|cloudfront + Set the preferred backend to use when submitting results and/or + communicating with test helpers. Can be either onion, https or cloudfront + +--version + Display the ooniprobe version and exit. + +ooniprobe +--------- + +Read this before running ooniprobe! +................................... +Running ooniprobe is a potentially risky activity. This greatly depends on the +jurisdiction in which you are in and which test you are running. It is +technically possible for a person observing your internet connection to be +aware of the fact that you are running ooniprobe. This means that if running +network measurement tests is something considered to be illegal in your country +then you could be spotted. + +Furthermore, ooniprobe takes no precautions to protect the install target machine +from forensics analysis. If the fact that you have installed or used ooni +probe is a liability for you, please be aware of this risk. + +What is this? +............. + +ooniprobe is the command line tool that volunteers and researches interested in +contributing data to the project should be running. + +If you are interested in using ooniprobe from a graphical user interface +refer to :program:`ooniprobe-agent` and see how to run that. + +ooniprobe allows the user to select what test should be run and what backend +should be used for storing the test report and/or assisting them in the running +of the test. + +ooniprobe tests are divided into two categories: **Traffic Manipulation** and +**Content Blocking**. + +**Traffic Manipulation** tests aim to detect the presence of some sort of +tampering with the internet traffic between the probe and a remote test helper +backend. As such they usually require the selection of a oonib backend +component for running the test. + +**Content Blocking** are aimed at enumerating the kind of content that is +blocked from the probes network point of view. As such they usually require to +have specified an input list for running the test. + +Examples +-------- + +Run the web_connectivity test on http://torproject.org: + + :program:`ooniprobe web_connectivity --url http://torproject.org/` + +Run the http_invalid_request_line test to detect middleboxes: + + :program:`ooniprobe http_invalid_request_line` + +Run the http_header_field_manipulation test to detect middleboxes: + + :program:`ooniprobe http_header_field_manipulation` + +List all the available tests: + + :program:`ooniprobe -s` + +Start the web user interface: + + :program:`ooniprobe -w` diff --git a/docs/source/manual/oonireport.rst b/docs/source/manual/oonireport.rst new file mode 100644 index 0000000..a7dced0 --- /dev/null +++ b/docs/source/manual/oonireport.rst @@ -0,0 +1,85 @@ +oonireport +========== + +Synopsis +-------- + +**oonireport** [*options*] upload | status [*path to report*] + +Description +----------- + +:program:`oonireport` is a tool for viewing which ooniprobe +reports have not been submitted to a collector and upload them. + + +:program:`ooniprobe -i /usr/share/ooni/decks-available/web.yaml` + +Options +------- + +-d, --default-collector + Upload the reports to the default collector that is looked up + with the canonical bouncer. + +-f, --configfile + Specify the configuration file to use. + +-c, --collector + Specify the collector to upload the result to. + +-b, --bouncer + Specify the bouncer to query for a collector. + + --version + + --help Display this help and exit. + +Commands +-------- + +**upload** + +If no argument is specified all the reports that have not been +submitted will be submitted to either the collector specified as +an option (-c) or to the collector that was previously used. + +If a report file is specified then only that report file will be +uploaded + +**status** + +Outputs all of the reports that are either "not uploaded", +"incomplete" or "pending". + +The possible statuses are: + +not-created + +If it has not been created, because the user specified to not use a +collector. + +creation-failed + +If we attempted to create the report, but failed to do so either because +the collector does not accept our report or because it was unreachable at the +time. + +created + +If the report has been successfully created, but has not yet been +closed. + +incomplete + +If the report has been created, but we have failed to update the +report with some data. + +oonireport +---------- + +This tool is used to upload reports that are stored on the users +filesystem and have not been submitted to a collector. This can be +either because the collector was unreachable when we attempted to +submit the report or because the policy of the collector did not +support the specified test. diff --git a/docs/source/manual/ooniresources.rst b/docs/source/manual/ooniresources.rst new file mode 100644 index 0000000..b7840e8 --- /dev/null +++ b/docs/source/manual/ooniresources.rst @@ -0,0 +1,15 @@ +ooniresources - DEPRECATED +======================== + +Synopsis +-------- + +**ooniresources** + +Description +----------- + +:program:`ooniresources`, has now been superseded by a system that +automatically manages downloads of new resources. + +There is no reason to use this anymore.

1 0

[ooni-probe/master] Annotate reports with the platform on which they are running. (#632)
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 4ef87a3b47e6939b8dfdddf674e5dccc22747cb0 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 11:56:57 2016 +0200 Annotate reports with the platform on which they are running. (#632) --- ooni/nettest.py | 2 ++ ooni/settings.py | 15 +++++++++++++++ 2 files changed, 17 insertions(+) diff --git a/ooni/nettest.py b/ooni/nettest.py index 1bb5678..5d33342 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -172,6 +172,8 @@ class NetTestLoader(object): annotations={}): self.options = options self.annotations = annotations + self.annotations['platform'] = self.annotations.get('platform', + config.platform) self.requiresTor = False diff --git a/ooni/settings.py b/ooni/settings.py index 44bd8dc..ef220ca 100644 --- a/ooni/settings.py +++ b/ooni/settings.py @@ -3,6 +3,7 @@ import sys import yaml import errno import getpass +import platform from ConfigParser import SafeConfigParser from twisted.internet import defer, reactor @@ -325,6 +326,20 @@ class OConfig(object): self.advanced.webui_port ) + @property + def platform(self): + if os.path.exists('/etc/default/lepidopter'): + return 'lepidopter' + system = platform.system() + if system == 'Darwin': + return 'macos' + elif system == 'Linux': + return 'linux' + elif system == 'Windows': + # Really? + return 'windows' + return 'unknown' + def get_data_file_path(self, file_name): for target_dir in self.data_directory_candidates: file_path = os.path.join(target_dir, file_name)

1 0

[ooni-probe/master] Update manpages and add manpage for ooniprobe-agent
by art＠torproject.org 14 Oct '16

14 Oct '16

commit d21fc48cb7b0a353c09d01ff04975f052549569e Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 19:36:13 2016 +0200 Update manpages and add manpage for ooniprobe-agent --- data/oonideckgen.1 | 66 +-- data/ooniprobe-agent.1 | 140 +++++++ data/ooniprobe.1 | 1092 ++++-------------------------------------------- data/oonireport.1 | 105 +++-- data/ooniresources.1 | 43 +- 5 files changed, 305 insertions(+), 1141 deletions(-) diff --git a/data/oonideckgen.1 b/data/oonideckgen.1 index ae30521..e3bcf62 100644 --- a/data/oonideckgen.1 +++ b/data/oonideckgen.1 @@ -1,8 +1,8 @@ .\" Man page generated from reStructuredText. . -.TH "oonideckgen" "1" "October 1, 2014" "1.1.4" "oonideckgen" +.TH "OONIDECKGEN" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" .SH NAME -oonideckgen - tool for generating ooniprobe test decks +oonideckgen \- DEPRECATED ooni tool . .nr rst2man-indent-level 0 . @@ -30,61 +30,23 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. - .SH SYNOPSIS -.B oonideckgen -.RB [ --version ] -.RB [ --help ] -.RB [ --collector -.IR httpo://collector-address.onion ] -.RB [ \-c -.IR country-code] -.RB [ \-o -.IR output] - -.SH DESCRIPTION -.sp -oonideckgen will generate a ooniprobe test deck for the specified country. This -will then allow the user to conduct measurements useful to assess internet -censorship in that country. - - -.SH OPTIONS - -.TP -.BR \-\^h " or " \-\-help -Display this help and exit. -.TP -.BR \-\^c " or " \-\-country-code -Specify the country code for which a test deck should be geneated -.TP -.BR \-\-collector -Specify the address of the collector to use for submitting reports. -.TP -.BR \-\^o " or " \-\-output -Specify in which directory the deck should be written. -.TP - - -.SH OONIDECKGEN .sp -This tools will generate a test deck that include the following tests: - -* blocking/http_requests, with a set of relevant sites for the country to be analysed - -* blocking/dns_consistency, with all the DNS resolver of the country to be analysed and a set of relevant sites - -* manipulation/http_invalid_request_line - -* manipulation/http_header_field_manipulation - -If deckgen is run with no argument a geoip lookup is done on the IP of the user -and that country is chosen. -In order to run deckgen you must have first have run ooniresources --update-inputs. +\fBoonideckgen\fP +.SH DESCRIPTION .sp - +\fBoonideckgen\fP, has now been superseded by the new test +deck format. +You will will find all the decks you need to run inside of +/usr/share/ooni/decks\-available/ and these can be run with: +.INDENT 0.0 +.INDENT 3.5 +\fBooniprobe \-i /usr/share/ooni/decks\-available/web.yaml\fP +.UNINDENT +.UNINDENT .SH AUTHOR The Tor Project .SH COPYRIGHT 2016, The Tor Project +.\" Generated by docutils manpage writer. . diff --git a/data/ooniprobe-agent.1 b/data/ooniprobe-agent.1 new file mode 100644 index 0000000..61ace5a --- /dev/null +++ b/data/ooniprobe-agent.1 @@ -0,0 +1,140 @@ +.\" Man page generated from reStructuredText. +. +.TH "OONIPROBE-AGENT" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" +.SH NAME +ooniprobe-agent \- an internet censorship measurement tool, system daemon. +. +.nr rst2man-indent-level 0 +. +.de1 rstReportMargin +\\$1 \\n[an-margin] +level \\n[rst2man-indent-level] +level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] +- +\\n[rst2man-indent0] +\\n[rst2man-indent1] +\\n[rst2man-indent2] +.. +.de1 INDENT +.\" .rstReportMargin pre: +. RS \\$1 +. nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] +. nr rst2man-indent-level +1 +.\" .rstReportMargin post: +.. +.de UNINDENT +. RE +.\" indent \\n[an-margin] +.\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] +.nr rst2man-indent-level -1 +.\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] +.in \\n[rst2man-indent\\n[rst2man-indent-level]]u +.. +.SH SYNOPSIS +.sp +\fBooniprobe\-agent\fP start | stop | run | status +.SH DESCRIPTION +.sp +\fBooniprobe\-agent\fP, is a system daemon responsible for automatically +running OONI measurements and exposing a web user interface. The web user +interface is accessible by default on \fI\%http://127.0.0.1:8842/\fP +.SH OPTIONS +.INDENT 0.0 +.TP +.B \-\-help +Display help and exit. +.TP +.B \-\-version +Display the ooniprobe version and exit. +.UNINDENT +.SH COMMANDS +.sp +\fBstart\fP +Start the ooniprobe\-agent in the background +.sp +\fBstop\fP +Stop the ooniprobe\-agent +.sp +\fBstatus\fP +Show status of the ooniprobe\-agent +.sp +\fBrun\fP +Run the ooniprobe\-agent in the foreground +.SH OONIPROBE +.SS Read this before running ooniprobe! +.sp +Running ooniprobe is a potentially risky activity. This greatly depends on the +jurisdiction in which you are in and which test you are running. It is +technically possible for a person observing your internet connection to be +aware of the fact that you are running ooniprobe. This means that if running +network measurement tests is something considered to be illegal in your country +then you could be spotted. +.sp +Furthermore, ooniprobe takes no precautions to protect the install target machine +from forensics analysis. If the fact that you have installed or used ooni +probe is a liability for you, please be aware of this risk. +.SS What is this? +.sp +ooniprobe is the command line tool that volunteers and researches interested in +contributing data to the project should be running. +.sp +If you are interested in using ooniprobe from a graphical user interface +refer to \fBooniprobe\-agent\fP and see how to run that. +.sp +ooniprobe allows the user to select what test should be run and what backend +should be used for storing the test report and/or assisting them in the running +of the test. +.sp +ooniprobe tests are divided into two categories: \fBTraffic Manipulation\fP and +\fBContent Blocking\fP\&. +.sp +\fBTraffic Manipulation\fP tests aim to detect the presence of some sort of +tampering with the internet traffic between the probe and a remote test helper +backend. As such they usually require the selection of a oonib backend +component for running the test. +.sp +\fBContent Blocking\fP are aimed at enumerating the kind of content that is +blocked from the probes network point of view. As such they usually require to +have specified an input list for running the test. +.SH EXAMPLES +.sp +Run the web_connectivity test on \fI\%http://torproject.org\fP: +.INDENT 0.0 +.INDENT 3.5 +\fBooniprobe web_connectivity \-\-url http://torproject.org/\fP +.UNINDENT +.UNINDENT +.sp +Run the http_invalid_request_line test to detect middleboxes: +.INDENT 0.0 +.INDENT 3.5 +\fBooniprobe http_invalid_request_line\fP +.UNINDENT +.UNINDENT +.sp +Run the http_header_field_manipulation test to detect middleboxes: +.INDENT 0.0 +.INDENT 3.5 +\fBooniprobe http_header_field_manipulation\fP +.UNINDENT +.UNINDENT +.sp +List all the available tests: +.INDENT 0.0 +.INDENT 3.5 +\fBooniprobe \-s\fP +.UNINDENT +.UNINDENT +.sp +Start the web user interface: +.INDENT 0.0 +.INDENT 3.5 +\fBooniprobe \-w\fP +.UNINDENT +.UNINDENT +.SH AUTHOR +The Tor Project +.SH COPYRIGHT +2016, The Tor Project +.\" Generated by docutils manpage writer. +. diff --git a/data/ooniprobe.1 b/data/ooniprobe.1 index 9a514b4..5f64a0a 100644 --- a/data/ooniprobe.1 +++ b/data/ooniprobe.1 @@ -1,7 +1,8 @@ +.\" Man page generated from reStructuredText. . -.TH "ooniprobe" "1" "April 26, 2016" "1.4.0" "ooniprobe" +.TH "OONIPROBE" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" .SH NAME -ooniprobe - a network censorship measurement tool. +ooniprobe \- an internet censorship measurement tool . .nr rst2man-indent-level 0 . @@ -29,157 +30,82 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. - .SH SYNOPSIS -.B ooniprobe -.RB [ \-hnspgN ] -.RB [ --version ] -.RB [ --spew ] -.RB [ \-o -.IR reportfile ] -.RB [ \-i -.IR testdeck ] -.RB [ \-c -.IR collector ] -.RB [ \-b -.IR bouncer ] -.RB [ \-l -.IR logfile ] -.RB [ \-O -.IR pcapfile ] -.RB [ \-f -.IR configfile ] -.RB [ \-d -.IR datadir ] -.RB [ \-a -.IR annotations] -.I "test_name" - +.sp +\fBooniprobe\fP [\fIoptions\fP] ([\fItest name\fP] | [\fIpath to nettest\fP\&.py]) .SH DESCRIPTION .sp -ooniprobe is tool for performing internet censorship measurements. Our goal is -to achieve a common data format and set of methodologies for conducting -censorship related research. - +\fBooniprobe\fP, is a tool for performing internet censorship +measurements. Our goal is to achieve a common data format and set of +methodologies for conducting censorship related research. .SH OPTIONS - +.INDENT 0.0 +.TP +.B \-h\fP,\fB \-\-help +Display help and exit. .TP -.BR \-\^h " or " \-\-help -Display this help and exit. +.B \-n\fP,\fB \-\-no\-collector +Disable writing to collector .TP -.BR \-\^n " or " \-\-no\-collector -Disable reporting the net test results to an oonib collector. +.B \-N\fP,\fB \-\-no\-njson +Disable writing to disk .TP -.BR \-\^N " or " \-\-no\-yamloo -Disable writing report file to disk. +.B \-g\fP,\fB \-\-no\-geoip +Disable geoip lookup on start .TP -.BR \-\^g " or " \-\-no\-geoip -Disable performing geoip lookup. +.B \-s\fP,\fB \-\-list +List the currently installed ooniprobe nettests .TP -.BR \-\^s " or " \-\-list -List all of the available net tests. +.B \-v\fP,\fB \-\-verbose +Show more verbose information .TP -.BR \-\^p " or " \-\-printdeck -Print to standard output the specified command line options as an ooniprobe test deck. +.B \-w\fP,\fB \-\-web\-ui +Start the web UI .TP -.BR \-\^o " or " \-\-reportfile -Specify the path to report file to write. +.B \-z\fP,\fB \-\-initialize +Initialize ooniprobe to begin running it .TP -.BR \-\^i " or " \-\-testdeck -Specify as input a test deck: a yaml file containing the tests to run and their -arguments. +.BI \-o\fP,\fB \-\-reportfile \ PATH_TO_FILE +Specify the report file name to write to. .TP -.BR \-\^c " or " \-\-collector -Specify the address of the collector of net test results. It is advisable to -always specify and bouncer and let it return a collector for the test or test -deck you are running. +.BI \-i\fP,\fB \-\-testdeck \ PATH_TO_DECK +Specify as input a test deck: a yaml file containing the tests to run and their arguments. .TP -.BR \-\^b " or " \-\-bouncer -Address of the bouncer that will inform the probe of which collector to use and -the addresses of test helpers. default: httpo://nkvphnp3p6agi5qq.onion +.BI \-c\fP,\fB \-\-collector \ COLLECTOR_ADDRESS +Specify the address of the collector for test results. In most cases a user +will prefer to specify a bouncer over this. .TP -.BR \-\^l " or " \-\-logfile -Path to the log file to write +.BI \-b\fP,\fB \-\-bouncer \ BOUNCER_ADDRESS +Specify the bouncer used to obtain the address of the collector and test helpers. .TP -.BR \-\^O " or " \-\-pcapfile -Prefix to the pcap file name. +.BI \-l\fP,\fB \-\-logfile \ PATH_TO_LOGFILE +Write to this logs to this filename. .TP -.BR \-\^f " or " \-\-configfile +.BI \-O\fP,\fB \-\-pcapfile \ PATH_TO_PCAPFILE +Write a PCAP of the ooniprobe session to this filename. +.TP +.BI \-f\fP,\fB \-\-configfile \ PATH_TO_CONFIG Specify a path to the ooniprobe configuration file. .TP -.BR \-\^d " or " \-\-datadir -Specify a path to the ooniprobe data directory +.B \-d\fP,\fB \-\-datadir +Specify a path to the ooniprobe data directory. +.UNINDENT +.INDENT 0.0 .TP -.BR \-\^a " or " \-\-annotations +.B \-a, \-\-annotations key:value[,key2:value2] Annotate the report with a key:value[, key:value] format. .TP -.BR \-\-spew -Print an insanely verbose log of everything that happens. -Useful when debugging freezes or locks in complex code. -.TP -.BR \-\-version -Display the ooniprobe version and exit. - -.SH OONIPROBE -.sp -Is the tool that volunteers and researches interested in contributing data to -the project should be running. -.sp -ooniprobe allows the user to select what test should be run and what backend -should be used for storing the test report and/or assisting them in the running -of the test. -.sp -ooniprobe tests are divided into two categories: \fBTraffic Manipulation\fP and -\fBContent Blocking\fP\&. -.sp -\fBTraffic Manipulation\fP tests aim to detect the presence of some sort of -tampering with the internet traffic between the probe and a remote test helper -backend. As such they usually require the selection of a oonib backend -component for running the test. -.sp -\fBContent Blocking\fP are aimed at enumerating the kind of content that is -blocked from the probes network point of view. As such they usually require to -have specified an input list for running the test. -.SS Threat Model -.sp -Our adversary is capable of doing country wide network surveillance and -manipulation of network traffic. -.sp -The goals of our adversary are: -.INDENT 0.0 -.INDENT 3.5 -.INDENT 0.0 -.IP \(bu 2 -Restrict access to certain content, while not degrading overall quality of -the network -.IP \(bu 2 -Monitor the network in a way that they are able to identify misuse of it in -real time +.B \-P, \-\-preferred\-backend onion|https|cloudfront +Set the preferred backend to use when submitting results and/or +communicating with test helpers. Can be either onion, https or cloudfront .UNINDENT -.UNINDENT -.UNINDENT -.sp -More specific to the running of network filtering detection tests: .INDENT 0.0 -.IP 1. 3 -Detect actors performing censorship detection tests -.IP 2. 3 -Fool people running such tests into believing that the network is -unrestricted - +.TP +.B \-\-version +Display the ooniprobe version and exit. .UNINDENT -.sp -\fINote\fP that while 2) => 1) it is not true that 1) => 2) as the identification of -such actors does not necessarily have to happen in real time. -While our intention is to minimize the risk of users running OONI probe to be -identified, this comes with a tradeoff in accuracy. It is therefore necessary in -certain tests to trade\-off fingerprintability in favour of tests accuracy. -.sp -This is why we divide tests based on what risk the user running it can face, -allowing the user to freely choose what threat model they wish to adhere to. -.SS Installation -.sp -\fBRead this before running ooniprobe!\fP +.SH OONIPROBE +.SS Read this before running ooniprobe! .sp Running ooniprobe is a potentially risky activity. This greatly depends on the jurisdiction in which you are in and which test you are running. It is @@ -191,920 +117,68 @@ then you could be spotted. Furthermore, ooniprobe takes no precautions to protect the install target machine from forensics analysis. If the fact that you have installed or used ooni probe is a liability for you, please be aware of this risk. -.SS Debian based systems -.sp -\fIsudo sh \-c \(aqecho "deb http://deb.ooni.nu/ooni wheezy main" >> /etc/apt/sources.list\(aq\fP -.sp -\fIgpg \-\-keyserver pgp.mit.edu \-\-recv\-key 0x49B8CDF4\fP -.sp -\fIgpg \-\-export 89AB86D4788F3785FE9EDA31F9E2D9B049B8CDF4 | sudo apt\-key add \-\fP -.sp -\fIsudo apt\-get update && sudo apt\-get install ooniprobe\fP -.SS Linux -.sp -We believe that ooniprobe runs reasonably well on Debian GNU/Linux wheezy as -well as versions of Ubuntu such as natty and later releases. Running ooniprobe -without installing it is supported with the following commands: -.sp -\fIgit clone https://git.torproject.org/ooni\-probe.git\fP -.sp -\fIcd ooni\-probe\fP -.sp -\fI\&./setup\-dependencies.sh\fP -.sp -\fIpython setup.py install\fP -.SS Setting up development environment -.sp -On debian based systems this can be done with: -.sp -\fIVsudo apt\-get install libgeoip\-dev python\-virtualenv virtualenvwrapper\fP -.sp -\fImkvirtualenv ooniprobe\fP -.sp -\fIpython setup.py install\fP -.sp -\fIpip install \-r requirements\-dev.txt\fP -.SS Other platforms (with Vagrant) -.sp -\fI\%Install Vagrant\fP -and \fI\%Install Virtualbox\fP -.sp -\fBOn OSX:\fP -.sp -If you don\(aqt have it install \fI\%homebrew\fP -.sp -\fIbrew install git\fP -.sp -\fBOn debian/ubuntu:\fP -.sp -\fIsudo apt\-get install git\fP -.INDENT 0.0 -.IP 1. 3 -Open a Terminal and run: -.UNINDENT -.sp -\fIgit clone https://git.torproject.org/ooni\-probe.git\fP -.sp -\fIcd ooni\-probe/\fP -.sp -\fIvagrant up\fP -.INDENT 0.0 -.IP 2. 3 -Login to the box with: -.UNINDENT -.sp -\fIvagrant ssh\fP -.sp -ooniprobe will be installed in \fI/ooni\fP\&. -.INDENT 0.0 -.IP 3. 3 -You can run tests with: -.UNINDENT -.sp -\fIooniprobe blocking/http_requests \-f /ooni/inputs/input\-pack/alexa\-top\-1k.txt\fP -.SS Using ooniprobe -.sp -\fBNet test\fP is a set of measurements to assess what kind of internet censorship is occurring. -.sp -\fBDecks\fP are collections of ooniprobe nettests with some associated inputs. -.sp -\fBCollector\fP is a service used to report the results of measurements. -.sp -\fBTest helper\fP is a service used by a probe for successfully performing its measurements. -.sp -\fBBouncer\fP is a service used to discover the addresses of test helpers and collectors. -.SS Configuring ooniprobe -.sp -You may edit the configuration for ooniprobe by editing the configuration file -found inside of \fI~/.ooni/ooniprobe.conf\fP\&. -.sp -By default ooniprobe will not include personal identifying information in the -test result, nor create a pcap file. This behavior can be personalized. -.SS Running decks -.sp -You will find all the installed decks inside of \fI/usr/share/ooni/decks\fP\&. -.sp -You may then run a deck by using the command line option \fI\-i\fP: -.sp -As root: -.sp -\fIooniprobe \-i /usr/share/ooni/decks/mlab.deck\fP -.sp -Or as a user: -.sp -\fIooniprobe \-i /usr/share/ooni/decks/mlab_no_root.deck\fP -.sp -Or: -.sp -As root: -.sp -\fIooniprobe \-i /usr/share/ooni/decks/complete.deck\fP -.sp -Or as a user: -.sp -\fIooniprobe \-i /usr/share/ooni/decks/complete_no_root.deck\fP -.sp -The above tests will require around 20\-30 minutes to complete depending on your network speed. -.sp -If you would prefer to run some faster tests you should run: -As root: -.sp -\fIooniprobe \-i /usr/share/ooni/decks/fast.deck\fP -.sp -Or as a user: -.sp -\fIooniprobe \-i /usr/share/ooni/decks/fast_no_root.deck\fP -.SS Running net tests -.sp -You may list all the installed stable net tests with: -.sp -\fIooniprobe \-s\fP -.sp -You may then run a nettest by specifying its name for example: -.sp -\fIooniprobe manipulation/http_header_field_manipulation\fP -.sp -It is also possible to specify inputs to tests as URLs: -.sp -\fIooniprobe blocking/http_requests \-f httpo://ihiderha53f36lsd.onion/input/37e60e13536f6afe47a830bfb6b371b5cf65da66d7ad65137344679b24fdccd1\fP -.sp -You can find the result of the test in your current working directory. -.sp -By default the report result will be collected by the default ooni collector -and the addresses of test helpers will be obtained from the default bouncer. -.sp -You may also specify your own collector or bouncer with the options \fI\-c\fP and -\fI\-b\fP\&. -.SS (Optional) Install obfsproxy -.sp -Install the latest version of obfsproxy for your platform. +.SS What is this? .sp -\fI\%Download Obfsproxy\fP -.SS Bridges and obfsproxy bridges +ooniprobe is the command line tool that volunteers and researches interested in +contributing data to the project should be running. .sp -ooniprobe submits reports to oonib report collectors through Tor to a hidden -service endpoint. By default, ooniprobe uses the installed system Tor, but can -also be configured to launch Tor (see the advanced.start_tor option in -ooniprobe.conf), and ooniprobe supports bridges (and obfsproxy bridges, if -obfsproxy is installed). The tor.bridges option in ooniprobe.conf sets the path -to a file that should contain a set of "bridge" lines (of the same format as -used in torrc, and as returned by \fI\%https://bridges.torproject.org\fP). If obfsproxy -bridges are to be used, the path to the obfsproxy binary must be configured. -See option advanced.obfsproxy_binary, in ooniprobe.conf. -.SS Setting capabilities on your virtualenv python binary +If you are interested in using ooniprobe from a graphical user interface +refer to \fBooniprobe\-agent\fP and see how to run that. .sp -If your distribution supports capabilities you can avoid needing to run OONI as root: -.sp -\fIsetcap cap_net_admin,cap_net_raw+eip /path/to/your/virtualenv\(aqs/python\fP -.SS Core ooniprobe Tests -.sp -The source for \fI\%Content blocking tests\fP -and \fI\%Traffic Manipulation tests\fP -can be found in the nettests/blocking and nettests/manipulation directories -respectively. -.SS Content Blocking Tests -.INDENT 0.0 -.INDENT 3.5 -.INDENT 0.0 -.IP \(bu 2 -\fI\%DNSConsistency\fP -.IP \(bu 2 -\fI\%HTTP Requests\fP -.IP \(bu 2 -\fI\%TCP Connect\fP -.UNINDENT -.UNINDENT -.UNINDENT -.SS Traffic Manipulation Tests -.INDENT 0.0 -.INDENT 3.5 -.INDENT 0.0 -.IP \(bu 2 -\fI\%HTTP Invalid Request Line:\fP -.IP \(bu 2 -\fI\%DNS Spoof\fP -.IP \(bu 2 -\fI\%HTTP Header Field Manipulation\fP -.IP \(bu 2 -\fI\%Traceroute\fP -.IP \(bu 2 -\fI\%HTTP Host\fP -.UNINDENT -.UNINDENT -.UNINDENT -.SS Other tests -.sp -We also have some other tests that are currently not fully supported or still -being experimented with. -.sp -You can find these in: -.INDENT 0.0 -.INDENT 3.5 -.INDENT 0.0 -.IP \(bu 2 -\fI\%ooni/nettests/experimental\fP -.UNINDENT -.UNINDENT -.UNINDENT -.sp -Tests that don\(aqt do a measurement but are useful for scanning can be found in: -.INDENT 0.0 -.INDENT 3.5 -.INDENT 0.0 -.IP \(bu 2 -\fI\%ooni/nettests/scanning\fP -.UNINDENT -.UNINDENT -.UNINDENT -.sp -Tests that involve running third party tools may be found in: -.INDENT 0.0 -.INDENT 3.5 -.INDENT 0.0 -.IP \(bu 2 -\fI\%ooni/nettests/third_party\fP -.UNINDENT -.UNINDENT -.UNINDENT -.SS Reports -.sp -The reports collected by ooniprobe are stored on -\fI\%https://ooni.torproject.org/reports/0.1/\fP \fBCC\fP / -.sp -Where \fBCC\fP is the two letter country code as specified by \fI\%ISO 31666\-2\fP\&. -.sp -For example the reports for Italy (\fBCC\fP is \fBit\fP) of the may be found in: -.sp -\fI\%https://ooni.torproject.org/reports/0.1/IT/\fP -.sp -This directory shall contain the various reports for the test using the -following convention: -.sp -\fBtestName\fP \- \fBdateInISO8601Format\fP \- \fBprobeASNumber\fP .yamloo -.sp -The date is expressed using \fI\%ISO 8601\fP -including seconds and with no \fB:\fP to delimit hours, minutes, days. -.sp -Like so: -.sp -\fBYEAR\fP \- \fBMONTH\fP \- \fBDAY\fP T \fBHOURS\fP \fBMINUTES\fP \fBSECONDS\fP Z -.sp -Look \fI\%here for the up to date list of ISO 8601 country codes\fP -.sp -The time is \fBalways\fP expressed in UTC. -.sp -If a collision is detected then an int (starting with 1) will get appended to -the test. -.sp -For example if two report that are created on the first of January 2012 at Noon -(UTC time) sharp from MIT (AS3) will be stored here: -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -https://ooni.torproject.org/reports/0.1/US/2012\-01\-01T120000Z_AS3.yamloo -https://ooni.torproject.org/reports/0.1/US/2012\-01\-01T120000Z_AS3.1.yamloo -.ft P -.fi -.UNINDENT -.UNINDENT -.sp -Note: it is highly unlikely that reports get created with the same exact -timestamp from the same exact ASN. If this does happen it could be index of -some malicious report poisoning attack in progress. -.SS Report format version changelog -.sp -In here shall go details about the major changes to the reporting format. -.SS version 0.1 -.sp -Initial format version. -.SS Writing OONI tests -.sp -The OONI testing API is heavily influenced and partially based on the python -\fBunittest\fP module and \fBtwisted.trial\fP\&. -.SS Test Cases -.sp -The atom of OONI Testing is called a Test Case. A test case class may contain -multiple Test Methods. -.INDENT 0.0 -.TP -.B class ooni.nettest.NetTestCase -This is the base of the OONI nettest universe. When you write a nettest -you will subclass this object. -.INDENT 7.0 -.IP \(bu 2 -inputs: can be set to a static set of inputs. All the tests (the methods -starting with the "test" prefix) will be run once per input. At every run -the _input_ attribute of the TestCase instance will be set to the value of -the current iteration over inputs. Any python iterable object can be set -to inputs. -.IP \(bu 2 -inputFile: attribute should be set to an array containing the command line -argument that should be used as the input file. Such array looks like -this: -.INDENT 2.0 -.INDENT 3.5 -\fB["commandlinearg", "c", "default value" "The description"]\fP -.UNINDENT -.UNINDENT -.sp -The second value of such arrray is the shorthand for the command line arg. -The user will then be able to specify inputs to the test via: -.INDENT 2.0 -.INDENT 3.5 -\fBooniprobe mytest.py \-\-commandlinearg path/to/file.txt\fP -.UNINDENT -.UNINDENT -.sp -or -.INDENT 2.0 -.INDENT 3.5 -\fBooniprobe mytest.py \-c path/to/file.txt\fP -.UNINDENT -.UNINDENT -.IP \(bu 2 -inputProcessor: should be set to a function that takes as argument a -filename and it will return the input to be passed to the test -instance. -.IP \(bu 2 -name: should be set to the name of the test. -.IP \(bu 2 -author: should contain the name and contact details for the test author. -The format for such string is as follows: -.INDENT 2.0 -.INDENT 3.5 -\fBThe Name <email(a)example.com>\fP -.UNINDENT -.UNINDENT -.IP \(bu 2 -version: is the version string of the test. -.IP \(bu 2 -requiresRoot: set to True if the test must be run as root. -.IP \(bu 2 -usageOptions: a subclass of twisted.python.usage.Options for processing of command line arguments -.IP \(bu 2 -localOptions: contains the parsed command line arguments. -.UNINDENT -.sp -Quirks: -Every class that is prefixed with test \fImust\fP return a twisted.internet.defer.Deferred. -.UNINDENT -.sp -If the test you plan to write is not listed on the \fI\%Tor OONI trac page\fP, you should -add it to the list and then add a description about it following the \fI\%Test -Template\fP -.sp -Tests are driven by inputs. For every input a new test instance is created, -internally the _setUp method is called that is defined inside of test -templates, then the setUp method that is overwritable by users. -.sp -Gotchas: -\fBnever\fP call reactor.start of reactor.stop inside of your test method and all -will be good. -.SS Inputs -.sp -Inputs are what is given as input to every iteration of the Test Case. -Iflyou have 100 inputs, then every test case will be run 100 times. -.sp -To configure a static set of inputs you should define the -\fBooni.nettest.NetTestCase\fP attribute \fBinputs\fP\&. The test will be -run \fBlen(inputs)\fP times. Any iterable object is a valid \fBinputs\fP -attribute. -.sp -If you would like to have inputs be determined from a user specified input -file, then you must set the \fBinputFile\fP attribute. This is an array that -specifies what command line option may be used to control this value. -.sp -By default the \fBinputProcessor\fP is set to read the file line by line and -strip newline characters. To change this behavior you must set the -\fBinputProcessor\fP attribute to a function that takes as argument a file -descriptor and yield the next item. The default \fBinputProcessor\fP looks like -this: -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -def lineByLine(filename): - fp = open(filename) - for x in fp.xreadlines(): - yield x.strip() - fp.close() -.ft P -.fi -.UNINDENT -.UNINDENT -.SS Setup and command line passing -.sp -Tests may define the \fIsetUp\fP method that will be called every time the -Test Case object is instantiated, in here you may place some common logic -to all your Test Methods that should be run before any testing occurs. -.sp -Command line arguments can be parsed thanks to the twisted -\fBtwisted.python.usage.UsageOptions\fP class. -.sp -You will have to subclass this and define the NetTestCase attribute -usageOptions to point to a subclass of this. -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -class UsageOptions(usage.Options): - optParameters = [[\(aqbackend\(aq, \(aqb\(aq, \(aqhttp://127.0.0.1:57001\(aq, - \(aqURL of the test backend to use\(aq] - ] - -class MyTestCase(nettest.NetTestCase): - usageOptions = UsageOptions - - inputFile = [\(aqfile\(aq, \(aqf\(aq, None, "Some foo file"] - requiredOptions = [\(aqbackend\(aq] - - def test_my_test(self): - self.localOptions[\(aqbackend\(aq] -.ft P -.fi -.UNINDENT -.UNINDENT -.sp -You will then be able to access the parsed command line arguments via the -class attribute localOptions. -.sp -The \fIrequiredOptions\fP attributes specifies an array of parameters that are -required for the test to run properly. -.sp -\fIinputFile\fP is a special class attribute that will be used for processing -of the inputFile. The filename that is read here will be given to the -\fBooni.nettest.NetTestCase.inputProcessor\fP method that will yield, by -default, one line of the file at a time. -.SS Test Methods -.sp -These shall be defined inside of your \fBooni.nettest.NetTestCase\fP -subclass. These will be class methods. -.sp -All class methods that are prefixed with test_ shall be run. Functions -that are relevant to your test should be all lowercase separated by -underscore. -.sp -To add data to the test report you may write directly to the report object -like so: -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -def test_my_function(): - result = do_something() - self.report[\(aqsomething\(aq] = result -.ft P -.fi -.UNINDENT -.UNINDENT -.sp -OONI will then handle the writing of the data to the final test report. -.sp -To access the current input you can use the \fBinput\fP attribute, for example: -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -def test_with_input(): - do_something_with_input(self.input) -.ft P -.fi -.UNINDENT -.UNINDENT -.sp -This will at each iteration over the list of inputs do something with the -input. -.SS Test Templates -.sp -Test templates assist you in writing tests. They already contain all the -common functionality that is useful to running a test of that type. They -also take care of writing the data they collect that is relevant to the -test run to the report file. -.sp -Currently implemented test templates are \fBooni.templates.scapyt\fP for -tests based on Scapy, \fBooni.templates.tcpt\fP for tests based on TCP, -\fBooni.templates.httpt\fP for tests based on HTTP, and -\fBooni.templates.dnst\fP for tests based on DNS. -.SS Scapy based tests -.sp -Scapy based tests will be a subclass of \fBooni.templates.scapyt.BaseScapyTest\fP\&. -.sp -It provides a wrapper around the scapy send and receive function that will -write the sent and received packets to the report with sanitization of the -src and destination IP addresses. -.sp -It has the same syntax as the Scapy sr function, except that it will -return a deferred. -.sp -To implement a simple ICMP ping based on this function you can do like so -(Taken from \fBnettest.examples.example_scapyt.ExampleICMPPingScapy\fP) -.INDENT 0.0 -.INDENT 3.5 +ooniprobe allows the user to select what test should be run and what backend +should be used for storing the test report and/or assisting them in the running +of the test. .sp -.nf -.ft C -from twisted.python import usage - -from scapy.all import IP, ICMP - -from ooni.templates import scapyt - -class UsageOptions(usage.Options): - optParameters = [[\(aqtarget\(aq, \(aqt\(aq, \(aq127.0.0.1\(aq, "Specify the target to ping"]] - -class ExampleICMPPingScapy(scapyt.BaseScapyTest): - name = "Example ICMP Ping Test" - - usageOptions = UsageOptions - - def test_icmp_ping(self): - def finished(packets): - print packets - answered, unanswered = packets - for snd, rcv in answered: - rcv.show() - - packets = IP(dst=self.localOptions[\(aqtarget\(aq])/ICMP() - d = self.sr(packets) - d.addCallback(finished) - return d -.ft P -.fi -.UNINDENT -.UNINDENT +ooniprobe tests are divided into two categories: \fBTraffic Manipulation\fP and +\fBContent Blocking\fP\&. .sp -The arguments taken by self.sr() are exactly the same as the scapy send and -receive function, the only difference is that instead of using the regular -scapy super socket it uses our twisted driven wrapper around it. +\fBTraffic Manipulation\fP tests aim to detect the presence of some sort of +tampering with the internet traffic between the probe and a remote test helper +backend. As such they usually require the selection of a oonib backend +component for running the test. .sp -Alternatively this test can also be written using the -\fBtwisted.defer.inlineCallbacks()\fP decorator, that makes it look more similar to -regular sequential code. -.INDENT 0.0 -.INDENT 3.5 +\fBContent Blocking\fP are aimed at enumerating the kind of content that is +blocked from the probes network point of view. As such they usually require to +have specified an input list for running the test. +.SH EXAMPLES .sp -.nf -.ft C -from twisted.python import usage -from twisted.internet import defer - -from scapy.all import IP, ICMP - -from ooni.templates import scapyt - -class UsageOptions(usage.Options): - optParameters = [[\(aqtarget\(aq, \(aqt\(aq, \(aq127.0.0.1\(aq, "Specify the target to ping"]] - -class ExampleICMPPingScapyYield(scapyt.BaseScapyTest): - name = "Example ICMP Ping Test" - - usageOptions = UsageOptions - - @defer.inlineCallbacks - def test_icmp_ping(self): - packets = IP(dst=self.localOptions[\(aqtarget\(aq])/ICMP() - answered, unanswered = yield self.sr(packets) - for snd, rcv in answered: - rcv.show() -.ft P -.fi -.UNINDENT -.UNINDENT -.SS Report Format +Run the web_connectivity test on \fI\%http://torproject.org\fP: .INDENT 0.0 .INDENT 3.5 -.sp -.nf -.ft C -########################################### -# OONI Probe Report for Example ICMP Ping Test test -# Thu Nov 22 18:20:43 2012 -########################################### -\-\-\- -{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1\-alpha, - start_time: 1353601243.0, test_name: Example ICMP Ping Test, test_version: 0.1} -\&... -\-\-\- -input: null -report: - answer_flags: [ipsrc] - answered_packets: - \- \- raw_packet: !!binary | - RQAAHAEdAAAuAbjKCAgICH8AAAEAAAAAAAAAAA== - summary: IP / ICMP 8.8.8.8 > 127.0.0.1 echo\-reply 0 - sent_packets: - \- \- raw_packet: !!binary | - RQAAHAABAABAAevPfwAAAQgICAgIAPf/AAAAAA== - summary: IP / ICMP 127.0.0.1 > 8.8.8.8 echo\-request 0 -test_name: test_icmp_ping -test_started: 1353604843.553605 -\&... -.ft P -.fi +\fBooniprobe web_connectivity \-\-url http://torproject.org/\fP .UNINDENT .UNINDENT -.SS TCP based tests -.sp -TCP based tests will subclass \fBooni.templates.tcpt.TCPTest\fP\&. .sp -This test template facilitates the sending of TCP payloads to the wire and -recording the response. +Run the http_invalid_request_line test to detect middleboxes: .INDENT 0.0 .INDENT 3.5 -.sp -.nf -.ft C -from twisted.internet.error import ConnectionRefusedError -from ooni.utils import log -from ooni.templates import tcpt - -class ExampleTCPT(tcpt.TCPTest): - def test_hello_world(self): - def got_response(response): - print "Got this data %s" % response - - def connection_failed(failure): - failure.trap(ConnectionRefusedError) - print "Connection Refused" - - self.address = "127.0.0.1" - self.port = 57002 - payload = "Hello World!\en\er" - d = self.sendPayload(payload) - d.addErrback(connection_failed) - d.addCallback(got_response) - return d -.ft P -.fi +\fBooniprobe http_invalid_request_line\fP .UNINDENT .UNINDENT .sp -The possible failures for a TCP connection are: -.sp -\fBtwisted.internet.error.NoRouteError\fP that corresponds to errno.ENETUNREACH -.sp -\fBtwisted.internet.error.ConnectionRefusedError\fP that corresponds to -errno.ECONNREFUSED -.sp -\fBtwisted.internet.error.TCPTimedOutError\fP that corresponds to errno.ETIMEDOUT -.SS Report format -.sp -The basic report of a TCP test looks like the following (this is an report -generated by running the above example against a TCP echo server). +Run the http_header_field_manipulation test to detect middleboxes: .INDENT 0.0 .INDENT 3.5 -.sp -.nf -.ft C -########################################### -# OONI Probe Report for Base TCP Test test -# Thu Nov 22 18:18:28 2012 -########################################### -\-\-\- -{probe_asn: null, probe_cc: null, probe_ip: 127.0.0.1, software_name: ooniprobe, software_version: 0.0.7.1\-alpha, - start_time: 1353601108.0, test_name: Base TCP Test, test_version: \(aq0.1\(aq} -\&... -\-\-\- -input: null -report: - errors: [] - received: ["Hello World!\en\er"] - sent: ["Hello World!\en\er"] -test_name: test_hello_world -test_started: 1353604708.705081 -\&... -.ft P -.fi +\fBooniprobe http_header_field_manipulation\fP .UNINDENT .UNINDENT .sp -TODO finish this with more details -.SS HTTP based tests -.sp -HTTP based tests will be a subclass of \fBooni.templates.httpt.HTTPTest\fP\&. -.sp -It provides methods \fBooni.templates.httpt.HTTPTest.processResponseBody()\fP and -\fBooni.templates.httpt.HTTPTest.processResponseHeaders()\fP for interacting with the -response body and headers respectively. -.sp -For example, to implement a HTTP test that returns the sha256 hash of the -response body (based on \fBnettests.examples.example_httpt\fP): -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -from ooni.utils import log -from ooni.templates import httpt -from hashlib import sha256 - -class SHA256HTTPBodyTest(httpt.HTTPTest): - name = "ChecksumHTTPBodyTest" - author = "Aaron Gibson" - version = 0.1 - - inputFile = [\(aqurl file\(aq, \(aqf\(aq, None, - \(aqList of URLS to perform GET requests to\(aq] - requiredOptions = [\(aqurl file\(aq] - - def test_http(self): - if self.input: - url = self.input - return self.doRequest(url) - else: - raise Exception("No input specified") - - def processResponseBody(self, body): - body_sha256sum = sha256(body).hexdigest() - self.report[\(aqchecksum\(aq] = body_sha256sum -.ft P -.fi -.UNINDENT -.UNINDENT -.SS Report format +List all the available tests: .INDENT 0.0 .INDENT 3.5 -.sp -.nf -.ft C -########################################### -# OONI Probe Report for ChecksumHTTPBodyTest test -# Thu Dec 6 17:31:57 2012 -########################################### -\-\-\- -options: - collector: null - help: 0 - logfile: null - pcapfile: null - reportfile: null - resume: 0 - subargs: [\-f, hosts] - test: nettests/examples/example_http_checksum.py -probe_asn: null -probe_cc: null -probe_ip: 127.0.0.1 -software_name: ooniprobe -software_version: 0.0.7.1\-alpha -start_time: 1354786317.0 -test_name: ChecksumHTTPBodyTest -test_version: 0.1 -\&... -\-\-\- -input: http://www.google.com -report: - agent: agent - checksum: d630fa2efd547d3656e349e96ff7af5496889dad959e8e29212af1ff843e7aa1 - requests: - \- request: - body: null - headers: - \- \- User\-Agent - \- \- [Opera/9.00 (Windows NT 5.1; U; en), \(aqOpera 9.0, Windows XP\(aq] - method: GET - url: http://www.google.com - response: - body: \(aq<!doctype html><html ... snip ... </html>\(aq - code: 200 - headers: - \- \- X\-XSS\-Protection - \- [1; mode=block] - \- \- Set\-Cookie - \- [\(aqPREF=ID=fada4216eb3684f9:FF=0:TM=1354800717:LM=1354800717:S=IT\-2GCkNAocyXlVa; - expires=Sat, 06\-Dec\-2014 13:31:57 GMT; path=/; domain=.google.com\(aq, \(aqNID=66=KWaLbNQumuGuYf0HrWlGm54u9l\-DKJwhFCMQXfhQPZM\-qniRhmF6QRGXUKXb_8CIUuCOHnyoC5oAX5jWNrsfk\-LLJLW530UiMp6hemTtDMh_e6GSiEB4GR3yOP_E0TCN; - expires=Fri, 07\-Jun\-2013 13:31:57 GMT; path=/; domain=.google.com; HttpOnly\(aq] - \- \- Expires - \- [\(aq\-1\(aq] - \- \- Server - \- [gws] - \- \- Connection - \- [close] - \- \- Cache\-Control - \- [\(aqprivate, max\-age=0\(aq] - \- \- Date - \- [\(aqThu, 06 Dec 2012 13:31:57 GMT\(aq] - \- \- P3P - \- [\(aqCP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 - for more info."\(aq] - \- \- Content\-Type - \- [text/html; charset=UTF\-8] - \- \- X\-Frame\-Options - \- [SAMEORIGIN] - socksproxy: null -test_name: test_http -test_runtime: 0.08298492431640625 -test_started: 1354800717.478403 -\&... -.ft P -.fi +\fBooniprobe \-s\fP .UNINDENT .UNINDENT -.SS DNS based tests -.sp -DNS based tests will be a subclass of \fBooni.templates.dnst.DNSTest\fP\&. -.sp -It provides methods \fBooni.templates.dnst.DNSTest.performPTRLookup()\fP -and \fBooni.templates.dnst.DNSTest.performALookup()\fP .sp -For example (taken from \fBnettests.examples.example_dnst\fP): +Start the web user interface: .INDENT 0.0 .INDENT 3.5 -.sp -.nf -.ft C -from ooni.templates.dnst import DNSTest - -class ExampleDNSTest(DNSTest): - def test_a_lookup(self): - def gotResult(result): - # Result is an array containing all the A record lookup results - print result - - d = self.performALookup(\(aqtorproject.org\(aq, (\(aq8.8.8.8\(aq, 53)) - d.addCallback(gotResult) - return d -.ft P -.fi +\fBooniprobe \-w\fP .UNINDENT .UNINDENT -.SS Report format -.INDENT 0.0 -.INDENT 3.5 -.sp -.nf -.ft C -########################################### -# OONI Probe Report for Base DNS Test test -# Thu Dec 6 17:42:51 2012 -########################################### -\-\-\- -options: - collector: null - help: 0 - logfile: null - pcapfile: null - reportfile: null - resume: 0 - subargs: [] - test: nettests/examples/example_dnst.py -probe_asn: null -probe_cc: null -probe_ip: 127.0.0.1 -software_name: ooniprobe -software_version: 0.0.7.1\-alpha -start_time: 1354786971.0 -test_name: Base DNS Test -test_version: 0.1 -\&... -\-\-\- -input: null -report: - queries: - \- addrs: [82.195.75.101, 86.59.30.40, 38.229.72.14, 38.229.72.16] - answers: - \- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=82.195.75.101 - ttl=782>] - \- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=86.59.30.40 - ttl=782>] - \- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.14 - ttl=782>] - \- [<RR name=torproject.org type=A class=IN ttl=782s auth=False>, <A address=38.229.72.16 - ttl=782>] - query: \(aq[Query(\(aq\(aqtorproject.org\(aq\(aq, 1, 1)]\(aq - query_type: A - resolver: [8.8.8.8, 53] -test_name: test_a_lookup -test_runtime: 0.028924942016601562 -test_started: 1354801371.980114 -\&... -.ft P -.fi -.UNINDENT -.UNINDENT -.sp -For a more complex example, see: \fBnettests.blocking.dnsconsistency\fP - -.SH GLOSSARY -.sp -Here we will summarize some of the jargon that is unique to OONI. -.sp -\fBTest Case\fP: a set of measurements performed on a to be tested network that -are logically grouped together -.sp -\fBReport\fP: is the output of a test run containing all the information that is -require for a researcher to assess what is the output of a test. -.sp -\fBYamlooni\fP: The format we use for Reports, that is based on YAML. -.sp -\fBInput\fP: What is given as input to a TestCase to perform a measurement. .SH AUTHOR The Tor Project .SH COPYRIGHT 2016, The Tor Project +.\" Generated by docutils manpage writer. . diff --git a/data/oonireport.1 b/data/oonireport.1 index 8a91009..5c611ff 100644 --- a/data/oonireport.1 +++ b/data/oonireport.1 @@ -1,8 +1,8 @@ .\" Man page generated from reStructuredText. . -.TH "oonireport" "1" "April 26, 2016" "0.1.0" "oonireport" +.TH "OONIREPORT" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" .SH NAME -oonireport - tool for managing ooniprobe reports. +oonireport \- upload ooniprobe measurements . .nr rst2man-indent-level 0 . @@ -30,71 +30,86 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. - .SH SYNOPSIS -.B oonireport -.RB [ --version ] -.RB [ --help ] -.RB [ \-c -.IR collector ] -.I "status | upload" -.I [ report file ] - +.sp +\fBoonireport\fP [\fIoptions\fP] upload | status [\fIpath to report\fP] .SH DESCRIPTION .sp -oonireport is a tool for viewing which ooniprobe reports have not been -submitted to a collector and upload them. - +\fBoonireport\fP is a tool for viewing which ooniprobe +reports have not been submitted to a collector and upload them. +.sp +\fBooniprobe \-i /usr/share/ooni/decks\-available/web.yaml\fP .SH OPTIONS - +.INDENT 0.0 .TP -.BR \-\^h " or " \-\-help -Display this help and exit. +.B \-d\fP,\fB \-\-default\-collector +Upload the reports to the default collector that is looked up +with the canonical bouncer. .TP -.BR \-\^c " or " \-\-collector -Specify the address of the collector to use for uploading reports. +.B \-f\fP,\fB \-\-configfile +Specify the configuration file to use. .TP -.BR upload -If no argument is specified all the reports that have not been submitted will -be submitted to either the collector specified as an option (\-c) or to the -collector that was previously used. -If a report file is specified then only that report file will be uploaded +.B \-c\fP,\fB \-\-collector +Specify the collector to upload the result to. +.TP +.B \-b\fP,\fB \-\-bouncer +Specify the bouncer to query for a collector. +.sp +\-\-version +.INDENT 7.0 .TP -.BR status -Outputs all of the reports that are either "not uploaded", "incomplete" or -"pending". +.B \-\-help +Display this help and exit. +.UNINDENT +.UNINDENT +.SH COMMANDS +.sp +\fBupload\fP +.sp +If no argument is specified all the reports that have not been +submitted will be submitted to either the collector specified as +an option (\-c) or to the collector that was previously used. +.sp +If a report file is specified then only that report file will be +uploaded +.sp +\fBstatus\fP +.sp +Outputs all of the reports that are either "not uploaded", +"incomplete" or "pending". +.sp The possible statuses are: - -.BR not-created +.sp +not\-created .sp If it has not been created, because the user specified to not use a collector. - -.BR creation-failed +.sp +creation\-failed .sp If we attempted to create the report, but failed to do so either because the collector does not accept our report or because it was unreachable at the time. - -.BR created .sp -If the report has been successfully created, but has not yet been closed. - -.BR incomplete +created .sp -If the report has been created, but we have failed to update the report -with some data. - -.SH OONIREPORT +If the report has been successfully created, but has not yet been +closed. .sp -This tool is used to upload reports that are stored on the users filesystem and -have not been submitted to a collector. This can be either because the -collector was unreachable when we attempted to submit the report or because the policy -of the collector did not support the specified test. +incomplete +.sp +If the report has been created, but we have failed to update the +report with some data. +.SH OONIREPORT .sp - +This tool is used to upload reports that are stored on the users +filesystem and have not been submitted to a collector. This can be +either because the collector was unreachable when we attempted to +submit the report or because the policy of the collector did not +support the specified test. .SH AUTHOR The Tor Project .SH COPYRIGHT 2016, The Tor Project +.\" Generated by docutils manpage writer. . diff --git a/data/ooniresources.1 b/data/ooniresources.1 index 1e16822..16b823d 100644 --- a/data/ooniresources.1 +++ b/data/ooniresources.1 @@ -1,8 +1,8 @@ .\" Man page generated from reStructuredText. . -.TH "ooniresources" "1" "April 26, 2016" "0.2.0" "ooniresources" +.TH "OONIRESOURCES" "1" "October 14, 2016" "2.0.0" "OONI: Open Observatory of Network Interference" .SH NAME -ooniresources - tool for updating resources for OONI tests +ooniresources \- DEPRECATED ooni tool . .nr rst2man-indent-level 0 . @@ -30,45 +30,18 @@ level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. - .SH SYNOPSIS -.B ooniresources -.RB [ --version ] -.RB [ --help ] - -.SH DESCRIPTION .sp -ooniresources will update the resources that are needed for the functioning of -ooniprobe and related tools. - -By invoking this command with no arguments the inputs and GeoIP data files -shall be updated. - -.SH OPTIONS - -.TP -.BR \-\^h " or " \-\-help -Display this help and exit. -.TP -.BR \-\-update\-inputs -(DEPRECATED) This will update the inputs that are used for generating test decks with -oonideckgen. -.TP -.BR \-\-update\-geoip -(DEPRECATED) This will update the maxmind geoip databases that are used for detecting the -probes country and ASN. -.TP - -.SH OONIRESOURCES +\fBooniresources\fP +.SH DESCRIPTION .sp -This tools will update the resources needed for running ooniprobe and -oonideckgen. -You don't need to run it very often, but it's just sufficient to run it once -every month or so as the data usually does not update or change that often. +\fBooniresources\fP, has now been superseded by a system that +automatically manages downloads of new resources. .sp - +There is no reason to use this anymore. .SH AUTHOR The Tor Project .SH COPYRIGHT 2016, The Tor Project +.\" Generated by docutils manpage writer. .

1 0

[ooni-probe/master] Use a non mutable default argument (as noted by @darkk)
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 84c710704d49492f23998bfaa88c34f5fb6ec75d Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 12:57:48 2016 +0200 Use a non mutable default argument (as noted by @darkk) * Log some warning if we notice that annotations is not a dictionary and clear it --- ooni/nettest.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/ooni/nettest.py b/ooni/nettest.py index b81678b..150858c 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -169,10 +169,12 @@ class NetTestLoader(object): requiresTor = False def __init__(self, options, test_file=None, test_string=None, - annotations={}): + annotations=None): self.options = options - # Ensure annotations are always dicts + if annotations is None: + annotations = {} if not isinstance(annotations, dict): + log.warn("BUG: Annotations is not a dictionary. Resetting it.") annotations = {} self.annotations = annotations self.annotations['platform'] = self.annotations.get('platform',

1 0

[tor/master] Whitespace cleaning
by nickm＠torproject.org 14 Oct '16

14 Oct '16

commit 55c468c5211d5b74acb677767f14d91cd0304771 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Oct 14 10:40:31 2016 -0400 Whitespace cleaning --- src/or/rendservice.c | 4 ++-- src/or/routerparse.c | 1 - src/test/test_config.c | 1 - src/test/test_controller.c | 1 - src/test/test_util.c | 1 - 5 files changed, 2 insertions(+), 6 deletions(-) diff --git a/src/or/rendservice.c b/src/or/rendservice.c index a2d7a37..3afe880 100644 --- a/src/or/rendservice.c +++ b/src/or/rendservice.c @@ -382,8 +382,8 @@ rend_service_parse_port_config(const char *string, const char *sep, ret = port_cfg_line_extract_addrport(addrport_element, &addrport, &is_unix, &rest); if (ret < 0) { - tor_asprintf(&err_msg, "Couldn't process address <%s> from hidden service " - "configuration", addrport_element); + tor_asprintf(&err_msg, "Couldn't process address <%s> from hidden " + "service configuration", addrport_element); goto err; } if (is_unix) { diff --git a/src/or/routerparse.c b/src/or/routerparse.c index db3d004..df9b768 100644 --- a/src/or/routerparse.c +++ b/src/or/routerparse.c @@ -2910,7 +2910,6 @@ routerstatus_parse_entry_from_string(memarea_t *area, rs->is_flagged_running = 1; /* Starting with consensus method 4. */ if (consensus_method >= MIN_METHOD_FOR_EXCLUDING_INVALID_NODES) rs->is_valid = 1; - } int found_protocol_list = 0; if ((tok = find_opt_by_keyword(tokens, K_PROTO))) { diff --git a/src/test/test_config.c b/src/test/test_config.c index 4efefa8..384bff4 100644 --- a/src/test/test_config.c +++ b/src/test/test_config.c @@ -3848,7 +3848,6 @@ test_config_port_cfg_line_extract_addrport(void *arg) tor_free(a); } - static config_line_t * mock_config_line(const char *key, const char *val) { diff --git a/src/test/test_controller.c b/src/test/test_controller.c index 4ae35c8..f19c846 100644 --- a/src/test/test_controller.c +++ b/src/test/test_controller.c @@ -200,7 +200,6 @@ test_rend_service_parse_port_config(void *arg) "in hidden service port configuration."); tor_free(err_msg); - done: rend_service_port_config_free(cfg); tor_free(err_msg); diff --git a/src/test/test_util.c b/src/test/test_util.c index a548b2a..7276c0c 100644 --- a/src/test/test_util.c +++ b/src/test/test_util.c @@ -1671,7 +1671,6 @@ test_util_config_line_crlf(void *arg) tor_free(k); tor_free(v); } - #ifndef _WIN32 static void test_util_expand_filename(void *arg)

1 0

[tor/master] adding instructions how to include new test files
by nickm＠torproject.org 14 Oct '16

14 Oct '16

commit 1a2aa4e3f8628e5d1343414e0034d2136ad26672 Author: Chelsea H. Komlo <chelsea.komlo(a)gmail.com> Date: Fri Oct 14 10:27:21 2016 -0400 adding instructions how to include new test files --- doc/HACKING/WritingTests.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/doc/HACKING/WritingTests.md b/doc/HACKING/WritingTests.md index 7bcadc6..de80bbd 100644 --- a/doc/HACKING/WritingTests.md +++ b/doc/HACKING/WritingTests.md @@ -152,6 +152,11 @@ or create a new C file there. Each test is a single function that must be indexed in the table at the end of the file. We use the label "done:" as a cleanup point for all test functions. +If you have created a new test file, you will need to: +1. Add the new test file to include.am +2. In `test.h`, include the new test cases (testcase_t) +3. In `test.c`, add the new test cases to testgroup_t testgroups + (Make sure you read `tinytest-manual.md` before proceeding.) I use the term "unit test" and "regression tests" very sloppily here.

1 0

[tor/master] Remove current_consensus macro.
by nickm＠torproject.org 14 Oct '16

14 Oct '16

commit 2196c7ad6496ae699667af65eb289950f8fe5dbb Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Sep 20 10:52:20 2016 -0400 Remove current_consensus macro. It's a macro that calls down to a function whose behavior has been getting progresively more complicated.... but we named it as if it were a variable. That's not so smart. So, replace it with a function call to a function that was just doing "return current_consensus". Fixes bug 20176. --- src/or/networkstatus.c | 47 +++++++++++++++++++++-------------------------- 1 file changed, 21 insertions(+), 26 deletions(-) diff --git a/src/or/networkstatus.c b/src/or/networkstatus.c index 28f5b42..0a32012 100644 --- a/src/or/networkstatus.c +++ b/src/or/networkstatus.c @@ -42,14 +42,6 @@ static strmap_t *named_server_map = NULL; * as unnamed for some server in the consensus. */ static strmap_t *unnamed_server_map = NULL; -/** Most recently received and validated v3 consensus network status, - * of whichever type we are using for our own circuits. This will be the same - * as one of current_ns_consensus or current_md_consensus. - */ -#define current_consensus \ - (we_use_microdescriptors_for_circuits(get_options()) ? \ - current_md_consensus : current_ns_consensus) - /** Most recently received and validated v3 "ns"-flavored consensus network * status. */ static networkstatus_t *current_ns_consensus = NULL; @@ -130,7 +122,7 @@ static void update_consensus_bootstrap_multiple_downloads( void networkstatus_reset_warnings(void) { - if (current_consensus) { + if (networkstatus_get_latest_consensus()) { SMARTLIST_FOREACH(nodelist_get_list(), node_t *, node, node->name_lookup_warned = 0); } @@ -206,7 +198,7 @@ router_reload_consensus_networkstatus(void) tor_free(filename); } - if (!current_consensus) { + if (!networkstatus_get_latest_consensus()) { if (!named_server_map) named_server_map = strmap_new(); if (!unnamed_server_map) @@ -647,7 +639,7 @@ router_get_mutable_consensus_status_by_descriptor_digest,( const char *digest)) { if (!consensus) - consensus = current_consensus; + consensus = networkstatus_get_latest_consensus(); if (!consensus) return NULL; if (!consensus->desc_digest_map) { @@ -728,9 +720,9 @@ router_get_dl_status_by_descriptor_digest,(const char *d)) routerstatus_t * router_get_mutable_consensus_status_by_id(const char *digest) { - if (!current_consensus) + if (!networkstatus_get_latest_consensus()) return NULL; - return smartlist_bsearch(current_consensus->routerstatus_list, digest, + return smartlist_bsearch(networkstatus_get_latest_consensus()->routerstatus_list, digest, compare_digest_to_routerstatus_entry); } @@ -1280,7 +1272,10 @@ networkstatus_get_dl_status_by_flavor_running,(consensus_flavor_t flavor)) MOCK_IMPL(networkstatus_t *, networkstatus_get_latest_consensus,(void)) { - return current_consensus; + if (we_use_microdescriptors_for_circuits(get_options())) + return current_md_consensus; + else + return current_ns_consensus; } /** Return the latest consensus we have whose flavor matches <b>f</b>, or NULL @@ -1303,10 +1298,10 @@ networkstatus_get_latest_consensus_by_flavor,(consensus_flavor_t f)) MOCK_IMPL(networkstatus_t *, networkstatus_get_live_consensus,(time_t now)) { - if (current_consensus && - current_consensus->valid_after <= now && - now <= current_consensus->valid_until) - return current_consensus; + if (networkstatus_get_latest_consensus() && + networkstatus_get_latest_consensus()->valid_after <= now && + now <= networkstatus_get_latest_consensus()->valid_until) + return networkstatus_get_latest_consensus(); else return NULL; } @@ -1736,13 +1731,13 @@ networkstatus_set_current_consensus(const char *consensus, const int is_usable_flavor = flav == usable_consensus_flavor(); if (is_usable_flavor) { - notify_control_networkstatus_changed(current_consensus, c); + notify_control_networkstatus_changed(networkstatus_get_latest_consensus(), c); } if (flav == FLAV_NS) { if (current_ns_consensus) { networkstatus_copy_old_consensus_info(c, current_ns_consensus); networkstatus_vote_free(current_ns_consensus); - /* Defensive programming : we should set current_consensus very soon, + /* Defensive programming : we should set networkstatus_get_latest_consensus() very soon, * but we're about to call some stuff in the meantime, and leaving this * dangling pointer around has proven to be trouble. */ current_ns_consensus = NULL; @@ -1804,7 +1799,7 @@ networkstatus_set_current_consensus(const char *consensus, connection_or_update_token_buckets(get_connection_array(), options); circuit_build_times_new_consensus_params(get_circuit_build_times_mutable(), - current_consensus); + networkstatus_get_latest_consensus()); } /* Reset the failure count only if this consensus is actually valid. */ @@ -1954,14 +1949,14 @@ routers_update_all_from_networkstatus(time_t now, int dir_version) static void routerstatus_list_update_named_server_map(void) { - if (!current_consensus) + if (!networkstatus_get_latest_consensus()) return; strmap_free(named_server_map, tor_free_); named_server_map = strmap_new(); strmap_free(unnamed_server_map, NULL); unnamed_server_map = strmap_new(); - SMARTLIST_FOREACH_BEGIN(current_consensus->routerstatus_list, + SMARTLIST_FOREACH_BEGIN(networkstatus_get_latest_consensus()->routerstatus_list, const routerstatus_t *, rs) { if (rs->is_named) { strmap_set_lc(named_server_map, rs->nickname, @@ -1982,7 +1977,7 @@ routers_update_status_from_consensus_networkstatus(smartlist_t *routers, { const or_options_t *options = get_options(); int authdir = authdir_mode_v3(options); - networkstatus_t *ns = current_consensus; + networkstatus_t *ns = networkstatus_get_latest_consensus(); if (!ns || !smartlist_len(ns->routerstatus_list)) return; @@ -2298,14 +2293,14 @@ getinfo_helper_networkstatus(control_connection_t *conn, const routerstatus_t *status; (void) conn; - if (!current_consensus) { + if (!networkstatus_get_latest_consensus()) { *answer = tor_strdup(""); return 0; } if (!strcmp(question, "ns/all")) { smartlist_t *statuses = smartlist_new(); - SMARTLIST_FOREACH(current_consensus->routerstatus_list, + SMARTLIST_FOREACH(networkstatus_get_latest_consensus()->routerstatus_list, const routerstatus_t *, rs, { smartlist_add(statuses, networkstatus_getinfo_helper_single(rs));

1 0

[tor/master] Simplify a few functions that dont need to call get_latest_consensus() so much
by nickm＠torproject.org 14 Oct '16

14 Oct '16

commit cff1274b0629b61d9d33f4927aa3c2cb60be6842 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 21 15:20:27 2016 -0400 Simplify a few functions that dont need to call get_latest_consensus() so much --- src/or/networkstatus.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/src/or/networkstatus.c b/src/or/networkstatus.c index c938885..f888816 100644 --- a/src/or/networkstatus.c +++ b/src/or/networkstatus.c @@ -122,10 +122,8 @@ static void update_consensus_bootstrap_multiple_downloads( void networkstatus_reset_warnings(void) { - if (networkstatus_get_latest_consensus()) { - SMARTLIST_FOREACH(nodelist_get_list(), node_t *, node, - node->name_lookup_warned = 0); - } + SMARTLIST_FOREACH(nodelist_get_list(), node_t *, node, + node->name_lookup_warned = 0); have_warned_about_old_version = 0; have_warned_about_new_version = 0; @@ -720,10 +718,10 @@ router_get_dl_status_by_descriptor_digest,(const char *d)) routerstatus_t * router_get_mutable_consensus_status_by_id(const char *digest) { - if (!networkstatus_get_latest_consensus()) + const networkstatus_t *ns = networkstatus_get_latest_consensus(); + if (!ns) return NULL; - const smartlist_t *rslist; - rslist = networkstatus_get_latest_consensus()->routerstatus_list; + smartlist_t *rslist = ns->routerstatus_list; return smartlist_bsearch(rslist, digest, compare_digest_to_routerstatus_entry); } @@ -1788,7 +1786,7 @@ networkstatus_set_current_consensus(const char *consensus, /* Update ewma and adjust policy if needed; first cache the old value */ old_ewma_enabled = cell_ewma_enabled(); /* Change the cell EWMA settings */ - cell_ewma_set_scale_factor(options, networkstatus_get_latest_consensus()); + cell_ewma_set_scale_factor(options, c); /* If we just enabled ewma, set the cmux policy on all active channels */ if (cell_ewma_enabled() && !old_ewma_enabled) { channel_set_cmux_policy_everywhere(&ewma_policy); @@ -1801,8 +1799,8 @@ networkstatus_set_current_consensus(const char *consensus, * current consensus really alter our view of any OR's rate limits? */ connection_or_update_token_buckets(get_connection_array(), options); - circuit_build_times_new_consensus_params(get_circuit_build_times_mutable(), - networkstatus_get_latest_consensus()); + circuit_build_times_new_consensus_params( + get_circuit_build_times_mutable(), c); } /* Reset the failure count only if this consensus is actually valid. */ @@ -1952,17 +1950,16 @@ routers_update_all_from_networkstatus(time_t now, int dir_version) static void routerstatus_list_update_named_server_map(void) { - if (!networkstatus_get_latest_consensus()) + networkstatus_t *ns = networkstatus_get_latest_consensus(); + if (!ns) return; strmap_free(named_server_map, tor_free_); named_server_map = strmap_new(); strmap_free(unnamed_server_map, NULL); unnamed_server_map = strmap_new(); - const smartlist_t *rslist; - rslist = networkstatus_get_latest_consensus()->routerstatus_list; - SMARTLIST_FOREACH_BEGIN(rslist, - const routerstatus_t *, rs) { + smartlist_t *rslist = ns->routerstatus_list; + SMARTLIST_FOREACH_BEGIN(rslist, const routerstatus_t *, rs) { if (rs->is_named) { strmap_set_lc(named_server_map, rs->nickname, tor_memdup(rs->identity_digest, DIGEST_LEN));

1 0

[tor/master] fix wide lines
by nickm＠torproject.org 14 Oct '16

14 Oct '16

commit 266d7733f8f1978ff668698b3d33dc42a865237d Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Sep 21 15:15:17 2016 -0400 fix wide lines --- src/or/networkstatus.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/src/or/networkstatus.c b/src/or/networkstatus.c index 0a32012..c938885 100644 --- a/src/or/networkstatus.c +++ b/src/or/networkstatus.c @@ -722,7 +722,9 @@ router_get_mutable_consensus_status_by_id(const char *digest) { if (!networkstatus_get_latest_consensus()) return NULL; - return smartlist_bsearch(networkstatus_get_latest_consensus()->routerstatus_list, digest, + const smartlist_t *rslist; + rslist = networkstatus_get_latest_consensus()->routerstatus_list; + return smartlist_bsearch(rslist, digest, compare_digest_to_routerstatus_entry); } @@ -1731,13 +1733,14 @@ networkstatus_set_current_consensus(const char *consensus, const int is_usable_flavor = flav == usable_consensus_flavor(); if (is_usable_flavor) { - notify_control_networkstatus_changed(networkstatus_get_latest_consensus(), c); + notify_control_networkstatus_changed( + networkstatus_get_latest_consensus(), c); } if (flav == FLAV_NS) { if (current_ns_consensus) { networkstatus_copy_old_consensus_info(c, current_ns_consensus); networkstatus_vote_free(current_ns_consensus); - /* Defensive programming : we should set networkstatus_get_latest_consensus() very soon, + /* Defensive programming : we should set current_ns_consensus very soon * but we're about to call some stuff in the meantime, and leaving this * dangling pointer around has proven to be trouble. */ current_ns_consensus = NULL; @@ -1956,7 +1959,9 @@ routerstatus_list_update_named_server_map(void) named_server_map = strmap_new(); strmap_free(unnamed_server_map, NULL); unnamed_server_map = strmap_new(); - SMARTLIST_FOREACH_BEGIN(networkstatus_get_latest_consensus()->routerstatus_list, + const smartlist_t *rslist; + rslist = networkstatus_get_latest_consensus()->routerstatus_list; + SMARTLIST_FOREACH_BEGIN(rslist, const routerstatus_t *, rs) { if (rs->is_named) { strmap_set_lc(named_server_map, rs->nickname,

1 0