October 2016 - tor-commits - lists.torproject.org

[ooni-probe/master] Add scheduler tests to ensure that lock is properly managed
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 2dab7a2bcaf9930ffda5a11a2b0569ca12922b18 Author: Leonid Evdokimov <leon(a)darkk.net.ru> Date: Thu Sep 29 18:52:43 2016 +0300 Add scheduler tests to ensure that lock is properly managed --- ooni/tests/test_scheduler.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/ooni/tests/test_scheduler.py b/ooni/tests/test_scheduler.py index 1350dde..a9501c0 100644 --- a/ooni/tests/test_scheduler.py +++ b/ooni/tests/test_scheduler.py @@ -20,6 +20,7 @@ class TestScheduler(unittest.TestCase): self.assertEqual(scheduled_task.should_run, True) self.assertFailure(scheduled_task.run(), NotImplementedError) self.assertEqual(scheduled_task.should_run, True) + self.assertFalse(os.path.islink(os.path.join(scheduler_directory, identifier + '.lock'))) shutil.rmtree(scheduler_directory) @defer.inlineCallbacks @@ -31,8 +32,9 @@ class TestScheduler(unittest.TestCase): scheduler_directory = tempfile.mkdtemp() spam_path = os.path.join(scheduler_directory, 'spam.txt') class DummyST(ScheduledTask): - def task(self): - with open(spam_path, 'w') as out_file: + def task(subself): + self.assertTrue(os.path.islink(os.path.join(scheduler_directory, subself.identifier + '.lock'))) + with open(spam_path, 'a') as out_file: out_file.write("I ran\n") schedule = "@daily" @@ -59,6 +61,8 @@ class TestScheduler(unittest.TestCase): lock = FileSystemlockAndMutex(lock_path) + os.symlink(str(2**30), lock_path) # that's non-existend PID for sure + lock_count = 100 unlock_count = 0 dl = []

1 0

[ooni-probe/master] Port MK's CONTRIBUTING.md to ooni-probe (#607)
by art＠torproject.org 14 Oct '16

14 Oct '16

commit a34e9525766f8715f09f1e1fe866cb58f506c017 Author: Simone Basso <bassosimone(a)gmail.com> Date: Fri Oct 7 16:10:11 2016 +0200 Port MK's CONTRIBUTING.md to ooni-probe (#607) * Port MK's CONTRIBUTING.md to ooni-probe * fix indentation * grammar * more grammar * Fix command to run ooni tests --- CONTRIBUTING.md | 87 ++++++++++++++++ HACKING | 286 --------------------------------------------------- docs/coding-style.md | 193 ++++++++++++++++++++++++++++++++++ 3 files changed, 280 insertions(+), 286 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..a671ac3 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,87 @@ +# Contributing to ooni-probe + +ooni-probe is a Free and Open Source software project that welcomes +new contributions. + +## How to contribute code + +For guidelines on what coding style we follow refer to +the [coding style document](/doc/coding-style.md). + +### 1. Fork + +You should first create a fork of ooni-probe by clicking on the fork button +and cloning your fork with: + +``` +git clone git@github.com:username/ooni-probe.git +cd ooni-probe +git remote add upstream https://github.com/TheTorProject/ooni-probe.git +``` + +### 2. Branch + +Branches should be created from the `master` branch with an accurately +labelled name. + +If you want to work on a new feature you could name your branch +`feature/myfeature`, if you are fixing a bug you could name it `bug/1234`. + +Create your branch from `master` like so: + +``` +git checkout -b mybranch +``` + +Then you can start hacking + +### 3. Commit + +Make sure git knows your username and email address with the following: + +``` +git config --global user.name "Jane Doe" +git config --global user.email "jane(a)example.com" +``` + +Try to keep the commits made on the branch small (a good branch should only +address one specific issue). + +### 4. Test + +Make sure all the existing unittests for ooni-probe are passing. + +``` +trial ooni +``` + +If you add extra code or modify existing code, be sure that it is covered +by the existing unittests and if not write them. In general, it would be good +for pull requests not to reduce the current code coverage of the project. + +If you are submitting a branch fixing a bug, you should also be submitting +a unittest that is capable of reproducing the bug you are attempting to fix. + +### 5. Open a Pull Request + +You can then push your feature branch to your remote and open a pull request. + +## Code Review process + +Small pull requests should be tagged as `hotfix` and can be self +merged. All other pull request should be reviewed by another core +developer who will take responsibility of the merge. The repository +should be configured such that it is not possible to merge into +`master` (or `stable`) if unit tests are not passing. + +If the diff is small, squash merge is preferred, otherwise preserve +the history. In general it is a good idea to keep your branches +in sync with master and rebase them from time to time before the +review has happenned. However if the review has already begun it's +better to merge and resolve the conflicts locally and push the merge +commit, to allow the reviewer to see how the conflicts were resolved. + +Before a release, review the code base, fixing simple bugs directly +and opening issues to describe more complex required fixes and +refactoring opportunities. Also during such review make sure that +the documentation is up to date with the code. diff --git a/HACKING b/HACKING deleted file mode 100644 index baa5bb0..0000000 --- a/HACKING +++ /dev/null @@ -1,286 +0,0 @@ -Hacking on OONI -*************** - -This documents gives guidelines on where to start looking -for helping out in developing OONI and what guidelines you -should follow when writing code. - -We try to follow the general python best practices and styling -guides as specified in PEP. - - Beautiful is better than ugly. - Explicit is better than implicit. - Simple is better than complex. - Complex is better than complicated. - Flat is better than nested. - Sparse is better than dense. - Readability counts. - Special cases aren't special enough to break the rules. - Although practicality beats purity. - Errors should never pass silently. - Unless explicitly silenced. - In the face of ambiguity, refuse the temptation to guess. - There should be one-- and preferably only one --obvious way to do it. - Although that way may not be obvious at first unless you're Dutch. - Now is better than never. - Although never is often better than *right* now. - If the implementation is hard to explain, it's a bad idea. - If the implementation is easy to explain, it may be a good idea. - Namespaces are one honking great idea -- let's do more of those! - - - Tim Peters, The Zen of Python - -Code Structure ---------- - -- HACKING - The document you are currently reading. - -- inputs/ - Contains input files for tests. - -- ooni/ - Contains the main ooni probe comand line client - -- ooni/inputunit.py - In here we have functions related to the creation of input - units. Input units are how the inputs to be fed to tests are - split up into. - -- ooni/nettest.py - In here is the NetTest API definition. This is how people - interested in writing ooniprobe tests will be specifying - them. - -- ooni/nodes.py - Mostly broken code for the remote dispatching of tests. - -- ooni/oonicli.py - In here we take care of running ooniprobe from the command - line interface - -- ooni/reporter.py - In here goes the logic for the creation of ooniprobe - reports. - -- ooni/runner.py - Handles running ooni.nettests as well as - ooni.plugoo.tests.OONITests. - -- ooni/settings.py - Parts of the code related to parsing OONI - configuration files and making them accessible - to other components of the software. - -- ooni/otime.py - Generation of timestamps, time conversions and all the rest - -- ooni/kit/ - In here go utilities that can be used by tests. - -- ooni/utils/ - In here go internal utilities that are useful to ooniprobe - -- ooni/geoip.py - In here go functions related to the understanding of - geographical information of the probe - -- ooni/utils/hacks.py - When some software has issues and we need to fix it in a - hackish way, we put it in here. This one day will be empty. - -- ooni/utils/log.py - log realted functions. - -- ooni/utils/net.py - utilities for networking related operations - -- ooni/utils/onion.py - Utilities for working with Tor. - XXX this code should be removed and merged into txtorcon. - -- ooni/utils/txscapy.py - Tools for making scapy work well with twisted. - -- ooniprobe.conf - The main OONI-probe configuration file. This can be used - to configure your OONI CLI, tell it where it should report - to, where the asset files are located, what should be used - for control, etc. - -Test related conventions ------------------------- - -These are the conventions for tests that are written in ooniprobe. That is what -goes inside of nettests/. - -Naming -...... - -All methods that are relevant to the test should be all lower case separated by -underscore. - -All variables that are specific to your test should be all lower case separated -by underscore. - -Simplicity -.......... - -Tests written in ooniprobe should be as short as possible and should contain as -little code not related to the measurement as possible. It should be possible -from reading the test to understand what it does without clutter. - -Everything that is not related directly to the test should go inside of the -test template of which the test you are writing is a subclass of. - - -Style guide ------------ - -This is an extract of the most important parts of PEP-8. When in doubt on -what code style should be followed first consult this doc, then PEP-8 and -if all fails use your best judgement or ask for help. - -The most important part to read is the following as it contains the guidelines -of naming of variables, functions and classes, as it does not follow pure -PEP-8. - -Naming convention -................. - -Class names should follow the CapWords convention. -Note: When using abbreviations in CapWords, capitalize all the letters - of the abbreviation. Thus HTTPServerError is better than - HttpServerError. - -Exception names should follow the class names convention as exceptions -should be classes. - -Method names should follow camelCase with the first letter non-capital. - -Class attributes should also follow camelCase with the first letter non-capital. - -Functions should follow camelCase with the first letter non-capital. - -Functions and variables that are inside the local scope of a class or method -should be all lowercase separated by an underscore. - -Indentation -........... - - Use 4 spaces per indentation level. - - This can be setup in vi with: - set tabstop=4 - set shiftwidth=4 - set expandtab - - - Continuation lines should be wrapper like this: - - foo = long_function_name(var_one, var_two, - var_three, var_four) - - or this: - - def long_function_name(var_one, - var_two, var_three, - var_four): - print(var_one) - - - They should NOT be wrapper like this: - - foo = long_function_name(var_one, var_two, - var_three, var_four) - - and NOT like this: - - # See how it creates confusion with what is inside the function? - def long_function_name(var_one, - var_two, var_three, - var_four): - print(var_one) - - -Tabs or Spaces? -............... - -Every time you insert a \t into any piece of code a kitten dies. - -Only spaces. Please. - -(code should be run with python -tt) - -Maximum Line Length -................... - -Maximum of 79 characters. 72 characters for long blocks of text is recommended. - -Blank Lines -........... - -Separate top-level function and class definitions with two blank lines. - -Method definitions inside of class are separated by a single blank line. - - -Encoding -........ - -Always use UTF-8 encoding. This can be specified by adding the encoding cookie -to the beginning of your python files: - - # -*- coding: UTF-8 - -All identifiers should be ASCII-only. All doc strings and comments should also -only be in ASCII. Non ASCII characters are allowed when they are related to -testing non-ASCII features or for the names of authors. - - -Imports -....... - -Import should be one per line as so: - - import os - import sys - from subprocess import Popen, PIPE - -Imports are always at the top of the file just after any module comments -and docstrings, berfore module globals and constants. - -Imports should be grouped in the following order: - -1. standard library imports -2. related third party imports -3. local application/library specific imports - -You should put a blank line between each group of imports. - - -Comments -........ - -Comments should always be up to date with the code. Don't have -comments that contraddict with the code. - -Comments should always be written in English. - -Blocks comments are indented to the same level of the code that -they refer to. They start with # and are followed by a single space. - -Use inline comments sparingly. # Gotcha? - - -Documentation strings -..................... - -Write docstrings for all public modules, functions, classes and -methods. Even better if you write them also for non-public methods. - -Place docstrings under the def. - -For a better overview on how to write docstrings consult: PEP-257 - - diff --git a/docs/coding-style.md b/docs/coding-style.md new file mode 100644 index 0000000..b1af0cf --- /dev/null +++ b/docs/coding-style.md @@ -0,0 +1,193 @@ +# Hacking on OONI + +This documents gives guidelines on where to start looking +for helping out in developing OONI and what guidelines you +should follow when writing code. + +We try to follow the general python best practices and styling +guides as specified in PEP. + + Beautiful is better than ugly. + Explicit is better than implicit. + Simple is better than complex. + Complex is better than complicated. + Flat is better than nested. + Sparse is better than dense. + Readability counts. + Special cases aren't special enough to break the rules. + Although practicality beats purity. + Errors should never pass silently. + Unless explicitly silenced. + In the face of ambiguity, refuse the temptation to guess. + There should be one-- and preferably only one --obvious way to do it. + Although that way may not be obvious at first unless you're Dutch. + Now is better than never. + Although never is often better than *right* now. + If the implementation is hard to explain, it's a bad idea. + If the implementation is easy to explain, it may be a good idea. + Namespaces are one honking great idea -- let's do more of those! + + - Tim Peters, The Zen of Python + +## Test related conventions + +These are the conventions for tests that are written in ooniprobe. That is what +goes inside of nettests/. + +### Naming + +All methods that are relevant to the test should be all lower case separated by +underscore. + +All variables that are specific to your test should be all lower case separated +by underscore. + +### Simplicity + +Tests written in ooniprobe should be as short as possible and should contain as +little code not related to the measurement as possible. It should be possible +from reading the test to understand what it does without clutter. + +Everything that is not related directly to the test should go inside of the +test template of which the test you are writing is a subclass of. + + +## Style guide + +This is an extract of the most important parts of PEP-8. When in doubt on +what code style should be followed first consult this doc, then PEP-8 and +if all fails use your best judgement or ask for help. + +The most important part to read is the following as it contains the guidelines +of naming of variables, functions and classes, as it does not follow pure +PEP-8. + +### Naming convention + +Class names should follow the CapWords convention. +Note: When using abbreviations in CapWords, capitalize all the letters + of the abbreviation. Thus HTTPServerError is better than + HttpServerError. + +Exception names should follow the class names convention as exceptions +should be classes. + +Method names should follow camelCase with the first letter non-capital. + +Class attributes should also follow camelCase with the first letter non-capital. + +Functions should follow camelCase with the first letter non-capital. + +Functions and variables that are inside the local scope of a class or method +should be all lowercase separated by an underscore. + +### Indentation + +Use 4 spaces per indentation level. + + This can be setup in vi with: + set tabstop=4 + set shiftwidth=4 + set expandtab + + +Continuation lines should be wrapped like this: + + foo = long_function_name(var_one, var_two, + var_three, var_four) + +or this: + + def long_function_name(var_one, + var_two, var_three, + var_four): + print(var_one) + + +They should NOT be wrapped like this: + + foo = long_function_name(var_one, var_two, + var_three, var_four) + +and NOT like this: + + # See how it creates confusion with what is inside the function? + def long_function_name(var_one, + var_two, var_three, + var_four): + print(var_one) + + +### Tabs or Spaces? + +Every time you insert a \t into any piece of code a kitten dies. + +Only spaces. Please. + +(code should be run with python -tt) + +### Maximum Line Length + +Maximum of 79 characters. 72 characters for long blocks of text is recommended. + +### Blank Lines + +Separate top-level function and class definitions with two blank lines. + +Method definitions inside of class are separated by a single blank line. + + +### Encoding + +Always use UTF-8 encoding. This can be specified by adding the encoding cookie +to the beginning of your python files: + + # -*- coding: UTF-8 + +All identifiers should be ASCII-only. All doc strings and comments should also +only be in ASCII. Non ASCII characters are allowed when they are related to +testing non-ASCII features or for the names of authors. + + +### Imports + +Import should be one per line as so: + + import os + import sys + from subprocess import Popen, PIPE + +Imports are always at the top of the file just after any module comments +and docstrings, berfore module globals and constants. + +Imports should be grouped in the following order: + +1. standard library imports +2. related third party imports +3. local application/library specific imports + +You should put a blank line between each group of imports. + + +### Comments + +Comments should always be up to date with the code. Don't have +comments that contraddict with the code. + +Comments should always be written in English. + +Blocks comments are indented to the same level of the code that +they refer to. They start with # and are followed by a single space. + +Use inline comments sparingly. # Gotcha? + + +### Documentation strings + +Write docstrings for all public modules, functions, classes and +methods. Even better if you write them also for non-public methods. + +Place docstrings under the def. + +For a better overview on how to write docstrings consult: PEP-257 +

1 0

[ooni-probe/master] Make no intersection amongst decks (#622)
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 16ff68c872818152784107f8f579b649f58225ce Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Oct 12 15:50:36 2016 +0200 Make no intersection amongst decks (#622) * Make no intersection amongst decks * Create 2 new decks that are currently quite weak, but can be expanded * Create dummy decks for all the default decks --- data/decks/http-invalid.yaml | 8 ++++++++ data/decks/im.yaml | 14 ++++++++++++++ data/decks/tor.yaml | 12 ++++++++++++ data/decks/web-full.yaml | 23 ----------------------- data/decks/web-no-invalid.yaml | 18 ------------------ data/decks/web.yaml | 18 ++++++++++++++++++ ooni/deck/store.py | 2 +- ooni/tests/test_scheduler.py | 18 +++++++++--------- 8 files changed, 62 insertions(+), 51 deletions(-) diff --git a/data/decks/http-invalid.yaml b/data/decks/http-invalid.yaml new file mode 100644 index 0000000..9690376 --- /dev/null +++ b/data/decks/http-invalid.yaml @@ -0,0 +1,8 @@ +--- +name: HTTP Invalid Request Line +description: This deck runs HTTP Invalid Request Line test +schedule: "@daily" +tasks: +- name: Runs the HTTP Invalid Request Line test + ooni: + test_name: http_invalid_request_line diff --git a/data/decks/im.yaml b/data/decks/im.yaml new file mode 100644 index 0000000..7609022 --- /dev/null +++ b/data/decks/im.yaml @@ -0,0 +1,14 @@ +--- +name: Instant Messaging deck +description: This test deck will check to see if instant messaging applications are working +schedule: "@daily" +tasks: +- name: Checks to see if WhatsApp web is working + ooni: + test_name: web_connectivity + url: 'https://web.whatsapp.com/' + +- name: Checks to see if Facebook is working + ooni: + test_name: web_connectivity + url: 'https://facebook.com/' diff --git a/data/decks/tor.yaml b/data/decks/tor.yaml new file mode 100644 index 0000000..9d0634f --- /dev/null +++ b/data/decks/tor.yaml @@ -0,0 +1,12 @@ +--- +name: Tor test deck +description: This deck runs test related to testing the reachability of the Tor network +schedule: "@daily" +tasks: +- name: Checks if the Tor client works + ooni: + test_name: vanilla_tor + +- name: Runs the Meek fronted request test + ooni: + test_name: meek_fronted_requests diff --git a/data/decks/web-full.yaml b/data/decks/web-full.yaml deleted file mode 100644 index ccb470d..0000000 --- a/data/decks/web-full.yaml +++ /dev/null @@ -1,23 +0,0 @@ ---- -name: Full Web test deck -description: This deck runs HTTP Header Field Manipulation, HTTP Invalid - Request Line and the Web Connectivity test -schedule: "@daily" -tasks: -- name: Runs the HTTP Header Field Manipulation test - ooni: - test_name: http_header_field_manipulation - -- name: Runs the HTTP Invalid Request Line test - ooni: - test_name: http_invalid_request_line - -- name: Runs the Web Connectivity Test - ooni: - test_name: web_connectivity - file: $citizenlab_global_urls - -- name: Runs the Web Connectivity Test - ooni: - test_name: web_connectivity - file: $citizenlab_${probe_cc}_urls diff --git a/data/decks/web-no-invalid.yaml b/data/decks/web-no-invalid.yaml deleted file mode 100644 index ea93488..0000000 --- a/data/decks/web-no-invalid.yaml +++ /dev/null @@ -1,18 +0,0 @@ ---- -name: Web test deck without HTTP Invalid Request Line -description: This deck runs HTTP Header Field Manipulation, and the Web Connectivity test -schedule: "@daily" -tasks: -- name: Runs the HTTP Header Field Manipulation test - ooni: - test_name: http_header_field_manipulation - -- name: Runs the Web Connectivity Test - ooni: - test_name: web_connectivity - file: $citizenlab_global_urls - -- name: Runs the Web Connectivity Test - ooni: - test_name: web_connectivity - file: $citizenlab_${probe_cc}_urls diff --git a/data/decks/web.yaml b/data/decks/web.yaml new file mode 100644 index 0000000..9232cec --- /dev/null +++ b/data/decks/web.yaml @@ -0,0 +1,18 @@ +--- +name: Web test deck +description: This deck runs HTTP Header Field Manipulation and the Web Connectivity test +schedule: "@daily" +tasks: +- name: Runs the HTTP Header Field Manipulation test + ooni: + test_name: http_header_field_manipulation + +- name: Runs the Web Connectivity Test + ooni: + test_name: web_connectivity + file: $citizenlab_global_urls + +- name: Runs the Web Connectivity Test + ooni: + test_name: web_connectivity + file: $citizenlab_${probe_cc}_urls diff --git a/ooni/deck/store.py b/ooni/deck/store.py index 8fdb856..956ea37 100644 --- a/ooni/deck/store.py +++ b/ooni/deck/store.py @@ -13,7 +13,7 @@ from ooni.resources import check_for_update from ooni.settings import config # These are the decks to be run by default. -DEFAULT_DECKS = ['web-full'] +DEFAULT_DECKS = ['web', 'tor', 'im', 'http-invalid'] class InputNotFound(Exception): pass diff --git a/ooni/tests/test_scheduler.py b/ooni/tests/test_scheduler.py index e954668..c0ab5eb 100644 --- a/ooni/tests/test_scheduler.py +++ b/ooni/tests/test_scheduler.py @@ -15,7 +15,7 @@ from twisted.trial import unittest from ooni.utils import generate_filename, randomSTR, randomDate, LONG_DATE from ooni.tests.bases import ConfigTestCase -from ooni.deck.store import DeckStore +from ooni.deck.store import DeckStore, DEFAULT_DECKS from ooni.agent.scheduler import ScheduledTask, DidNotRun from ooni.agent.scheduler import FileSystemlockAndMutex from ooni.agent.scheduler import SchedulerService @@ -182,14 +182,13 @@ def write_dummy_measurements(fh, size=100): return written_size -DUMMY_WEB_FULL = """ +DUMMY_DECK = """ --- -name: Full Web test deck -description: This deck runs HTTP Header Field Manipulation, HTTP Invalid - Request and the Web Connectivity test +name: Dummy deck +description: Dummy deck schedule: "@daily" tasks: -- name: Runs the HTTP Header Field Manipulation test +- name: Dummy bar ooni: test_name: http_header_field_manipulation """ @@ -232,9 +231,10 @@ class TestSchedulerService(ConfigTestCase): self.mock_deck.setup.return_value = defer.succeed(None) self.mock_deck.run.return_value = defer.succeed(None) - with open(os.path.join(self.decks_available_directory, - 'web-full.yaml'), 'w') as out_file: - out_file.write(DUMMY_WEB_FULL) + for deck_name in DEFAULT_DECKS: + with open(os.path.join(self.decks_available_directory, + '%s.yaml' % deck_name), 'w') as out_file: + out_file.write(DUMMY_DECK) def create_dummy_measurements(self, count=10, size=10*1024): for _ in range(count):

1 0

[ooni-probe/master] Avoid twisted complains about missing pidfile during shutdown
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 5414f5227973587daf67440501cfc71e454f1d65 Author: Leonid Evdokimov <leon(a)darkk.net.ru> Date: Wed Oct 12 18:18:20 2016 +0300 Avoid twisted complains about missing pidfile during shutdown Twisted wants to clean it's pidfiles itself, so `ooniprobe-agent` should remove the pidfile if and only if it's stale. --- ooni/scripts/ooniprobe_agent.py | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/ooni/scripts/ooniprobe_agent.py b/ooni/scripts/ooniprobe_agent.py index e670d86..d7f4750 100644 --- a/ooni/scripts/ooniprobe_agent.py +++ b/ooni/scripts/ooniprobe_agent.py @@ -121,6 +121,17 @@ def get_running_pidfile(): return pidfile raise NotRunning +def is_stale_pidfile(pidfile): + try: + with open(pidfile) as fd: + pid = int(fd.read()) + except Exception: + return False # that's either garbage in the pid-file or a race + return not is_process_running(pid) + +def get_stale_pidfiles(): + return [f for f in [config.system_pid_path, config.user_pid_path] if is_stale_pidfile(f)] + def status_agent(): try: get_running_pidfile() @@ -130,7 +141,7 @@ def status_agent(): print("ooniprobe-agent is NOT running") return 1 -def stop_agent(): +def do_stop_agent(): # This function is borrowed from tahoe try: pidfile = get_running_pidfile() @@ -144,12 +155,7 @@ def stop_agent(): os.kill(pid, signal.SIGTERM) except OSError as ose: if ose.errno == errno.ESRCH: - print("No process was running. Cleaning up.") - # the process didn't exist, so wipe the pid file - try: - os.remove(pidfile) - except EnvironmentError as exc: - print("Failed to delete the pidfile {0}".format(exc)) + print("No process was running.") # it's just a race return 2 elif ose.errno == errno.EPERM: # The process is owned by root. We assume it's running @@ -157,10 +163,7 @@ def stop_agent(): return 3 else: raise - try: - os.remove(pidfile) - except EnvironmentError: - pass + # the process wants to clean it's own pidfile itself start = time.time() time.sleep(0.1) wait = 40 @@ -197,6 +200,16 @@ def stop_agent(): # we define rc=1 to mean "I think something is still running, sorry" return 1 +def stop_agent(): + retval = do_stop_agent() + for pidfile in get_stale_pidfiles(): + try: + os.remove(pidfile) + print("Cleaned up stale pidfile {0}".format(pidfile)) + except EnvironmentError: + print("Failed to delete the pidfile {0}: {1}".format(pidfile, exc)) + return retval + def run(): options = AgentOptions() options.parseOptions()

1 0

[ooni-probe/master] SIG_DFL is not a signal, it's a signal handler
by art＠torproject.org 14 Oct '16

14 Oct '16

commit aabbd9b5c07484ee5e4e7493778527b0801859e9 Author: Leonid Evdokimov <leon(a)darkk.net.ru> Date: Wed Oct 12 17:55:54 2016 +0300 SIG_DFL is not a signal, it's a signal handler It's semantically wrong to call kill with SIG_DFL although it's having correct `0` value by chance. --- ooni/scripts/ooniprobe_agent.py | 2 +- ooni/utils/__init__.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/ooni/scripts/ooniprobe_agent.py b/ooni/scripts/ooniprobe_agent.py index 49694eb..e670d86 100644 --- a/ooni/scripts/ooniprobe_agent.py +++ b/ooni/scripts/ooniprobe_agent.py @@ -168,7 +168,7 @@ def stop_agent(): while True: # poll once per second until we see the process is no longer running try: - os.kill(pid, signal.SIG_DFL) + os.kill(pid, 0) except OSError: print("process %d is dead" % pid) return diff --git a/ooni/utils/__init__.py b/ooni/utils/__init__.py index def533a..6abd04f 100644 --- a/ooni/utils/__init__.py +++ b/ooni/utils/__init__.py @@ -174,7 +174,7 @@ def get_ooni_root(): def is_process_running(pid): try: - os.kill(pid, signal.SIG_DFL) + os.kill(pid, 0) running = True except OSError as ose: if ose.errno == errno.EPERM:

1 0

[ooni-probe/master] When generating a summary do so inside of a separate thread to avoid blocking the reactor thread
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 44478df607a7a5b3e3fbdb129651ad4c9a2b2729 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 7 20:13:38 2016 +0200 When generating a summary do so inside of a separate thread to avoid blocking the reactor thread --- ooni/measurements.py | 15 ++++++++++++--- ooni/ui/web/server.py | 6 ++++-- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/ooni/measurements.py b/ooni/measurements.py index 422fc55..46a447e 100644 --- a/ooni/measurements.py +++ b/ooni/measurements.py @@ -1,5 +1,7 @@ import json +from twisted.internet import defer +from twisted.internet.threads import deferToThread from twisted.python.filepath import FilePath from ooni.utils import log, is_process_running from ooni.utils.files import directory_usage @@ -56,6 +58,7 @@ def generate_summary(input_file, output_file): with open(output_file, "w") as fw: json.dump(results, fw) + return results class MeasurementNotFound(Exception): pass @@ -103,21 +106,27 @@ def get_measurement(measurement_id, compute_size=False): def get_summary(measurement_id): + """ + Returns a deferred that will fire with the content of the summary + or will errback with MeasurementInProgress if the measurement has not + yet finished running. + """ measurement_path = FilePath(config.measurements_directory) measurement = measurement_path.child(measurement_id) if measurement.child("measurements.njson.progress").exists(): - raise MeasurementInProgress + return defer.fail(MeasurementInProgress) summary = measurement.child("summary.json") if not summary.exists(): - generate_summary( + return deferToThread( + generate_summary, measurement.child("measurements.njson").path, summary.path ) with summary.open("r") as f: - return json.load(f) + return defer.succeed(json.load(f)) def list_measurements(compute_size=False): diff --git a/ooni/ui/web/server.py b/ooni/ui/web/server.py index 385bf18..fa35af0 100644 --- a/ooni/ui/web/server.py +++ b/ooni/ui/web/server.py @@ -527,7 +527,9 @@ class WebUIAPI(object): @app.route('/api/measurement/<string:measurement_id>', methods=["GET"]) @xsrf_protect(check=False) @requires_true(attrs=['_is_initialized']) + @defer.inlineCallbacks def api_measurement_summary(self, request, measurement_id): + log.warn("SUMMARY") try: measurement = get_measurement(measurement_id) except InsecurePath: @@ -540,8 +542,8 @@ class WebUIAPI(object): if measurement['completed'] is False: raise WebUIError(400, "measurement in progress") - summary = get_summary(measurement_id) - return self.render_json(summary, request) + summary = yield get_summary(measurement_id) + defer.returnValue(self.render_json(summary, request)) @app.route('/api/measurement/<string:measurement_id>', methods=["DELETE"]) @xsrf_protect(check=True)

1 0

[ooni-probe/master] Point to the correct web UI URL when calling ooniprobe -w (#625)
by art＠torproject.org 14 Oct '16

14 Oct '16

commit cadb8010cd190626dbe28100943990c3e2036d74 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 10:35:11 2016 +0200 Point to the correct web UI URL when calling ooniprobe -w (#625) This fixes #623 --- ooni/scripts/ooniprobe.py | 12 ++++++------ ooni/scripts/ooniprobe_agent.py | 4 +--- ooni/settings.py | 7 +++++++ 3 files changed, 14 insertions(+), 9 deletions(-) diff --git a/ooni/scripts/ooniprobe.py b/ooni/scripts/ooniprobe.py index c8ffee2..e8b9af4 100644 --- a/ooni/scripts/ooniprobe.py +++ b/ooni/scripts/ooniprobe.py @@ -13,19 +13,19 @@ def ooniprobe(reactor): if global_options['queue']: return runWithDaemonDirector(global_options) - if global_options['initialize']: - initializeOoniprobe(global_options) - return defer.succeed(None) - if global_options['web-ui']: - from ooni.scripts.ooniprobe_agent import WEB_UI_URL + from ooni.settings import config from ooni.scripts.ooniprobe_agent import status_agent, start_agent if status_agent() != 0: p = Process(target=start_agent) p.start() p.join() print("Started ooniprobe-agent") - webbrowser.open_new(WEB_UI_URL) + webbrowser.open_new(config.web_ui_url) + return defer.succeed(None) + + if global_options['initialize']: + initializeOoniprobe(global_options) return defer.succeed(None) return runWithDirector(global_options) diff --git a/ooni/scripts/ooniprobe_agent.py b/ooni/scripts/ooniprobe_agent.py index d7f4750..c7987d0 100644 --- a/ooni/scripts/ooniprobe_agent.py +++ b/ooni/scripts/ooniprobe_agent.py @@ -94,9 +94,7 @@ def start_agent(options=None): pass print("Starting ooniprobe agent.") - WEB_UI_URL = "http://{0}:{1}".format( - config.advanced.webui_address, config.advanced.webui_port) - print("To view the GUI go to %s" % WEB_UI_URL) + print("To view the GUI go to %s" % config.web_ui_url) log.start() twistd.runApp(twistd_config) return 0 diff --git a/ooni/settings.py b/ooni/settings.py index b4bb594..44bd8dc 100644 --- a/ooni/settings.py +++ b/ooni/settings.py @@ -318,6 +318,13 @@ class OConfig(object): else: return os.path.join(home, '.ooni') + @property + def web_ui_url(self): + return "http://{0}:{1}".format( + self.advanced.webui_address, + self.advanced.webui_port + ) + def get_data_file_path(self, file_name): for target_dir in self.data_directory_candidates: file_path = os.path.join(target_dir, file_name)

1 0

[ooni-probe/master] Skip the outer loop also on empty measurements
by art＠torproject.org 14 Oct '16

14 Oct '16

commit ffab2ef266e3facfd0d9a9c5120cda2b8c3a811b Author: Arturo Filastò <arturo(a)filasto.net> Date: Wed Oct 12 16:45:03 2016 +0200 Skip the outer loop also on empty measurements --- ooni/nettest.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/ooni/nettest.py b/ooni/nettest.py index 7a2afc8..1bb5678 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -608,6 +608,11 @@ class NetTest(object): self.state.taskCreated() yield measurement + # This is to skip setting callbacks on measurements that + # cannot be run. + if len(measurements) == 0: + continue + # When the measurement.done callbacks have all fired # call the postProcessor before writing the report if self.report:

1 0

[ooni-probe/master] Update changelog entry for 2.0.0 release
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 838b3c544ef76f3613f524a89ac2a4ff8bf6056a Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 14:54:49 2016 +0200 Update changelog entry for 2.0.0 release * Bump the version number --- ChangeLog.rst | 27 +++++++++++++++++++++++++++ ooni/__init__.py | 3 ++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/ChangeLog.rst b/ChangeLog.rst index f847675..d63ca7f 100644 --- a/ChangeLog.rst +++ b/ChangeLog.rst @@ -1,6 +1,33 @@ Changelog ========= +v2.0.0 (Fri, 14 Oct 2016) +------------------------------ + +Stable release of ooniprobe 2.0.0 + +Feature list: + +* System daemon for running tests periodically (https://github.com/TheTorProject/ooni-probe/issues/576) + +* Web user interface for viewing measurement results (https://github.com/TheTorProject/ooni-probe/issues/575) + +* New deck format (https://github.com/TheTorProject/ooni-probe/issues/571) + +* Local reports are written in JSON (https://github.com/TheTorProject/ooni-probe/issues/557) + +* Include decks for testing reachability of Tor, websites of IM apps + +* Include the platform as an annotation inside of reports + +Bugfixing since previous release candidates: + +* Fix -w option of ooniprobe (https://github.com/TheTorProject/ooni-probe/issues/623) + +* Scheduler lockfile for RunDecks not being released (https://github.com/TheTorProject/ooni-probe/issues/612) + +* Missing country testing list (https://github.com/TheTorProject/ooni-probe/issues/606) + v2.0.0-rc.3 (Mon, 19 Sep 2016) ------------------------------ diff --git a/ooni/__init__.py b/ooni/__init__.py index afedecd..d950362 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -1,7 +1,8 @@ # -*- encoding: utf-8 -*- __author__ = "Open Observatory of Network Interference" -__version__ = "2.0.0-rc.4.dev" +# XXX note this rc is not public just meant for internal testing +__version__ = "2.0.0-rc4" __all__ = [ 'agent',

1 0

[ooni-probe/master] Ensure annotations are always dicts
by art＠torproject.org 14 Oct '16

14 Oct '16

commit 528ace6a2eec664d86bca38214462d9874b370e2 Author: Arturo Filastò <arturo(a)filasto.net> Date: Fri Oct 14 12:38:07 2016 +0200 Ensure annotations are always dicts --- ooni/nettest.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ooni/nettest.py b/ooni/nettest.py index 5d33342..b81678b 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -171,6 +171,9 @@ class NetTestLoader(object): def __init__(self, options, test_file=None, test_string=None, annotations={}): self.options = options + # Ensure annotations are always dicts + if not isinstance(annotations, dict): + annotations = {} self.annotations = annotations self.annotations['platform'] = self.annotations.get('platform', config.platform)

1 0