[tor-commits] [ooni-probe/master] OONI Control Interface
art at torproject.org
art at torproject.org
Fri Apr 29 09:42:26 UTC 2016
commit 754ce68db5e11e0d9b6df7fbc5e7b898489af061
Author: Poly <poly at darkdepths.net>
Date: Tue Apr 5 15:14:25 2016 +0000
OONI Control Interface
control interface initial commit
small updates to spec
formatting
added deletion to results API
transition deck start to POST
added netttest support and more
spelling mistakes :P
added more error code specs
more error codes
more error codes 2
completed error codes
s/deck/decks/g
added link to API spec
fixed link
---
docs/source/architecture.rst | 128 +-----------
docs/source/control_interface.rst | 429 ++++++++++++++++++++++++++++++++++++++
2 files changed, 435 insertions(+), 122 deletions(-)
diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst
index d72a75c..5e7d253 100644
--- a/docs/source/architecture.rst
+++ b/docs/source/architecture.rst
@@ -179,128 +179,12 @@ Here we describe how an ooniprobe run should look like:
report file and/or the collector will be updated.
-ooniprobe API
--------------
-
-Note: This is currently not under active development, but we are looking for
-people interested in hacking on it!
-
-The goals of the ooniprobe API is that of allowing applications to interact
-with an ooniprobe.
-
-Such API will be exposed as an HTTP service that communicates with applications
-thanks to JSON formatted messages.
-
-When ooniprobe is launched it will start as a daemon. Through the HTTP based
-API it will be possible to `start tests`_, `stop tests`_ and `monitor test
-progress`_.
-
-By having it use HTTP it will be possible to decouple the problem of running
-tests from that of controlling them.
-You can think of it as the Tor Control port of ooniprobe.
-
-The kinds of clients that will be interested in using such API are:
-
- * The ooniprobe *command line interface*
-
- * The ooniprobe *HTML5/JS web application*
-
- * Any *third party* tool that is interested in running ooniprobe tests with
- custom arguments.
-
-Towards a RPC like interface
-----------------------------
-
-Such API will allow ooniprobes to expose such API as a Tor Hidden
-Service (if the user wishes to do so). This will allow people running ooniprobe
-to give the .onion address of the probe to a censorship researcher and they
-will then be able to run tests from the network vantage point of the probe.
-
-Draft API specification
-------------------------
-
-Through the ooniprobe API it will be possible to `start tests`_, `stop tests`_ and `monitor test
-progress`_.
-
-List tests
-..........
-
-`GET /test`
-
-Shall return the list of available tests as an array.
-
-This is how a response looks like
-::
-
- [{'id': 'http_requests',
- 'name': 'HTTP Requests Test',
- 'description': 'This test perform a HTTP GET request for the / resource over the test network and over Tor',
- 'type': [ 'blocking' ],
- 'version': '0.1',
- 'arguments': {
- 'urllist': 'Specify the list of URLs to be used for the test'
- }
- }]
-
-*type* may be either **blocking** or **manipulation**.
-
-Start tests
-...........
-
-
-`POST /test/<test_id>/start`
-
-Is used to start a test with the specified test_id.
-
-Inside of the request you will specify the arguments supported by the test
-
-This is how a request could look like
-::
- {
- 'urllist':
- ['http://google.com/', 'http://torproject.org/']
- }
-
-The server will then respond with the test object
-::
- {
- 'status': 'running',
- 'percentage': 0,
- 'current_input': 'http://google.com/',
- 'urllist':
- ['http://google.com/', 'http://torproject.org/']
- }
-
-
-Stop tests
-...........
-
-`POST /test/<test_id>/stop`
-
-This will terminate the execution of the test with the specified test_id.
-
-The request may optionally contain a reason for stopping the test such as
-::
- {
- 'reason': 'some reason'
- }
-
-Monitor test progress
-.....................
-
-`GET /test/<test_id>`
-
-Will return the status of a test
-
-Like so for example
-::
- {
- 'status': 'running',
- 'percentage': 0,
- 'current_input': 'http://google.com/',
- 'urllist':
- ['http://google.com/', 'http://torproject.org/']
- }
+OONIprobe Control Interface
+---------------------------
+.. XXX update this section once interface is implemented.
+The ooniprobe client provides a rich and simple JSON-based interface for
+control over HTTP. While the implementation of this interface is currently
+a work in progress, the specification may be found `here <control_interface.rst>`_.
Implementation status
diff --git a/docs/source/control_interface.rst b/docs/source/control_interface.rst
new file mode 100644
index 0000000..0104e4d
--- /dev/null
+++ b/docs/source/control_interface.rst
@@ -0,0 +1,429 @@
+====================================
+HTTP Control Interface Specification
+====================================
+
+The ``ooniprobe`` client provides a HTTP-based control interface. The goal of
+this interface is to allow applications to interact with ooniprobe in a
+standardized manner. The control interface aims to be a RESTful, stateless
+and simple protocol.
+
+This control interface will be exposed as an HTTP service that communicates
+using JSON encoded messages. The service shall be provided via the ooniprobe
+daemon, ``oonid``.
+
+While the HTTP Control Interface is **currently under development** with
+a HTML/JS WebGUI client in mind, it will also be compatible with other
+clients, such as Ledilopter (ooniprobe on the Raspberry Pi) and 3rd party
+tools.
+
+
+.. contents:: **Table of Contents**
+ :depth: 2
+
+General Notes
+.............
+This document is the working specification for ooniprobe HTTP Control
+Interface version 0.
+
+All the fields defined in this specification are mandatory, unless explicitly
+marked otherwise.
+
+All successful client requests should return a 2xx status code, and should
+return return a JSON encoded body if required in the specification below.
+Failed (or deferred) requests will cause a 3xx, 4xx or 5xx status code on
+response. All failure responses must return a JSON encoded body with the
+following format::
+
+ {
+ 'error_code': <some_integer_error_code>,
+ 'error_message': <some_string_explanation>
+ }
+
+In particular, should the client make a request to a invalid URI, the service
+shall respond with ``Status-code: 404`` and the reponse body::
+
+ {
+ 'error_code': 404,
+ 'error_message': 'Resource Not Found'
+ }
+
+If the client makes a request with invalid syntax that prevents the service
+from understanding the request, the service shall return ``Status-code: 400``
+and the response body::
+
+ {
+ 'error_code': 400,
+ 'error_message': 'Invalid syntax'
+ }
+
+Error handling is the client's responsibility.
+
+All date and time records in this specification shall be encoded according to
+`ISO 8601 <https://en.wikipedia.org/wiki/ISO_8601>`_.
+
+Decks
+.....
+Testing in OONI is centered around 'decks'. A deck is defined as a collection
+of network tests and their associated inputs.
+
+List decks
+^^^^^^^^^^
+To retrieve an array of all available decks:
+
+``GET /decks``
+
+The server shall repond with ``Status-code: 200`` and a body format::
+
+ [{'id': 'deck-it',
+ 'name': 'Deck for Italy',
+ 'description': 'blah blah blah',
+ 'nettests': [ 'http_invalid_request_line',
+ 'http_header_field_manipulation',
+ 'dns_consistency'],
+ }]
+
+Where ``nettests`` must refer to valid test IDs. The service may return an
+empty list if no tests are found.
+
+If the service is unable to retrieve the list of decks, it shall return a
+reponse with ``Status-code: 500`` and a body format::
+
+ {
+ 'error_code': 500,
+ 'error_message': 'Internal Server Error - Could not find directory "decks"'
+ }
+
+Generate decks
+^^^^^^^^^^^^^^
+Decks can either be generated for a specific country, or you can let
+oonideckgen try to automatically detect the country.
+
+``POST /decks``
+
+Request format::
+
+ {
+ 'country': 'CN' // optional, oonideckgen will autodetect otherwise
+ }
+
+On success, the service shall respond with ``status code 200`` and with the
+following body format::
+
+ {
+ 'deck_id': 'deck-cn'
+ }
+
+Deck IDs are not guaranteed to be unique - if the exact same deck has been
+created through an earlier request, the service shall reply with the old
+deck ID.
+
+If the operation fails the service shall respond with the appropriate status
+code and message.
+
+In particular, if the client makes invalid deck generation request, the
+service shall reply with ``Status-Code: 400`` and body message formatted::
+
+ {
+ 'error_code': 400,
+ 'error_message': 'Bad Request - "xy" is not a valid ISO country code'
+ }
+
+If the server is unable to generate a deck due to an internal error, it shall
+respond with ``Status-Code: 500`` and a body message formatted::
+
+ {
+ 'error_code': 500,
+ 'error_message': 'Internal Error - oonideckgen: couldn't fetch "http://someurl.com"'
+ }
+
+Start deck
+^^^^^^^^^^
+``POST /decks/<deck_id>/start``
+
+To run a deck, the above POST request is sent where ``deck_id`` must be a
+valid deck ID.
+
+Request format::
+
+ {
+ 'collector': true, // optional, defaults to true
+ 'bouncer': 'http://someaddress.onion' // optional, defaults to httpo://XXX
+ }
+
+On success, the server shall respond with ``Status-Code: 200`` and with the
+following body format::
+
+ {
+ 'current_nettest': 'dns_consistency',
+ 'time_started': '2014-03-12T13:37:27+00:00'
+ }
+
+If the service is unable to start the test, it shall respond with the
+appropriate status code and message.
+
+In particular, if the client attempts to run multiple decks simoultensouly,
+the service shall respond with ``Status-Code: 503`` and the body::
+
+ {
+ 'error_code': 503,
+ 'error_message': 'Unable to handle request - another deck is already running'
+ }
+
+If the service is unable to start the deck due to an interal error (for example, corrupted input files) it shall respond with ``Status-Code: 500`` and
+the body::
+
+ {
+ 'error_code': 500,
+ 'error_message': 'Unable to handle request - oonid: unable to find input file "DNE.txt"'
+ }
+
+Stop deck
+^^^^^^^^^
+``GET /decks/<deck_id>/stop``
+
+To stop a deck, the above GET request is sent where ``deck_id`` must be a
+valid deck ID.
+
+On success, the service shall respond with ``Status-Code: 204 - No Content``.
+
+If the requested deck is not running, the service shall repond with ``Status-Code: 400``
+and the body formatted::
+
+ {
+ 'error_code': 400,
+ 'error_message': 'Invalid Request - Deck is not running'
+ }
+
+Should the server be unable to stop the test, it shall repond with
+``Status-Code: 500`` and display a reason in the body in the following format::
+
+ {
+ 'error_code': 500,
+ 'error_message': 'Unable to handle request - out of RAM'
+ }
+
+Deck progress
+^^^^^^^^^^^^^
+``GET /decks/<deck_id>``
+
+Returns the deck progress if the deck is running, or the deck results in
+JSON format if the deck is complete.
+
+If the deck progress is successfully found, the service shall respond with
+``Status-code: 200`` and a response body formatted as follows::
+
+ {
+ 'status': 'running',
+ 'percentage': 32,
+ 'current_nettest': 'http_headers',
+ 'results': null
+ }
+
+Another possible response body::
+
+ {
+ 'status': 'complete',
+ 'percentage': 100,
+ 'current_nettest': null,
+ 'results': '<result_id>'
+ }
+
+The ``status`` field may be one of: ``stopped``, ``running`` or ``complete``.
+``results`` must be a valid result ID or ``null`` if the test is not yet
+finished.
+
+Otherwise, if the deck exists but the progress request fails, the service
+shall respond with ``Status-code: 500`` and an explanation in the following
+format::
+
+ {
+ 'error_code': 500,
+ 'error_message': 'Internal Server Error - could not find deck result ID'
+ }
+
+Net Tests
+.........
+In OONI, a ``nettest`` represents an individual anomaly detection technique.
+To run, nettests also require an input file which specifies on which URIs the
+test is to be performed.
+
+Usually, several nettests are bundled along with their inputs in a deck, which
+makes it easier for the end user to run. Nonetheless, the control interface
+allows for nettests to be run individually.
+
+List tests
+^^^^^^^^^^
+To retrieve an array of all available nettests:
+
+``GET /tests``
+
+On success, the service shall repond with ``Status-code: 200`` and a body formatted::
+
+ [
+ {'id': 'dns-consistency',
+ 'name': 'DNS Consistency',
+ 'description': 'Compares the results of two DNS lookups',
+ 'type': 'blocking',
+ 'version': '0.1',
+ 'arguments': {
+ 'urllist': 'Specify the list of URLs to be used for the test'
+ }, ...
+ }, ...
+ ]
+
+Where ``nettests`` must refer to valid test IDs. The field ``type`` may be
+of value ``blocking`` or ``manipulation``. If no decks are found, the service
+will still respond with ``Status-code: 200`` and an empty list.
+
+If the service is unable to retrieve the list of decks, it shall return a
+reponse with ``Status-code: 500`` and a body format::
+
+ {
+ 'error_code': 500,
+ 'error_message': 'Internal Server Error - Could not find directory "decks"'
+ }
+
+Starting a Test
+^^^^^^^^^^^^^^^
+To run a given test, the client must send the following request:
+``POST /tests/<test_id>/start``
+
+With the request body::
+
+ {
+ 'urllist':
+ ['http://google.com/', 'http://torproject.org']
+ }
+
+The server shall respond with ``status code 200`` with the body::
+
+ {
+ 'time_started': '2014-03-12T13:37:27+00:00',
+ 'percentage': 55,
+ 'current_input': 'http://google.com',
+ 'arguments': [<list of supplied arguments>]
+ }
+
+Else, the reponse shall be a error status code and an explanation. In
+particular, the service shall respond with ``Status-Code: 400`` if the user
+provides an invalid argument or ``Status-Code: 500`` if the server is unable
+to start the test due to an internal reason.
+
+Stopping a Test
+^^^^^^^^^^^^^^^
+To terminate a given test, the client sends the following request:
+``GET /tests/<test_id>/stop``
+
+The server shall respond with status code 204 - no content - or with an error
+message if it is unable to stop the test.
+
+If the requested nettest is not running, the service shall repond with ``Status-Code: 400``
+and the body formatted::
+
+ {
+ 'error_code': 400,
+ 'error_message': 'Invalid Request - nettest is not running'
+ }
+
+Should the server be unable to stop the test, it shall repond with
+``Status-Code: 503`` and display a reason in the body in the following format::
+
+ {
+ 'error_code': 503,
+ 'error_message': 'Unable to handle request - out of RAM'
+ }
+
+Test progress
+^^^^^^^^^^^^^
+``GET /tests/<test_id>``
+
+The service will respond with ``Status-Code 200`` and a body formatted::
+
+ {
+ 'status': 'running',
+ 'time_started': '2014-03-12T13:37:27+00:00',
+ 'percentage': 55,
+ 'current_input': 'http://google.com',
+ 'arguments': [<list of supplied arguments>],
+ 'results': <result_id>
+ }
+
+The ``status`` field may be one of: ``stopped``, ``running`` or ``complete``.
+``results`` must be a valid result ID or ``null`` if the test is not yet
+finished.
+
+Should the service fail to determine the progress of a nettest, it shall return
+``Status-code: 503`` along with a suitable error message.
+
+Results
+.......
+``GET /results``
+
+Returns a list of all stored results of previous runs.
+
+The service will respond with ``Status-code: 200`` and results formatted::
+
+ [
+ {'id': '<result_id>',
+ 'type': 'deck',
+ 'deck': '<deck_id>',
+ 'time_started': '2014-03-12T13:37:27+00:00',
+ 'time_finished': '2014-03-12T13:37:27+00:00',
+ 'collector': true,
+ 'collector-address': 'httpo://nkvphnp3p6agi5qq.onion',
+ 'nettests': ['http_headers', 'http_requests', ...]
+ }, ...
+ ]
+
+Where ``type`` shall be either ``deck`` or ``nettest``. If the result is of
+type ``deck``, then the fields ``deck_id`` and ``nettests`` are mandatory.
+Likewise, if the result is of type ``nettest`` the field ``test_id`` shall
+be mandatory.
+
+To get the results of the individual nettests (that are part of a deck)
+in JSON format:
+
+``GET /results/<result_id>/nettest/<nettest_id>``
+
+Please be warned that this will return the raw output of the test, which may be
+in excess of several tens of megabytes.
+
+The server should return ``Status-Code 500`` and an explanation should it
+fail to collect the results.
+
+Deleting Results
+^^^^^^^^^^^^^^^^
+
+To delete a particular result:
+
+``GET /results/<result_id>/delete``
+
+After deletion, the service shall reply with ``Status-code: 204`` - no content.
+The server shall reply with ``Status-code: 503`` should deletion fail.
+
+Resources
+.........
+To update ooniprobe's geoIP databases or input files, the client may send
+the following request:
+
+``POST /resources/update``
+
+With the body formatted as::
+
+ {
+ 'update_geoIP': true,
+ 'update_inputs': false
+ }
+
+The server shall reply with the ``204`` status code or if the update fails,
+``503`` status code.
+
+TODO
+....
+Open questions include:
+
+* Authentication. Users probably don't want anyone who can access port 80
+ on their machines to control ooniprobe.
+
+* Protocol signaling. Ideally we want some way to make clients aware of
+ different protocol versions.
More information about the tor-commits
mailing list