tor-commits December 2012

tor-commits@lists.torproject.org

17 participants
1600 discussions

[translation/vidalia_help] Update translations for vidalia_help
by translation＠torproject.org 22 Dec '12

22 Dec '12

commit bb6284590d9dd2a1f989850f922807f756e58469 Author: Translation commit bot <translation(a)torproject.org> Date: Sat Dec 22 01:15:29 2012 +0000 Update translations for vidalia_help --- es/config.po | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/es/config.po b/es/config.po index 9e7a2e2..063175b 100644 --- a/es/config.po +++ b/es/config.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2010-06-26 16:58+0200\n" -"PO-Revision-Date: 2012-12-22 00:43+0000\n" +"PO-Revision-Date: 2012-12-22 00:48+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -266,7 +266,7 @@ msgid "" "have Vidalia start Tor using a specific torrc, Tor's configuration " "file. If you leave this field blank, Tor will uses its own default torrc " "location." -msgstr "Archivo de configuración de Tor (opcional): Puede usar esta opción para hacer que Vidalia inicie Tor usando un archivo de configuración de Tor torrc específico. Si deja este campo en blanco, Tor usará su propio archivo de configuración torrc de la ubicación predeterminada." +msgstr "Archivo de configuración de Tor (opcional): Puede usar esta opción para hacer que Vidalia inicie Tor usando un archivo de configuración de Tor torrc específico. Si deja este campo en blanco, Tor usará su propio archivo de configuración torrc en la ubicación predeterminada." #. type: Content of: <html><body><ul><li> #: en/config.html:175 @@ -275,7 +275,7 @@ msgid "" "in which Tor will store its saved data, such as cached Tor relay " "information, Tor relay keys, and configuration files. If you leave this " "field blank, Tor will use its own default data directory location." -msgstr "Carpeta de datos de Tor (opcional): Puede especificar la carpeta en la que Tor almacenará sus datos operativos, como la información de caché del repetidor Tor, las claves del repetidor, y los archivos de configuración. Si deja este campo en blanco, Tor usará su propia ubicación predeterminada para la carpeta de datos." +msgstr "Carpeta de datos de Tor (opcional): Puede especificar la carpeta en la que Tor almacenará sus datos, como la información de caché del repetidor Tor, las claves del repetidor, y archivos de configuración. Si deja este campo en blanco, Tor usará su propia ubicación predeterminada para la carpeta de datos." #. type: Content of: <html><body><ul><li> #: en/config.html:181

1 0

[translation/vidalia_help_completed] Update translations for vidalia_help_completed
by translation＠torproject.org 22 Dec '12

22 Dec '12

commit 1785639878389511db092d26128c11b2c282065f Author: Translation commit bot <translation(a)torproject.org> Date: Sat Dec 22 00:45:40 2012 +0000 Update translations for vidalia_help_completed --- es/config.po | 22 +++++++++++----------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/es/config.po b/es/config.po index 495f3d6..9e7a2e2 100644 --- a/es/config.po +++ b/es/config.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2010-06-26 16:58+0200\n" -"PO-Revision-Date: 2012-12-21 22:32+0000\n" +"PO-Revision-Date: 2012-12-22 00:43+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -103,7 +103,7 @@ msgid "" "listening on the ports allowed by your firewall or proxy. Simply enter a " "list of ports permitted by your firewall or proxy, separated by commas. " "(Example: 80,443,8080)" -msgstr "Mi cortafuegos (firewall) sólo me permite conectarse a determinados puertos: Si está detrás de un firewall o un proxy restrictivo que limita los puertos a los que le permite conectar, puede configurar Tor para conectar directamente sólo con repetidores que escuchen en los puertos permitidos por su firewall o proxy. Basta con introducir una lista de puertos permitidos, separados por comas. (Ejemplo: 80,443,8080)" +msgstr "Mi cortafuegos (firewall) sólo me permite conectar a determinados puertos: Si está detrás de un firewall o un proxy restrictivo que limita los puertos a los que se le permite conectar, puede configurar Tor para conectarse directamente sólo con repetidores que escuchen en los puertos permitidos por su firewall o proxy. Basta con introducir una lista de puertos permitidos, separados por comas. (Ejemplo: 80,443,8080)" #. type: Content of: <html><body><ul><li> #: en/config.html:69 @@ -114,12 +114,12 @@ msgid "" "to the Tor network through relays called bridges (Tor 0.2.0.3-alpha " "or newer only). You can add bridge relays by specifying either their address" " and port number, or their address, port number, and fingerprint." -msgstr "Mi ISP bloquea conexiones a la red Tor: Si se conecta a su proveedor de servicios de Internet (ISP) bloquea conexiones a la red Tor, su software Tor puede tratar de evitar ser filtrado cifrando sus conexiones con repositorios de repetidores y conectando a la red Tor a través de un tipo de repetidores llamados repetidores puente o bridges (sólo con Tor 0.2.0.3-alpha o posterior). Puede añadir bridges especificando su dirección y número de puerto; o su dirección, número de puerto y huella de validación de claves (fingerprint)." +msgstr "Mi ISP bloquea conexiones a la red Tor: Si su proveedor de servicios de Internet (ISP) bloquea conexiones a la red Tor, su software Tor puede tratar de evitar ser filtrado cifrando sus conexiones con repositorios de repetidores y conectando a la red Tor a través de un tipo de repetidores llamados repetidores puente o bridges (sólo con Tor 0.2.0.3-alpha o posterior). Puede añadir bridges especificando su dirección y número de puerto; o su dirección, número de puerto y huella de validación de claves (fingerprint)." #. type: Content of: <html><body><ul><li> #: en/config.html:77 msgid "Below are examples of valid bridge address formats:" -msgstr "A continuación tenemos ejemplos de formatos válidos de direcciones de repetidores puente (bridges):" +msgstr "A continuación tenemos ejemplos de formatos válidos de direcciones de bridges:" #. type: Content of: <html><body><ul><li><ul><li> #: en/config.html:80 @@ -188,7 +188,7 @@ msgid "" "incorrectly, or if you prefer a different language, you can choose another " "language from the dropdown box. You will need to restart Vidalia after " "changing the displayed language for the changes to take effect." -msgstr "Idioma: La interfaz de Vidalia ha sido traducida a varios idiomas con la ayuda de voluntarios. Cuando Vidalia se ejecuta por primera vez, tratará de averiguar el idioma que su equipo está utilizando en ese momento. Si Vidalia hace una estimación incorrecta, o si prefiere un idioma diferente, puede elegir otro idioma en el cuadro desplegable. Tendrá que reiniciar Vidalia después de cambiar la configuración de idioma, para que los cambios surtan efecto." +msgstr "Idioma: La interfaz de Vidalia ha sido traducida a varios idiomas con la ayuda de voluntarios. Cuando Vidalia se ejecuta por primera vez, tratará de averiguar el idioma que su equipo está utilizando en ese momento. Si Vidalia hace una estimación incorrecta, o si prefiere un idioma diferente, puede elegir otro idioma en el cuadro desplegable. Tendrá que reiniciar Vidalia después de cambiar la configuración de idioma para que los cambios surtan efecto." #. type: Content of: <html><body><ul><li> #: en/config.html:124 @@ -196,12 +196,12 @@ msgid "" "Style: In most cases, Vidalia will default to using your platform's " "default interface style. If you dislike the default, you can choose " "whichever interface style you prefer from the dropdown box." -msgstr "Estilo: En la mayoría de los casos, Vidalia usará por defecto el estilo de la interfaz predeterminado de su plataforma. Si no le gusta el valor por defecto, puede elegir cualquier estilo de interfaz que prefiera en el cuadro desplegable." +msgstr "Estilo: En la mayoría de los casos, Vidalia usará por defecto el estilo predeterminado de la interfaz de su plataforma. Si no le gusta el valor por defecto, puede elegir cualquier estilo de interfaz que prefiera en el cuadro desplegable." #. type: Content of: <html><body> #: en/config.html:131 msgid "<a name=\"advanced\"/>" -msgstr "<a name=\"avanzada\"/>" +msgstr "<a name=\"avanzadas\"/>" #. type: Content of: <html><body><h3> #: en/config.html:132 @@ -222,7 +222,7 @@ msgid "" " Vidalia uses to talk to Tor. This doesn't need to be changed unless you " "have a conflict with another service on your machine, or if you are using " "Vidalia to control and monitor a Tor process running on another machine." -msgstr "Dirección y puerto de control: El puerto de control (ControlPort) es el puerto que Vidalia usa para comunicarse con su proceso de Tor. Esto no necesita ser cambiado a menos que haya un conflicto de puertos con otro servicio en su máquina, o si usted está usando Vidalia para controlar y monitorizar un proceso de Tor ejecutandose en otra máquina." +msgstr "Dirección y puerto de control: El puerto de control (ControlPort) es el puerto que Vidalia usa para comunicarse con su proceso de Tor. Esto no necesita ser cambiado a menos que haya un conflicto de puertos con otro servicio en su máquina, o si usted está usando Vidalia para controlar y monitorizar un proceso de Tor que se ejecuta en otra máquina." #. type: Content of: <html><body><ul><li> #: en/config.html:144 @@ -230,7 +230,7 @@ msgid "" "Control Port Authentication: Control port authentication is used to " "limit the applications on your machine that can connect to and reconfigure " "your Tor installation. The available authentication methods are:" -msgstr "Autentificación del puerto de control: La autentificación del puerto de control se utiliza como barrera para las aplicaciones que se ejecuten en su equipo que puedan tratar de conectarse a su proceso de Tor para reconfigurar su instalación. Los métodos de autentificación disponibles son:" +msgstr "Autentificación del puerto de control: La autentificación del puerto de control se utiliza como barrera para las aplicaciones que se ejecuten en su equipo, que puedan tratar de conectarse a su proceso de Tor y reconfigurar su instalación. Los métodos de autentificación disponibles son:" #. type: Content of: <html><body><ul><li><ul><li> #: en/config.html:150 @@ -248,7 +248,7 @@ msgid "" "connects to Tor's control port. If Vidalia starts Tor for you, you can have " "Vidalia randomly generate a new password each time it starts Tor by checking" " the Randomly Generate checkbox." -msgstr "Contraseña (por defecto) -- Si se selecciona este método, puede especificar una contraseña que Tor pedirá cada vez que un usuario o una aplicación se conecte al puerto de control de su proceso de Tor. Si Vidalia inicia Tor por usted, puede hacer que Vidalia genere una contraseña aleatoria nueva cada vez que inicia Tor, marcando la casilla Generar aleatoriamente." +msgstr "Contraseña (por defecto) -- Si se selecciona este método, puede especificar una contraseña que Tor pedirá cada vez que un usuario o una aplicación se conecte al puerto de control de su proceso de Tor. Si Vidalia inicia Tor por usted, puede hacer que Vidalia genere una contraseña aleatoria nueva cada vez que inicia Tor, marcando la casilla Generar aleatoriamente." #. type: Content of: <html><body><ul><li><ul><li> #: en/config.html:162 @@ -257,7 +257,7 @@ msgid "" " (or, cookie) containing random bytes to its data directory when it " "starts. Any user or application that tries to connect to Tor's control port " "must be able to provide the contents of this cookie." -msgstr "Cookie -- Si se selecciona la autentificación por cookies, al iniciarse Tor escribirá un archivo (cookie) que contiene bytes aleatorios en su carpeta de datos. Cualquier usuario o aplicación que intente conectarse al puerto de control de Tor tiene que ser capaz de proporcionar el contenido de esta cookie." +msgstr "Cookie -- Si se selecciona la autentificación por cookies, al iniciarse Tor escribirá un archivo (cookie) que contiene bytes aleatorios en su carpeta de datos. Cualquier usuario o aplicación que intente conectarse al puerto de control de Tor tiene que ser capaz de proporcionar el contenido de esa cookie." #. type: Content of: <html><body><ul><li> #: en/config.html:170

1 0

[translation/vidalia_help] Update translations for vidalia_help
by translation＠torproject.org 22 Dec '12

22 Dec '12

commit 5d2b5234664a7e5b2a808e6d28661b20a1b87cff Author: Translation commit bot <translation(a)torproject.org> Date: Sat Dec 22 00:45:32 2012 +0000 Update translations for vidalia_help --- es/config.po | 22 +++++++++++----------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/es/config.po b/es/config.po index 495f3d6..9e7a2e2 100644 --- a/es/config.po +++ b/es/config.po @@ -9,7 +9,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" "POT-Creation-Date: 2010-06-26 16:58+0200\n" -"PO-Revision-Date: 2012-12-21 22:32+0000\n" +"PO-Revision-Date: 2012-12-22 00:43+0000\n" "Last-Translator: strel <strelnic(a)gmail.com>\n" "Language-Team: Spanish (http://www.transifex.com/projects/p/torproject/language/es/)\n" "MIME-Version: 1.0\n" @@ -103,7 +103,7 @@ msgid "" "listening on the ports allowed by your firewall or proxy. Simply enter a " "list of ports permitted by your firewall or proxy, separated by commas. " "(Example: 80,443,8080)" -msgstr "Mi cortafuegos (firewall) sólo me permite conectarse a determinados puertos: Si está detrás de un firewall o un proxy restrictivo que limita los puertos a los que le permite conectar, puede configurar Tor para conectar directamente sólo con repetidores que escuchen en los puertos permitidos por su firewall o proxy. Basta con introducir una lista de puertos permitidos, separados por comas. (Ejemplo: 80,443,8080)" +msgstr "Mi cortafuegos (firewall) sólo me permite conectar a determinados puertos: Si está detrás de un firewall o un proxy restrictivo que limita los puertos a los que se le permite conectar, puede configurar Tor para conectarse directamente sólo con repetidores que escuchen en los puertos permitidos por su firewall o proxy. Basta con introducir una lista de puertos permitidos, separados por comas. (Ejemplo: 80,443,8080)" #. type: Content of: <html><body><ul><li> #: en/config.html:69 @@ -114,12 +114,12 @@ msgid "" "to the Tor network through relays called bridges (Tor 0.2.0.3-alpha " "or newer only). You can add bridge relays by specifying either their address" " and port number, or their address, port number, and fingerprint." -msgstr "Mi ISP bloquea conexiones a la red Tor: Si se conecta a su proveedor de servicios de Internet (ISP) bloquea conexiones a la red Tor, su software Tor puede tratar de evitar ser filtrado cifrando sus conexiones con repositorios de repetidores y conectando a la red Tor a través de un tipo de repetidores llamados repetidores puente o bridges (sólo con Tor 0.2.0.3-alpha o posterior). Puede añadir bridges especificando su dirección y número de puerto; o su dirección, número de puerto y huella de validación de claves (fingerprint)." +msgstr "Mi ISP bloquea conexiones a la red Tor: Si su proveedor de servicios de Internet (ISP) bloquea conexiones a la red Tor, su software Tor puede tratar de evitar ser filtrado cifrando sus conexiones con repositorios de repetidores y conectando a la red Tor a través de un tipo de repetidores llamados repetidores puente o bridges (sólo con Tor 0.2.0.3-alpha o posterior). Puede añadir bridges especificando su dirección y número de puerto; o su dirección, número de puerto y huella de validación de claves (fingerprint)." #. type: Content of: <html><body><ul><li> #: en/config.html:77 msgid "Below are examples of valid bridge address formats:" -msgstr "A continuación tenemos ejemplos de formatos válidos de direcciones de repetidores puente (bridges):" +msgstr "A continuación tenemos ejemplos de formatos válidos de direcciones de bridges:" #. type: Content of: <html><body><ul><li><ul><li> #: en/config.html:80 @@ -188,7 +188,7 @@ msgid "" "incorrectly, or if you prefer a different language, you can choose another " "language from the dropdown box. You will need to restart Vidalia after " "changing the displayed language for the changes to take effect." -msgstr "Idioma: La interfaz de Vidalia ha sido traducida a varios idiomas con la ayuda de voluntarios. Cuando Vidalia se ejecuta por primera vez, tratará de averiguar el idioma que su equipo está utilizando en ese momento. Si Vidalia hace una estimación incorrecta, o si prefiere un idioma diferente, puede elegir otro idioma en el cuadro desplegable. Tendrá que reiniciar Vidalia después de cambiar la configuración de idioma, para que los cambios surtan efecto." +msgstr "Idioma: La interfaz de Vidalia ha sido traducida a varios idiomas con la ayuda de voluntarios. Cuando Vidalia se ejecuta por primera vez, tratará de averiguar el idioma que su equipo está utilizando en ese momento. Si Vidalia hace una estimación incorrecta, o si prefiere un idioma diferente, puede elegir otro idioma en el cuadro desplegable. Tendrá que reiniciar Vidalia después de cambiar la configuración de idioma para que los cambios surtan efecto." #. type: Content of: <html><body><ul><li> #: en/config.html:124 @@ -196,12 +196,12 @@ msgid "" "Style: In most cases, Vidalia will default to using your platform's " "default interface style. If you dislike the default, you can choose " "whichever interface style you prefer from the dropdown box." -msgstr "Estilo: En la mayoría de los casos, Vidalia usará por defecto el estilo de la interfaz predeterminado de su plataforma. Si no le gusta el valor por defecto, puede elegir cualquier estilo de interfaz que prefiera en el cuadro desplegable." +msgstr "Estilo: En la mayoría de los casos, Vidalia usará por defecto el estilo predeterminado de la interfaz de su plataforma. Si no le gusta el valor por defecto, puede elegir cualquier estilo de interfaz que prefiera en el cuadro desplegable." #. type: Content of: <html><body> #: en/config.html:131 msgid "<a name=\"advanced\"/>" -msgstr "<a name=\"avanzada\"/>" +msgstr "<a name=\"avanzadas\"/>" #. type: Content of: <html><body><h3> #: en/config.html:132 @@ -222,7 +222,7 @@ msgid "" " Vidalia uses to talk to Tor. This doesn't need to be changed unless you " "have a conflict with another service on your machine, or if you are using " "Vidalia to control and monitor a Tor process running on another machine." -msgstr "Dirección y puerto de control: El puerto de control (ControlPort) es el puerto que Vidalia usa para comunicarse con su proceso de Tor. Esto no necesita ser cambiado a menos que haya un conflicto de puertos con otro servicio en su máquina, o si usted está usando Vidalia para controlar y monitorizar un proceso de Tor ejecutandose en otra máquina." +msgstr "Dirección y puerto de control: El puerto de control (ControlPort) es el puerto que Vidalia usa para comunicarse con su proceso de Tor. Esto no necesita ser cambiado a menos que haya un conflicto de puertos con otro servicio en su máquina, o si usted está usando Vidalia para controlar y monitorizar un proceso de Tor que se ejecuta en otra máquina." #. type: Content of: <html><body><ul><li> #: en/config.html:144 @@ -230,7 +230,7 @@ msgid "" "Control Port Authentication: Control port authentication is used to " "limit the applications on your machine that can connect to and reconfigure " "your Tor installation. The available authentication methods are:" -msgstr "Autentificación del puerto de control: La autentificación del puerto de control se utiliza como barrera para las aplicaciones que se ejecuten en su equipo que puedan tratar de conectarse a su proceso de Tor para reconfigurar su instalación. Los métodos de autentificación disponibles son:" +msgstr "Autentificación del puerto de control: La autentificación del puerto de control se utiliza como barrera para las aplicaciones que se ejecuten en su equipo, que puedan tratar de conectarse a su proceso de Tor y reconfigurar su instalación. Los métodos de autentificación disponibles son:" #. type: Content of: <html><body><ul><li><ul><li> #: en/config.html:150 @@ -248,7 +248,7 @@ msgid "" "connects to Tor's control port. If Vidalia starts Tor for you, you can have " "Vidalia randomly generate a new password each time it starts Tor by checking" " the Randomly Generate checkbox." -msgstr "Contraseña (por defecto) -- Si se selecciona este método, puede especificar una contraseña que Tor pedirá cada vez que un usuario o una aplicación se conecte al puerto de control de su proceso de Tor. Si Vidalia inicia Tor por usted, puede hacer que Vidalia genere una contraseña aleatoria nueva cada vez que inicia Tor, marcando la casilla Generar aleatoriamente." +msgstr "Contraseña (por defecto) -- Si se selecciona este método, puede especificar una contraseña que Tor pedirá cada vez que un usuario o una aplicación se conecte al puerto de control de su proceso de Tor. Si Vidalia inicia Tor por usted, puede hacer que Vidalia genere una contraseña aleatoria nueva cada vez que inicia Tor, marcando la casilla Generar aleatoriamente." #. type: Content of: <html><body><ul><li><ul><li> #: en/config.html:162 @@ -257,7 +257,7 @@ msgid "" " (or, cookie) containing random bytes to its data directory when it " "starts. Any user or application that tries to connect to Tor's control port " "must be able to provide the contents of this cookie." -msgstr "Cookie -- Si se selecciona la autentificación por cookies, al iniciarse Tor escribirá un archivo (cookie) que contiene bytes aleatorios en su carpeta de datos. Cualquier usuario o aplicación que intente conectarse al puerto de control de Tor tiene que ser capaz de proporcionar el contenido de esta cookie." +msgstr "Cookie -- Si se selecciona la autentificación por cookies, al iniciarse Tor escribirá un archivo (cookie) que contiene bytes aleatorios en su carpeta de datos. Cualquier usuario o aplicación que intente conectarse al puerto de control de Tor tiene que ser capaz de proporcionar el contenido de esa cookie." #. type: Content of: <html><body><ul><li> #: en/config.html:170

1 0

[ooni-probe/master] Add tor data directories into repo
by art＠torproject.org 22 Dec '12

22 Dec '12

commit e5714106fc4262fa6241424037c51fb1763d4e9c Author: aagbsn <aagbsn(a)extc.org> Date: Tue Dec 11 20:53:32 2012 +0000 Add tor data directories into repo (cherry picked from commit 4b8f1599710a837494739caf8543a3f03b50b957) --- .gitignore | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) diff --git a/.gitignore b/.gitignore index 4cb0775..9500420 100644 --- a/.gitignore +++ b/.gitignore @@ -13,7 +13,6 @@ ooni/lib/txtorcon ooni/plugins/dropin.cache ooni/report*.yaml oonib/oonibackend.conf -oonib/data/* fluff/ pcaps/ report*.yaml

1 0

[ooni-probe/master] Fix bug in printing log.warn
by art＠torproject.org 22 Dec '12

22 Dec '12

commit 573921df72ddd04bec92ac6f02adc2f589e511f7 Author: Arturo Filastò <art(a)fuffa.org> Date: Sat Dec 22 00:02:32 2012 +0100 Fix bug in printing log.warn --- ooni/oonicli.py | 6 ++---- 1 files changed, 2 insertions(+), 4 deletions(-) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 582a6ae..1c3edf6 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -174,10 +174,8 @@ def run(): try: checkForRoot() except PermissionsError, pe: - m = ("Capturing packets requires administrator/root privileges. ", - "Run ooniprobe as root or set 'includepcap = false' in ", - "ooniprobe.conf .") - log.warn("%s" % m) + log.warn("Capturing packets requires administrator/root privileges. ") + log.warn("Run ooniprobe as root or set 'includepcap = false' in ooniprobe.conf .") sys.exit(1) else: log.msg("Starting packet capture")

1 0

[ooni-probe/master] Revert back to 9212cba (0.0.9) that was the last known working state
by art＠torproject.org 22 Dec '12

22 Dec '12

commit 3ed62967a0119e2bf3393be4af98111f534b88ed Author: Arturo Filastò <art(a)fuffa.org> Date: Sat Dec 22 00:56:26 2012 +0100 Revert back to 9212cba (0.0.9) that was the last known working state commits between 20cd1df9de76b54a77aac22e44f457272354c0cb and 8c0e47c66ca78d6a94beacd90b0bb07a00cde013 broke master in a way that is complicated to debug. Such commits should be resubmitted as a pull request on github so that I can review them. In future when making changes to core ooniprobe components, please submit a pull request on github so that I can review the changes and test them before merging. --- .gitignore | 9 +- .../_static/images/ooniprobe-architecture.png | Bin 96574 -> 0 bytes docs/source/architecture.rst | 144 +--------- nettests/experimental/bridge_reachability/echo.py | 150 +++------- .../experimental/bridge_reachability/tcpsyn.py | 191 ------------ ooni/inputunit.py | 20 +- ooni/lib/001-scapy_missing-exc.sh.patch | 78 ----- ooni/nettest.py | 83 ++---- ooni/oonicli.py | 105 ++----- ooni/reporter.py | 41 ++-- ooni/runner.py | 321 ++++++-------------- ooni/utils/__init__.py | 22 +- ooni/utils/geodata.py | 18 +- ooni/utils/hacks.py | 8 - ooni/utils/log.py | 44 +--- ooni/utils/net.py | 124 +------- ooni/utils/txscapy.py | 79 ++--- oonib/README.md | 94 ------ ooniprobe.conf.sample | 16 +- scripts/before_i_commit.sh | 7 +- 20 files changed, 287 insertions(+), 1267 deletions(-) diff --git a/.gitignore b/.gitignore index 40df4cf..4cb0775 100644 --- a/.gitignore +++ b/.gitignore @@ -1,12 +1,8 @@ *.pyc assets/* -build/* -docs/build/* -data/*.dat ENV/* -fluff/ +build/* package/* -pcaps/* proxy-lists/ip-cc.txt proxy-lists/ips.txt proxy-lists/italy-dns-ips.txt @@ -18,12 +14,9 @@ ooni/plugins/dropin.cache ooni/report*.yaml oonib/oonibackend.conf oonib/data/* -*.resume fluff/ pcaps/ report*.yaml docs/build/* data/*.dat ooniprobe.conf -report*.yaml -report*.yamloo diff --git a/docs/source/_static/images/ooniprobe-architecture.png b/docs/source/_static/images/ooniprobe-architecture.png deleted file mode 100644 index 06065fc..0000000 Binary files a/docs/source/_static/images/ooniprobe-architecture.png and /dev/null differ diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index cb50402..7091c29 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -4,6 +4,8 @@ Architecture :Contact: art(a)torproject.org :Copyright: This document has been placed in the public domain. +.. [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/CnCVie…, width=550px, align=right)]] + The goal of this document is provide an overview of how ooni works, what are it's pieces and how they interact with one another. @@ -14,9 +16,6 @@ To get an idea of what is implemented and with what sort of quality see the The two main components of ooni are `oonib`_ and `ooniprobe`_. -.. image:: _static/images/ooniprobe-architecture.png - :width: 700px - ooniprobe --------- @@ -182,153 +181,24 @@ Draft API specification Through the ooniprobe API it will be possible to `start tests`_, `stop tests`_ and `monitor test progress`_. -List tests -.......... - -`GET /test` - -Shall return the list of available tests as an array. - -This is how a response looks like -:: - - [{'id': 'http_requests', - 'name': 'HTTP Requests Test', - 'description': 'This test perform a HTTP GET request for the / resource over the test network and over Tor', - 'type': [ 'blocking' ], - 'version': '0.1', - 'arguments': { - 'urllist': 'Specify the list of URLs to be used for the test' - } - }] - -*type* may be either **blocking** or **manipulation**. - Start tests ........... - -`POST /test/<test_id>/start` - -Is used to start a test with the specified test_id. - -Inside of the request you will specify the arguments supported by the test - -This is how a request could look like -:: - { - 'urllist': - ['http://google.com/', 'http://torproject.org/'] - } - -The server will then respond with the test object -:: - { - 'status': 'running', - 'percentage': 0, - 'current_input': 'http://google.com/', - 'urllist': - ['http://google.com/', 'http://torproject.org/'] - } - +.. TODO Stop tests ........... -`POST /test/<test_id>/stop` - -This will terminate the execution of the test with the specified test_id. - -The request may optionally contain a reason for stopping the test such as -:: - { - 'reason': 'some reason' - } +.. TODO Monitor test progress -..................... - -`GET /test/<test_id>` - -Will return the status of a test - -Like so for example -:: - { - 'status': 'running', - 'percentage': 0, - 'current_input': 'http://google.com/', - 'urllist': - ['http://google.com/', 'http://torproject.org/'] - } +...................... +.. TODO Implementation status ===================== -ooniprobe -......... - -**Reporting** - - * To flat YAML file: *alpha* - - * To remote httpo backend: *alpha* - -**Test templates** - - * HTTP test template: *alpha* - - * Scapy test template: *alpha* - - * DNS test template: *alpha* - - * TCP test template: *prototype* - -**Tests** - -To see the list of implemented tests see: -https://ooni.torproject.org/docs/#core-ooniprobe-tests - -**ooniprobe API** - - * Specification: *draft* - - * HTTP API: *not implemented* - -**ooniprobe HTML5/JS user interface** - - Not implemented. - -**ooniprobe build system** - - Not implemented. - -**ooniprobe command line interface** - - Implemented in alpha quality, though needs to be ported to use the HTTP based - API. - -oonib -..... - -**Collector** - - * collection of YAML reports to flat file: *alpha* - - * collection of pcap reports: *not implemented* - - * association of reports with test helpers: *not implemented* - -**Test helpers** - - * HTTP Return JSON Helper: *alpha* - - * DNS Test helper: *prototype* - - * Test Helper - collector mapping: *Not implemented* - - * TCP Test helper: *prototype* +.. TODO - * Daphn3 Test helper: *prototype* diff --git a/nettests/experimental/bridge_reachability/echo.py b/nettests/experimental/bridge_reachability/echo.py index 0f422ca..d4033dd 100644 --- a/nettests/experimental/bridge_reachability/echo.py +++ b/nettests/experimental/bridge_reachability/echo.py @@ -16,39 +16,34 @@ import os import sys from twisted.python import usage -from twisted.internet import reactor, defer, address +from twisted.internet import reactor, defer from ooni import nettest from ooni.utils import log, net, Storage, txscapy try: - from scapy.all import IP, ICMP - from scapy.all import sr1 - from ooni.utils import txscapy -except Exception, e: + from scapy.all import IP, ICMP + from scapy.all import sr1 + from ooni.lib import txscapy + from ooni.lib.txscapy import txsr, txsend + from ooni.templates.scapyt import BaseScapyTest +except: log.msg("This test requires scapy, see www.secdev.org/projects/scapy") - log.exception(e) class UsageOptions(usage.Options): - """ - Options for EchoTest. - - Note: 'count', 'size', and 'ttl' have yet to be implemented. - """ optParameters = [ ['dst', 'd', None, 'Host IP to ping'], ['file', 'f', None, 'File of list of IPs to ping'], - ['pcap', 'p', None, 'Save pcap to this file'], ['interface', 'i', None, 'Network interface to use'], - ['receive', 'r', True, 'Receive response packets'], - ['timeout', 't', 2, 'Seconds to wait if no response', int], ['count', 'c', 1, 'Number of packets to send', int], - ['size', 's', 56, 'Bytes to send in ICMP data field', int], - ['ttl', 'l', 25, 'Set the IP Time to Live', int]] + ['size', 's', 56, 'Number of bytes to send in ICMP data field', int], + ['ttl', 'l', 25, 'Set the IP Time to Live', int], + ['timeout', 't', 2, 'Seconds until timeout if no response', int], + ['pcap', 'p', None, 'Save pcap to this file'], + ['receive', 'r', True, 'Receive response packets']] class EchoTest(nettest.NetTestCase): """ - Basic ping test. This takes an input file containing one IP or hostname - per line. + xxx fill me in """ name = 'echo' author = 'Isis Lovecruft <isis(a)torproject.org>' @@ -60,22 +55,6 @@ class EchoTest(nettest.NetTestCase): #requiredOptions = ['dst'] def setUp(self, *a, **kw): - """ - Send an ICMP-8 packet to a host IP, and process the response. - - @param timeout: - Seconds after sending the last packet to timeout. - @param interface: - The interface to restrict listening to. - @param dst: - A single host to ping. - @param file: - A file of hosts to ping, one per line. - @param receive: - Whether or not to receive replies. Defaults to True. - @param pcap: - The file to save packet captures to. - """ self.destinations = {} if self.localOptions: @@ -90,7 +69,7 @@ class EchoTest(nettest.NetTestCase): iface = txscapy.getDefaultIface() except Exception, e: log.msg("No network interface specified!") - log.exception(e) + log.err(e) else: log.msg("Using system default interface: %s" % iface) self.interface = iface @@ -106,12 +85,13 @@ class EchoTest(nettest.NetTestCase): if not self.dst: if self.file: self.dstProcessor(self.file) - for address, details in self.destinations.items(): - for labels, data in details.items(): - if not 'response' in labels: - self.dst = details['dst_ip'] + for key, value in self.destinations.items(): + for label, data in value.items(): + if not 'ans' in data: + self.dst = label else: self.addDest(self.dst) + log.debug("self.dst is now: %s" % self.dst) log.debug("Initialization of %s test completed." % self.name) @@ -120,6 +100,8 @@ class EchoTest(nettest.NetTestCase): self.destinations[d] = {'dst_ip': d} def dstProcessor(self, inputfile): + from ipaddr import IPAddress + if os.path.isfile(inputfile): with open(inputfile) as f: for line in f.readlines(): @@ -127,74 +109,24 @@ class EchoTest(nettest.NetTestCase): continue self.addDest(line) - def build_packets(self): - """ - Construct a list of packets to send out. - """ - packets = [] - for dest, data in self.destinations.items(): - pkt = IP(dst=dest)/ICMP() - packets.append(pkt) - ## XXX if a domain was specified, we need a way to check that - ## its IP matches the one we're seeing in pkt.src - #try: - # address.IPAddress(dest) - #except: - # data['dst_ip'] = pkt.dst - return packets - def test_icmp(self): - """ - Send the list of ICMP packets. - - TODO: add end summary progress report for % answered, etc. - """ - try: - def nicely(packets): - """Print scapy summary nicely.""" - return list([x.summary() for x in packets]) - - def process_answered((answered, sent)): - """Callback function for txscapy.sr().""" - self.report['sent'] = nicely(sent) - self.report['answered'] = [nicely(ans) for ans in answered] - - for req, resp in answered: - log.msg("Received echo-reply:\n%s" % resp.summary()) - for dest, data in self.destinations.items(): - if data['dst_ip'] == resp.src: - data['response'] = resp.summary() - data['censored'] = False - for snd in sent: - if snd.dst == resp.src: - answered.remove((req, resp)) - return (answered, sent) - - def process_unanswered((unanswered, sent)): - """ - Callback function for remaining packets and destinations which - do not have an associated response. - """ - if len(unanswered) > 0: - nicer = [nicely(unans) for unans in unanswered] - log.msg("Unanswered/remaining packets:\n%s" - % nicer) - self.report['unanswered'] = nicer - for dest, data in self.destinations.items(): - if not 'response' in data: - log.msg("No reply from %s. Possible censorship event." - % dest) - data['response'] = None - data['censored'] = True - return (unanswered, sent) - - packets = self.build_packets() - d = txscapy.sr(packets, iface=self.interface, multi=True) - d.addCallback(process_answered) - d.addErrback(log.exception) - d.addCallback(process_unanswered) - d.addErrback(log.exception) - self.report['destinations'] = self.destinations - return d - except Exception, e: - log.exception(e) + def process_response(echo_reply, dest): + ans, unans = echo_reply + if ans: + log.msg("Recieved echo reply from %s: %s" % (dest, ans)) + else: + log.msg("No reply was received from %s. Possible censorship event." % dest) + log.debug("Unanswered packets: %s" % unans) + self.report[dest] = echo_reply + + for label, data in self.destinations.items(): + reply = sr1(IP(dst=lebal)/ICMP()) + process = process_reponse(reply, label) + + #(ans, unans) = ping + #self.destinations[self.dst].update({'ans': ans, + # 'unans': unans, + # 'response_packet': ping}) + #return ping + + #return reply diff --git a/nettests/experimental/bridge_reachability/tcpsyn.py b/nettests/experimental/bridge_reachability/tcpsyn.py deleted file mode 100644 index b3226c8..0000000 --- a/nettests/experimental/bridge_reachability/tcpsyn.py +++ /dev/null @@ -1,191 +0,0 @@ -#!/usr/bin/env python -# -*- coding: utf-8 -*- -# -# +-----------+ -# | tcpflags.py | -# +-----------+ -# Send packets with various TCP flags set to a test server -# to check that it is reachable. -# -# @authors: Isis Lovecruft, <isis(a)torproject.org> -# @version: 0.0.1-pre-alpha -# @license: copyright (c) 2012 Isis Lovecruft -# see attached LICENCE file -# - -import os -import sys - -from twisted.python import usage -from twisted.python.failure import Failure -from twisted.internet import reactor, defer -from ooni import nettest, config -from ooni.utils import net, log -from ooni.utils.otime import timestamp - -try: - from scapy.all import TCP, IP, sr - from ooni.utils import txscapy -except: - log.msg("This test requires scapy, see www.secdev.org/projects/scapy") - - -class TCPFlagsOptions(usage.Options): - """Options for TCPTest.""" - optParameters = [ - ['dst', 'd', None, 'Host IP to ping'], - ['port', 'p', None, 'Host port'], - ['flags', 's', 'S', 'Comma separated flags to set, eg. "SA"'], - ['count', 'c', 3, 'Number of SYN packets to send', int], - ['interface', 'i', None, 'Network interface to use'], - ['hexdump', 'x', False, 'Show hexdump of responses']] - -class TCPFlagsTest(nettest.NetTestCase): - """ - Sends only a TCP SYN packet to a host IP:PORT, and waits for either a - SYN/ACK, a RST, or an ICMP error. - - TCPSynTest can take an input file containing one IP:Port pair per line, or - the commandline switches --dst <IP> and --port <PORT> can be used. - """ - name = 'TCP Flags' - author = 'Isis Lovecruft <isis(a)torproject.org>' - description = 'A TCP SYN/ACK/FIN test to see if a host is reachable.' - version = '0.1.1' - requiresRoot = True - - usageOptions = TCPFlagsOptions - inputFile = ['file', 'f', None, 'File of list of IP:PORTs to ping'] - - destinations = {} - - def setUp(self, *a, **kw): - """Configure commandline parameters for TCPSynTest.""" - if self.localOptions: - for key, value in self.localOptions.items(): - setattr(self, key, value) - if not self.interface: - try: - iface = net.getDefaultIface() - self.interface = iface - except net.IfaceError: - self.abortClass("Could not find a working network interface!") - if self.flags: - self.flags = self.flags.split(',') - if config.advanced.debug: - defer.setDebugging('on') - - def addToDestinations(self, addr=None, port='443'): - """ - Validate and add an IP address and port to the dictionary of - destinations to send to. If the host's IP is already in the - destinations, then only add the port. - - @param addr: A string representing an IPv4 or IPv6 address. - @param port: A string representing a port number. - @returns: A 2-tuple containing the address and port. - """ - if addr is None: - return (None, None) # do we want to return SkipTest? - - dst, dport = net.checkIPandPort(addr, port) - if not dst in self.destinations.keys(): - self.destinations[dst] = {'dst': dst, - 'dport': [dport]} - else: - log.debug("Got additional port for destination.") - self.destinations[dst]['dport'].append(dport) - return (dst, dport) - - def inputProcessor(self, input_file=None): - """ - Pull the IPs and PORTs from the commandline options first, and then - from the input file, and place them in a dict for storing test results - as they arrive. - """ - if self.localOptions['dst'] is not None \ - and self.localOptions['port'] is not None: - log.debug("Processing commandline destination") - yield self.addToDestinations(self.localOptions['dst'], - self.localOptions['port']) - if input_file and os.path.isfile(input_file): - log.debug("Processing input file %s" % input_file) - with open(input_file) as f: - for line in f.readlines(): - if line.startswith('#'): - continue - one = line.strip() - raw_ip, raw_port = one.rsplit(':', 1) - yield self.addToDestinations(raw_ip, raw_port) - - def test_tcp_flags(self): - """ - Generate, send, and listen for responses to, a list of TCP/IP packets - to an address and port pair taken from the current input, and a string - specifying the TCP flags to set. - - @param flags: - A string representing the TCP flags to be set, i.e. "SA" or "F". - Defaults to "S". - """ - def build_packets(addr, port): - """Construct a list of packets to send out.""" - packets = [] - for flag in self.flags: - log.debug("Generating packets with %s flags for %s:%d..." - % (flag, addr, port)) - for x in xrange(self.count): - packets.append( IP(dst=addr)/TCP(dport=port, flags=flag) ) - return packets - - def process_packets(packet_list): - """ - If the source address of packet in :param:packet_list matches one of - our input destinations, then extract some of the information from it - to the test report. - - @param packet_list: - A :class:scapy.plist.PacketList - """ - log.msg("Processing received packets...") - results, unanswered = packet_list - - for (q, r) in results: - request = {'dst': q.dst, - 'dport': q.dport, - 'summary': q.summary(), - 'hexdump': None, - 'sent_time': q.time} - response = {'src': r['IP'].src, - 'flags': r['IP'].flags, - 'summary': r.summary(), - 'hexdump': None, - 'recv_time': r.time, - 'delay': r.time - q.time} - if self.hexdump: - request['hexdump'] = q.hexdump() - response['hexdump'] = r.hexdump() - - for dest, data in self.destinations.items(): - if response['src'] == data['dst']: - log.msg(" Received response from %s:\n%s ==> %s" % ( - response['src'], q.mysummary(), r.mysummary())) - if self.hexdump: - log.msg("%s\n%s" % (q.hexdump(), r.hexdump())) - - self.report['request'] = request - self.report['response'] = response - - if unanswered is not None: - unans = [un.summary() for un in unanswered] - log.msg(" Waiting on responses from:\n%s" % '\n'.join(unans)) - self.report['unanswered'] = unans - - try: - self.report = {} - (addr, port) = self.input - pkts = build_packets(addr, port) - d = process_packets(sr(pkts, iface=self.interface, timeout=5)) - return d - except Exception, ex: - log.exception(ex) diff --git a/ooni/inputunit.py b/ooni/inputunit.py index 0b4377f..2ef89d8 100644 --- a/ooni/inputunit.py +++ b/ooni/inputunit.py @@ -6,10 +6,9 @@ # units. Input units are how the inputs to be fed to tests are # split up into. # -# :authors: Arturo Filastò, Isis Lovecruft +# :authors: Arturo Filastò # :license: see included LICENSE file - class InputUnitFactory(object): """ This is a factory that takes the size of input units to be generated a set @@ -62,37 +61,24 @@ class InputUnitFactory(object): class InputUnit(object): """ This is a python iterable object that contains the input elements to be - passed onto a :class:`ooni.nettest.NetTestCase`. + passed onto a TestCase. """ def __init__(self, inputs=[]): - """ - Create an iterable from a list of inputs, which can be given to a NetTestCase. - - @param inputs: A list of inputs for a NetTestCase. - """ self._inputs = iter(inputs) - # _inputs_copy is to avoid stealing things from - # the iterator when __repr__ is called: - self._inputs_copy = inputs def __str__(self): - """Prints the original input list.""" - return "<%s inputs=%s>" % (self.__class__, self._inputs_copy) + return "<%s inputs=%s>" % (self.__class__, self._inputs) def __add__(self, inputs): - """Add a list of inputs to the iterator.""" for i in inputs: self._inputs.append(i) def __iter__(self): - """Self explanatory.""" return self def next(self): - """Return the next item from the InputUnit iterator.""" return self._inputs.next() def append(self, input): - """Add an item to the end of the InputUnit iterator.""" self._inputs.append(input) diff --git a/ooni/lib/001-scapy_missing-exc.sh.patch b/ooni/lib/001-scapy_missing-exc.sh.patch deleted file mode 100644 index 3f5095c..0000000 --- a/ooni/lib/001-scapy_missing-exc.sh.patch +++ /dev/null @@ -1,78 +0,0 @@ -# -# Add a missing Exception class to /scapy/arch/linux.py -# -# To apply this patch: -# STEP 1: Chdir to the source directory. If you have scapy installed, this is -# likely located at /usr/share/pyshared. -# STEP 2: Run the 'patch' program with this file as input, i.e.: -# -# /usr/share/pyshared$ patch -p1 ./scapy/arch/linux.py </path/to/this/file> -# -# To apply this patch with the use of 'applypatch': -# STEP 1: Chdir to the source directory. -# STEP 2: Run the 'applypatch' program with this patch file as input. -# -# @authors: Isis Lovecruft -# @license: This file is part of ooniprobe, see LICENSE file for details. -# @copyright: 2012 Isis Lovecruft -# -#### End of Preamble #### - -#### Patch data follows #### -diff -c 'scapy/arch/linux.py' 'scapy/arch/linux.py-patched' -Index: ./scapy/arch/linux.py -*** ./scapy/arch/linux.py Thu Dec 13 21:26:59 2012 ---- ./scapy/arch/linux.py-patched Thu Dec 13 21:27:53 2012 -*************** -*** 14,21 **** - from scapy.data import * - from scapy.supersocket import SuperSocket - import scapy.arch -! from scapy.error import warning -! - - - # From bits/ioctls.h ---- 14,20 ---- - from scapy.data import * - from scapy.supersocket import SuperSocket - import scapy.arch -! from scapy.error import warning,Scapy_Exception - - - # From bits/ioctls.h -#### End of Patch data #### - -#### ApplyPatch data follows #### -# Data version : 1.0 -# Date generated : Thu Dec 13 21:33:12 2012 -# Generated by : makepatch 2.03 -# Recurse directories : Yes -# Excluded files : (\A|/).*\~\Z -# (\A|/).*\.a\Z -# (\A|/).*\.bak\Z -# (\A|/).*\.BAK\Z -# (\A|/).*\.elc\Z -# (\A|/).*\.exe\Z -# (\A|/).*\.gz\Z -# (\A|/).*\.ln\Z -# (\A|/).*\.o\Z -# (\A|/).*\.obj\Z -# (\A|/).*\.olb\Z -# (\A|/).*\.old\Z -# (\A|/).*\.orig\Z -# (\A|/).*\.rej\Z -# (\A|/).*\.so\Z -# (\A|/).*\.Z\Z -# (\A|/)\.del\-.*\Z -# (\A|/)\.make\.state\Z -# (\A|/)\.nse_depinfo\Z -# (\A|/)core\Z -# (\A|/)tags\Z -# (\A|/)TAGS\Z -# p 'scapy/arch/linux.py' 16563 1355434073 0100644 -#### End of ApplyPatch data #### - -#### End of Patch kit [created: Thu Dec 13 21:33:12 2012] #### -#### Patch checksum: 56 1830 50583 #### -#### Checksum: 74 2507 41533 #### diff --git a/ooni/nettest.py b/ooni/nettest.py index 13012c7..fcc945a 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -1,22 +1,9 @@ -# -*- encoding: utf-8 -*- -# -# nettest.py -# ---------- -# In here is the NetTest API definition. This is how people -# interested in writing ooniprobe tests will be specifying them -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - -import sys -import os import itertools import traceback -import inspect +import sys +import os -from twisted.trial import unittest, itrial -from twisted.trial import util as txtrutil -from twisted.trial.test import skipping +from twisted.trial import unittest, itrial, util from twisted.internet import defer, utils from twisted.python import usage @@ -148,13 +135,16 @@ class NetTestCase(object): requiresRoot = False localOptions = {} - def _setUp(self): - """This is the internal setup method to be overwritten by templates.""" + """ + This is the internal setup method to be overwritten by templates. + """ pass def setUp(self): - """Place your logic to be executed when the test is being setup here.""" + """ + Place here your logic to be executed when the test is being setup. + """ pass def postProcessor(self, report): @@ -197,6 +187,12 @@ class NetTestCase(object): else: pass + def _checkRequiredOptions(self): + for required_option in self.requiredOptions: + log.debug("Checking if %s is present" % required_option) + if not self.localOptions[required_option]: + raise usage.UsageError("%s not specified!" % required_option) + def _processOptions(self): if self.inputFilename: inputProcessor = self.inputProcessor @@ -210,53 +206,10 @@ class NetTestCase(object): return inputProcessor(inputFilename) self.inputs = inputProcessorIterator() - return {'inputs': self.inputs, - 'name': self.name, - 'version': self.version} + return {'inputs': self.inputs, + 'name': self.name, 'version': self.version + } def __repr__(self): return "<%s inputs=%s>" % (self.__class__, self.inputs) - def _abortMethod(self, reason, method=None): - if method is None: - test_method = self._testMethod - else: - test_method = getattr(self.__class__, method, False) - - if inspect.ismethod(test_method): - method_name = test_method.im_func.func_name - setattr(test_method, 'skip', reason) - raise skipping.SkipTest("Aborting %s for reason: %s" - % (method_name, reason) ) - else: - log.debug("_abortMethod(): could not find method %s" % test_method) - - def abortInput(self, reason): - """ - Abort the current input. - - @param reason: A string explaining why this test is being skipped. - @raises: A :class:`twisted.trial.test.skipping.SkipTest <SkipTest>` - """ - raise skipping.SkipTest(" Reason: %s\nCurrent input: %s" - % (reason, self.input)) - - def abortMethod(self, reason, test_method=None): - """ - Abort all remaining inputs for the current test method. - - @param reason: A string explaining why the current test_method is - being skipped. - @param test_method: (optional) The test_method to skip, defaults to - the currently running test_method. - """ - return self._abortMethod(reason, test_method) - - def abortClass(self, reason='unspecified'): - """ - Abort the entire NetTestCase class. - - @param reason: A string explaining why the class is being skipped. - """ - log.msg("Aborting %s: %s" % (self.__class__.name, reason)) - setattr(self.__class__, 'skip', reason) diff --git a/ooni/oonicli.py b/ooni/oonicli.py index 1c3edf6..b06bde9 100644 --- a/ooni/oonicli.py +++ b/ooni/oonicli.py @@ -1,13 +1,4 @@ -# -*- coding: UTF-8 -# -# oonicli -# ------- -# In here we take care of running ooniprobe from the command -# line interface -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - +#-*- coding: utf-8 -*- import sys import os @@ -20,17 +11,14 @@ from twisted.application import app from twisted.python import usage, failure from twisted.python.util import spewer -# Supress scapy's "No route found for IPv6 destination" warnings: -import logging as pylogging -pylogging.getLogger("scapy.runtime").setLevel(pylogging.ERROR) - from ooni import nettest, runner, reporter, config + from ooni.inputunit import InputUnitFactory + from ooni.utils import net -from ooni.utils import checkForRoot, PermissionsError +from ooni.utils import checkForRoot, NotRootError from ooni.utils import log - class Options(usage.Options): synopsis = """%s [options] [path to test].py """ % (os.path.basename(sys.argv[0]),) @@ -42,14 +30,13 @@ class Options(usage.Options): optFlags = [["help", "h"], ["resume", "r"]] - optParameters = [ - ["reportfile", "o", None, "report file name"], - ["testdeck", "i", None, - "Specify a test deck: a yaml file containing tests and their arguments"], - ["collector", "c", None, - "Address of the collector of test results. (e.g.: http://127.0.0.1:8888)"], - ["logfile", "l", None, "log file name"], - ["pcapfile", "p", None, "pcap file name"]] + optParameters = [["reportfile", "o", None, "report file name"], + ["testdeck", "i", None, + "Specify as input a test deck: a yaml file containig the tests to run an their arguments"], + ["collector", "c", None, + "Address of the collector of test results. (example: http://127.0.0.1:8888)"], + ["logfile", "l", None, "log file name"], + ["pcapfile", "p", None, "pcap file name"]] compData = usage.Completions( extraActions=[usage.CompleteFiles( @@ -82,42 +69,23 @@ class Options(usage.Options): except: raise usage.UsageError("No test filename specified!") -class CooperativeTimer(object): - """ - A simple timer for the callback to functions on - :class:`twisted.internet.task.Cooperator <t.i.t.Cooperator>`. see - :meth:`oonicli.runTestList <runTestList>`. - - @param seconds: - An integer specifying the second to wait in between updating the - status and ETA bars. - """ - def __init__(self, seconds=5): - self.max_timer_interval = float(seconds) - self.end = time.time() + self.max_timer_interval - - def __call__(self): - return time.time() >= self.end - -def updateStatusBar(stop_func): +def updateStatusBar(): for test_filename in config.state.keys(): # The ETA is not updated so we we will not print it out for the # moment. eta = config.state[test_filename].eta() progress = config.state[test_filename].progress() - while progress is not None: - print "[%s] %s%%" % (test_filename, progress) - else: - print "[%s] All tests in file completed." % test_filename - stop_func() + progress_bar_frmt = "[%s] %s%%" % (test_filename, progress) + print progress_bar_frmt def testsEnded(*arg, **kw): - """You can place here all the post shutdown tasks.""" - log.debug("Finished running all tests") + """ + You can place here all the post shutdown tasks. + """ + log.debug("testsEnded: Finished running all tests") config.start_reactor = False - if not reactor.running: - try: reactor.stop() - except: reactor.runUntilCurrent() + try: reactor.stop() + except: pass def testFailed(failure): log.err("Failed in running a test inside a test list") @@ -141,14 +109,9 @@ def runTestList(none, test_list): d2.addCallback(testsEnded) d2.addErrback(testFailed) - try: - # Print every 5 second the list of current tests running - coop = task.Cooperator(started=False) - coop.cooperate(updateStatusBar) #this will need a .next() method - coop.start() - except StopIteration: - return d2 - + # Print every 5 second the list of current tests running + l = task.LoopingCall(updateStatusBar) + l.start(5.0) return d2 def errorRunningTests(failure): @@ -156,8 +119,9 @@ def errorRunningTests(failure): failure.printTraceback() def run(): - """Call me to begin testing from a file.""" - + """ + Parses command line arguments of test. + """ cmd_line_options = Options() if len(sys.argv) == 1: cmd_line_options.getUsage() @@ -171,15 +135,10 @@ def run(): config.cmd_line_options = cmd_line_options if config.privacy.includepcap: - try: - checkForRoot() - except PermissionsError, pe: - log.warn("Capturing packets requires administrator/root privileges. ") - log.warn("Run ooniprobe as root or set 'includepcap = false' in ooniprobe.conf .") - sys.exit(1) - else: - log.msg("Starting packet capture") - runner.startSniffing() + log.msg("Starting") + if not config.reports.pcap: + config.generatePcapFilename() + runner.startSniffing() resume = cmd_line_options['resume'] @@ -212,6 +171,4 @@ def run(): d = runTestList(None, test_list) d.addErrback(errorRunningTests) - # XXX I believe we don't actually need this: - #reactor.run() - + reactor.run() diff --git a/ooni/reporter.py b/ooni/reporter.py index aac163d..728c3f5 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -1,20 +1,11 @@ -#-*- coding: utf-8 -*- -# -# reporter.py -# ----------- -# In here goes the logic for the creation of ooniprobe reports. -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - import traceback import itertools import logging -import sys -import os import time import yaml import json +import sys +import os import re from yaml.representer import * @@ -26,9 +17,7 @@ from twisted.trial import reporter from twisted.internet import defer, reactor from twisted.internet.error import ConnectionRefusedError -from ooni import config, otime -from ooni.utils import log, geodata, pushFilenameStack -from ooni.utils.net import BodyReceiver, StringProducer, userAgents +from ooni.utils import log try: from scapy.packet import Packet @@ -36,6 +25,12 @@ except ImportError: log.err("Scapy is not installed.") +from ooni import otime +from ooni.utils import geodata, pushFilenameStack +from ooni.utils.net import BodyReceiver, StringProducer, userAgents + +from ooni import config + def createPacketReport(packet_list): """ Takes as input a packet a list. @@ -157,7 +152,7 @@ def getTestDetails(options): 'test_version': options['version'], 'software_name': 'ooniprobe', 'software_version': software_version - } + } return test_details class OReporter(object): @@ -180,7 +175,7 @@ class OReporter(object): pass def testDone(self, test, test_name): - log.debug("Calling reporter to record results") + log.msg("Finished running %s" % test_name) test_report = dict(test.report) if isinstance(test.input, Packet): @@ -278,8 +273,8 @@ class OONIBReporter(OReporter): try: self.agent = Agent(reactor, sockshost="127.0.0.1", socksport=int(config.tor.socks_port)) - except Exception, ex: - log.exception(ex) + except Exception, e: + log.exception(e) OReporter.__init__(self, cmd_line_options) @@ -319,8 +314,8 @@ class OONIBReporter(OReporter): try: test_details = getTestDetails(options) - except Exception, ex: - log.exception(ex) + except Exception, e: + log.exception(e) test_details['options'] = self.cmd_line_options @@ -339,7 +334,7 @@ class OONIBReporter(OReporter): 'test_name': test_name, 'test_version': test_version, 'content': content - } + } log.msg("Reporting %s" % url) request_json = json.dumps(request) @@ -357,8 +352,8 @@ class OONIBReporter(OReporter): log.err("Connection to reporting backend failed (ConnectionRefusedError)") raise OONIBReportCreationError - except Exception, ex: - log.exception(ex) + except Exception, e: + log.exception(e) raise OONIBReportCreationError # This is a little trix to allow us to unspool the response. We create diff --git a/ooni/runner.py b/ooni/runner.py index d8b6c4e..4ebfa0b 100644 --- a/ooni/runner.py +++ b/ooni/runner.py @@ -1,79 +1,31 @@ -#-*- coding: utf-8 -*- -# -# runner.py -# --------- -# Handles running ooni.nettests as well as -# ooni.plugoo.tests.OONITests. -# -# :authors: Arturo Filastò, Isis Lovecruft -# :license: see included LICENSE file - import os import sys import time import inspect import traceback import itertools + import yaml -from twisted.python import reflect, usage, failure -from twisted.internet import defer, reactor, threads -from twisted.trial import reporter as txreporter -from twisted.trial import util as txutil +from twisted.python import reflect, usage +from twisted.internet import defer from twisted.trial.runner import filenameToModule -from twisted.trial.unittest import utils as txutils -from twisted.trial.unittest import SkipTest +from twisted.internet import reactor, threads from txtorcon import TorProtocolFactory, TorConfig from txtorcon import TorState, launch_tor -from ooni import config, nettest, reporter -from ooni.inputunit import InputUnitFactory +from ooni import config + from ooni.reporter import OONIBReporter, YAMLReporter, OONIBReportError from ooni.inputunit import InputUnitFactory from ooni.nettest import NetTestCase, NoPostProcessor -from ooni.utils import log, checkForRoot -from ooni.utils import PermissionsError, Storage +from ooni.utils import log, checkForRoot, pushFilenameStack +from ooni.utils import NotRootError, Storage from ooni.utils.net import randomFreePort - -class NoTestCasesFound(Exception): - pass - -class InvalidResumeFile(Exception): - pass - -class noResumeSession(Exception): - pass - -class InvalidConfigFile(Exception): - message = "Invalid setting in ooniprobe.conf: " - -class UnableToStartTor(Exception): - pass - - -def isTestCase(obj): - """Return True if obj is a subclass of NetTestCase, False otherwise.""" - try: - return issubclass(obj, nettest.NetTestCase) - except TypeError: - return False - -def checkRequiredOptions(test_instance): - """ - If test_instance has an attribute 'requiredOptions', then check that - those options were utilised on the commandline. - """ - required = getattr(test_instance, 'requiredOptions', None) - if required: - for required_option in required: - log.debug("Checking if %s is present" % required_option) - if not test_instance.localOptions[required_option]: - raise usage.UsageError("%s not specified!" % required_option) - def processTest(obj, cmd_line_options): """ Process the parameters and :class:`twisted.python.usage.Options` of a @@ -86,46 +38,58 @@ def processTest(obj, cmd_line_options): :param cmd_line_options: A configured and instantiated :class:`twisted.python.usage.Options` class. - """ - if obj.requiresRoot: - try: - checkForRoot() - except PermissionsError: - log.err("%s requires root to run" % obj.name) - sys.exit(1) + """ if not hasattr(obj.usageOptions, 'optParameters'): obj.usageOptions.optParameters = [] + if obj.inputFile: + obj.usageOptions.optParameters.append(obj.inputFile) + if obj.baseParameters: for parameter in obj.baseParameters: obj.usageOptions.optParameters.append(parameter) + if obj.baseFlags: if not hasattr(obj.usageOptions, 'optFlags'): obj.usageOptions.optFlags = [] for flag in obj.baseFlags: obj.usageOptions.optFlags.append(flag) - if obj.inputFile: # inputFile is the optParameters list - obj.usageOptions.optParameters.append(obj.inputFile) options = obj.usageOptions() + options.parseOptions(cmd_line_options['subargs']) obj.localOptions = options - if obj.inputFile: # inputFilename is the actual filename + if obj.inputFile: obj.inputFilename = options[obj.inputFile[0]] try: - log.debug("Parsing commandline options") - tmp_test_instance = obj() - checkRequiredOptions(tmp_test_instance) - except usage.UsageError, ue: - log.err("%s" % ue) + log.debug("processing options") + tmp_test_case_object = obj() + tmp_test_case_object._checkRequiredOptions() + + except usage.UsageError, e: + test_name = tmp_test_case_object.name + log.err("There was an error in running %s!" % test_name) + log.err("%s" % e) options.opt_help() - raise usage.UsageError("Error parsing command line args for %s" - % tmp_test_case_object.name) - else: - return obj + raise usage.UsageError("Error in parsing command line args for %s" % test_name) + + if obj.requiresRoot: + try: + checkForRoot() + except NotRootError: + log.err("%s requires root to run" % obj.name) + sys.exit(1) + + return obj + +def isTestCase(obj): + try: + return issubclass(obj, NetTestCase) + except TypeError: + return False def findTestClassesFromFile(cmd_line_options): """ @@ -139,19 +103,13 @@ def findTestClassesFromFile(cmd_line_options): A list of class objects found in a file or module given on the commandline. """ - classes = [] filename = cmd_line_options['test'] - relative = filename.rsplit('/', 1)[1] - try: - module = filenameToModule(filename) - except ValueError, ve: - log.fail("%r doesn't exist." % relative) - else: - for name, val in inspect.getmembers(module): - if isTestCase(val): - classes.append(processTest(val, cmd_line_options)) - finally: - return classes + classes = [] + module = filenameToModule(filename) + for name, val in inspect.getmembers(module): + if isTestCase(val): + classes.append(processTest(val, cmd_line_options)) + return classes def makeTestCases(klass, tests, method_prefix): """ @@ -163,6 +121,9 @@ def makeTestCases(klass, tests, method_prefix): cases.append((klass, method_prefix+test)) return cases +class NoTestCasesFound(Exception): + pass + def loadTestsAndOptions(classes, cmd_line_options): """ Takes a list of test classes and returns their testcases and options. @@ -184,45 +145,8 @@ def loadTestsAndOptions(classes, cmd_line_options): return test_cases, options -def getTestTimeout(test_instance, test_method): - """ - Returns the timeout value set on this test. Check on the instance first, - the the class, then the module, then package. As soon as it finds - something with a timeout attribute, returns that. Returns the value set in - ooniprobe.conf, :attr:`ooni.config.advanced.default_timeout - <default_timeout>` if it cannot find anything. - - See twisted.trial.unittest.TestCase docstring for more details. - - @param test_instance: - The instance of a :class:`ooni.nettest.NetTestCase` currently running. - @param test_method: - The test_instance.test_method currently being processed. - """ - default = config.advanced.default_timeout - - try: - tm = getattr(test_instance, test_method) - except: - log.debug("runner.getTestTimeout() couldn't find %s.%s!" - % (test_instance, test_method)) - try: - return float(default) - except (ValueError, TypeError): - raise InvalidConfigFile("'default_timeout' must be a number!") - else: - test_instance._parents = [tm, test_instance] - test_instance._parents.extend(txutil.getPythonContainers(tm)) - timeout = txutil.acquireAttribute( - test_instance._parents, 'timeout', default) - try: - return float(timeout) - except (ValueError, TypeError): - log.warn("'timeout' attribute must be a number!") - return float(default) - def runTestCasesWithInput(test_cases, test_input, yaml_reporter, - oonib_reporter=None): + oonib_reporter=None): """ Runs in parallel all the test methods that are inside of the specified test case. Reporting happens every time a Test Method has concluded running. @@ -248,30 +172,8 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, # This is used to store a copy of all the test reports tests_report = {} - def test_timeout(d, test_instance): - timeout_error = defer.TimeoutError( - "%s test for %s timed out after %s seconds" - % (test_instance.name, test_instance.input, test_instance.timeout)) - timeout_fail = failure.Failure(err) - try: - d.errback(timeout_fail) - except defer.AlreadyCalledError: - # if the deferred has already been called but the *back chain is - # still unfinished, safely crash the reactor and report the timeout - reactor.crash() - test_instance._timedOut = True # see test_instance._wait - test_instance._test_result.addExpectedFailure(test_instance, fail) - test_timeout = txutils.suppressWarnings( - test_timeout, txutil.suppress(category=DeprecationWarning)) - - def test_skip_class(reason): - try: - d.errback(failure.Failure(SkipTest("%s" % reason))) - except defer.AlreadyCalledError: - pass # XXX not sure what to do here... - def test_done(result, test_instance, test_name): - log.msg("Successfully finished running %s" % test_name) + log.msg("Finished running %s" % test_name) log.debug("Deferred callback result: %s" % result) tests_report[test_name] = dict(test_instance.report) if not oonib_reporter: @@ -280,12 +182,9 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, d2 = yaml_reporter.testDone(test_instance, test_name) return defer.DeferredList([d1, d2]) - def test_error(error, test_instance, test_name): - if isinstance(error, SkipTest): - log.warn("%s" % error.message) - else: - log.err("Error in running %s" % test_name) - log.exception(error) + def test_error(failure, test_instance, test_name): + log.err("Error in running %s" % test_name) + log.exception(failure) return def tests_done(result, test_class): @@ -301,54 +200,29 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, d1 = oonib_reporter.testDone(test_instance, 'summary') d2 = yaml_reporter.testDone(test_instance, 'summary') return defer.DeferredList([d1, d2]) - except nettest.NoPostProcessor: + except NoPostProcessor: log.debug("No post processor configured") return dl = [] for test_case in test_cases: + log.debug("Processing %s" % test_case[1]) test_class = test_case[0] test_method = test_case[1] - log.debug("%s: Setting up: %s" % (test_class.name, test_method)) + + log.msg("Running %s with %s..." % (test_method, test_input)) test_instance = test_class() test_instance.input = test_input test_instance.report = {} - - # XXX txreporter.TestResult is expected by test_timeout(), but we - # should eventually replace it with a stub class - test_instance._test_result = txreporter.TestResult() # use this to keep track of the test runtime test_instance._start_time = time.time() # call setups on the test test_instance._setUp() test_instance.setUp() - - # get the timeout and _parents, in case it was set in setUp() - test_instance.timeout = getTestTimeout(test_instance, test_method) - test_instance.timedOut = False - test = getattr(test_instance, test_method) - test_instance._testMethod = test d = defer.maybeDeferred(test) - - # register the timer with the reactor - call_timeout = reactor.callLater(test_instance.timeout, test_timeout, d, - test_instance) - d.addBoth(lambda x: call_timeout.active() and call_timeout.cancel() or x) - - # check if anything has been aborted or marked as 'skip' - if hasattr(test_instance.__class__, 'skip'): - reason = getattr(test_instance.__class__, 'skip') - else: - reason = txutil.acquireAttribute(test_instance._parents, 'skip', None) - if reason is not None: - log.warn("%s marked some tests to be skipped. Reason: %s" - % (test_instance.name, reason)) - call_skip = reactor.callLater(0, test_skip_class, reason) - d.addBoth(lambda x: call_skip.active() and call_skip.cancel() or x) - d.addCallback(test_done, test_instance, test_method) d.addErrback(test_error, test_instance, test_method) dl.append(d) @@ -357,8 +231,8 @@ def runTestCasesWithInput(test_cases, test_input, yaml_reporter, test_methods_d.addCallback(tests_done, test_cases[0][0]) return test_methods_d -def runTestCasesWithInputUnit(test_cases, input_unit, yaml_reporter, - oonib_reporter): +def runTestCasesWithInputUnit(test_cases, input_unit, yaml_reporter, + oonib_reporter): """ Runs the Test Cases that are given as input parallely. A Test Case is a subclass of ooni.nettest.NetTestCase and a list of @@ -367,21 +241,26 @@ def runTestCasesWithInputUnit(test_cases, input_unit, yaml_reporter, The deferred list will fire once all the test methods have been run once per item in the input unit. - @param test_cases: - A tuple containing the test_class and test_method as strings. - @param input_unit: - A generator that contains the inputs to be run on the test. - @return: - A DeferredList containing all the tests to be run at this time. + test_cases: A list of tuples containing the test class and the test method as a string. + + input_unit: A generator that yields an input per iteration + """ + log.debug("Running test cases with input unit") dl = [] for test_input in input_unit: - log.debug("Running test with this input %s" % str(test_input)) + log.debug("Running test with this input %s" % test_input) d = runTestCasesWithInput(test_cases, test_input, yaml_reporter, oonib_reporter) dl.append(d) return defer.DeferredList(dl) +class InvalidResumeFile(Exception): + pass + +class noResumeSession(Exception): + pass + def loadResumeFile(): """ Sets the singleton stateDict object to the content of the resume file. @@ -390,6 +269,7 @@ def loadResumeFile(): Raises: :class:ooni.runner.InvalidResumeFile if the resume file is not valid + """ if not config.stateDict: try: @@ -425,6 +305,7 @@ def resumeTest(test_filename, input_unit_factory): :class:ooni.inputunit.InputUnitFactory that is at the index of the previous test run. + """ try: idx = config.stateDict[test_filename] @@ -445,7 +326,7 @@ def resumeTest(test_filename, input_unit_factory): @defer.inlineCallbacks def updateResumeFile(test_filename): """ - Update the resume file with the current stateDict state. + update the resume file with the current stateDict state. """ log.debug("Acquiring lock for %s" % test_filename) yield config.resume_lock.acquire() @@ -467,17 +348,20 @@ def increaseInputUnitIdx(test_filename): including the .py extension. input_unit_idx (int): the current input unit index for the test. + """ config.stateDict[test_filename] += 1 yield updateResumeFile(test_filename) -def updateProgressMeters(test_filename, input_unit_factory, test_case_number): - """Update the progress meters for keeping track of test state.""" +def updateProgressMeters(test_filename, input_unit_factory, + test_case_number): + """ + Update the progress meters for keeping track of test state. + """ if not config.state.test_filename: config.state[test_filename] = Storage() - per_item_avg = float(2) - config.state[test_filename].per_item_average = per_item_avg + config.state[test_filename].per_item_average = 2.0 input_unit_idx = float(config.stateDict[test_filename]) input_unit_items = float(len(input_unit_factory) + 1) @@ -485,36 +369,29 @@ def updateProgressMeters(test_filename, input_unit_factory, test_case_number): total_iterations = input_unit_items * test_case_number current_iteration = input_unit_idx * test_case_number - log.debug("Total InputUnits: %s" % input_unit_items) + log.debug("input_unit_items: %s" % input_unit_items) + log.debug("test_case_number: %s" % test_case_number) + log.debug("Test case number: %s" % test_case_number) log.debug("Total iterations: %s" % total_iterations) log.debug("Current iteration: %s" % current_iteration) def progress(): - current_progress = (current_iteration / total_iterations) * 100.0 - while float(current_progress) < float(100): - return current_progress + return (current_iteration / total_iterations) * 100.0 + config.state[test_filename].progress = progress def eta(): - return (total_iterations - current_iteration) * per_item_avg + return (total_iterations - current_iteration) \ + * config.state[test_filename].per_item_average config.state[test_filename].eta = eta config.state[test_filename].input_unit_idx = input_unit_idx config.state[test_filename].input_unit_items = input_unit_items + @defer.inlineCallbacks def runTestCases(test_cases, options, cmd_line_options): - """ - Run all test cases found in specified files and modules. - - @param test_cases: - A list of tuples, each tuple in containing the test_class - and test_method to run. - @param cmd_line_options: - The parsed :attr:`twisted.python.usage.Options.optParameters` - obtained from the main ooni commandline. - """ log.debug("Running %s" % test_cases) log.debug("Options %s" % options) log.debug("cmd_line_options %s" % dict(cmd_line_options)) @@ -576,6 +453,9 @@ def runTestCases(test_cases, options, cmd_line_options): log.exception("Problem in running test") yaml_reporter.finish() +class UnableToStartTor(Exception): + pass + def startTor(): """ Starts Tor Launches a Tor with :param: socks_port :param: control_port @@ -651,7 +531,7 @@ def startSniffing(): from ooni.utils.txscapy import ScapyFactory, ScapySniffer try: checkForRoot() - except PermissionsError: + except NotRootError: print "[!] Includepcap options requires root priviledges to run" print " you should run ooniprobe as root or disable the options in ooniprobe.conf" sys.exit(1) @@ -659,8 +539,7 @@ def startSniffing(): print "Starting sniffer" config.scapyFactory = ScapyFactory(config.advanced.interface) - pcapfile = config.reports.pcap - if pcapfile and os.path.exists(pcapfile): + if os.path.exists(config.reports.pcap): print "Report PCAP already exists with filename %s" % config.reports.pcap print "Renaming files with such name..." pushFilenameStack(config.reports.pcap) @@ -677,12 +556,6 @@ def loadTest(cmd_line_options): # Ideally this would get all wrapped in a nice little class that get's # instanced with it's cmd_line_options as an instance attribute classes = findTestClassesFromFile(cmd_line_options) - try: - test_cases, options = loadTestsAndOptions(classes, cmd_line_options) - return test_cases, options, cmd_line_options - except NoTestCasesFound, ntcf: - log.err(ntcf) - if not 'testdeck' in cmd_line_options: # exit if this was this only test - sys.exit(1) # file and there aren't any tests - else: - pass # there are more tests, so continue + test_cases, options = loadTestsAndOptions(classes, cmd_line_options) + + return test_cases, options, cmd_line_options diff --git a/ooni/utils/__init__.py b/ooni/utils/__init__.py index efa609a..8510a3b 100644 --- a/ooni/utils/__init__.py +++ b/ooni/utils/__init__.py @@ -1,10 +1,10 @@ -import imp -import os import logging import string import random import glob import yaml +import imp +import os class Storage(dict): """ @@ -48,18 +48,16 @@ class Storage(dict): for (k, v) in value.items(): self[k] = v -class PermissionsError(Exception): - """This test requires administrator or root permissions.""" +class NotRootError(Exception): + pass def checkForRoot(): - """Check permissions.""" if os.getuid() != 0: - raise PermissionsError + raise NotRootError("This test requires root") def randomSTR(length, num=True): """ - Returns a random, all-uppercase, alpha-numeric (if num=True), string of - specified character length. + Returns a random all uppercase alfa-numerical (if num True) string long length """ chars = string.ascii_uppercase if num: @@ -68,8 +66,7 @@ def randomSTR(length, num=True): def randomstr(length, num=True): """ - Returns a random, all-lowercase, alpha-numeric (if num=True), string - specified character length. + Returns a random all lowercase alfa-numerical (if num True) string long length """ chars = string.ascii_lowercase if num: @@ -78,14 +75,15 @@ def randomstr(length, num=True): def randomStr(length, num=True): """ - Returns a random a mixed lowercase, uppercase, alpha-numeric (if num=True) - string of specified character length. + Returns a random a mixed lowercase, uppercase, alfanumerical (if num True) + string long length """ chars = string.ascii_lowercase + string.ascii_uppercase if num: chars += string.digits return ''.join(random.choice(chars) for x in range(length)) + def pushFilenameStack(filename): """ Takes as input a target filename and checks to see if a file by such name diff --git a/ooni/utils/geodata.py b/ooni/utils/geodata.py index d0e730e..d9883ba 100644 --- a/ooni/utils/geodata.py +++ b/ooni/utils/geodata.py @@ -1,12 +1,3 @@ -# -*- encoding: utf-8 -*- -# -# geodata.py -# ********** -# In here go functions related to the understanding of -# geographical information of the probe -# -# :licence: see LICENSE - import re import os @@ -39,12 +30,11 @@ def IPToLocation(ipaddr): asn_dat = pygeoip.GeoIP(asn_file) location['asn'] = asn_dat.org_by_addr(ipaddr) + except IOError: - try: - raise GeoIPDataFilesNotFound( - "Couldn't find GeoIP files. Go to ./data and run \"make geoip\".") - except GeoIPDataFilesNotFound, gnf: - log.err(gnf) + log.err("Could not find GeoIP data files. Go into data/ " + "and run make geoip") + raise GeoIPDataFilesNotFound return location diff --git a/ooni/utils/hacks.py b/ooni/utils/hacks.py index 4bbdf48..64b5a53 100644 --- a/ooni/utils/hacks.py +++ b/ooni/utils/hacks.py @@ -1,13 +1,5 @@ -# -*- encoding: utf-8 -*- -# -# hacks.py -# ******** # When some software has issues and we need to fix it in a # hackish way, we put it in here. This one day will be empty. -# -# :authors: Arturo Filastò, Isis Lovecruft -# :licence: see LICENSE - import copy_reg diff --git a/ooni/utils/log.py b/ooni/utils/log.py index 95163f3..0740c10 100644 --- a/ooni/utils/log.py +++ b/ooni/utils/log.py @@ -1,13 +1,7 @@ -# -*- encoding: utf-8 -*- -# -# :authors: Arturo Filastò -# :licence: see LICENSE - -from functools import wraps import sys import os -import traceback import logging +import traceback from twisted.python import log as txlog from twisted.python import util @@ -17,6 +11,9 @@ from twisted.python.logfile import DailyLogFile from ooni import otime from ooni import config +## Get rid of the annoying "No route found for +## IPv6 destination warnings": +logging.getLogger("scapy.runtime").setLevel(logging.ERROR) class LogWithNoPrefix(txlog.FileLogObserver): def emit(self, eventDict): @@ -50,16 +47,12 @@ def stop(): def msg(msg, *arg, **kw): print "%s" % msg -def debug(message, *arg, **kw): +def debug(msg, *arg, **kw): if config.advanced.debug: - print "[D] %s" % message - -def warn(message, *arg, **kw): - if config.advanced.show_warnings: - print "[W] %s" % message + print "[D] %s" % msg -def err(message, *arg, **kw): - print "[!] %s" % message +def err(msg, *arg, **kw): + print "[!] %s" % msg def exception(error): """ @@ -72,27 +65,6 @@ def exception(error): exc_type, exc_value, exc_traceback = sys.exc_info() traceback.print_exception(exc_type, exc_value, exc_traceback) -def fail(*failure): - logging.critical(failure) - -def catch(func): - """ - Quick wrapper to add around test methods for debugging purposes, - catches the given Exception. Use like so: - - @log.catcher - def foo(bar): - if bar == 'baz': - raise Exception("catch me no matter what I am") - foo("baz") - """ - def _catch(*args, **kwargs): - try: - func(*args, **kwargs) - except Exception, exc: - exception(exc) - return _catch - class LoggerFactory(object): """ This is a logger factory to be used by oonib diff --git a/ooni/utils/net.py b/ooni/utils/net.py index 2495197..824d720 100644 --- a/ooni/utils/net.py +++ b/ooni/utils/net.py @@ -1,39 +1,17 @@ -# -*- encoding: utf-8 -*- -# -# net.py -# -------- -# OONI utilities for network infrastructure and hardware. -# -# :authors: Isis Lovecruft, Arturo Filasto -# :version: 0.0.1-pre-alpha -# :license: (c) 2012 Isis Lovecruft, Arturo Filasto -# see attached LICENCE file - import sys import socket from random import randint -from ipaddr import IPAddress from zope.interface import implements from twisted.internet import protocol, defer from twisted.internet import threads, reactor from twisted.web.iweb import IBodyProducer -from scapy.all import utils from ooni.utils import log, txscapy -from ooni.utils import PermissionsError #if sys.platform.system() == 'Windows': # import _winreg as winreg -PLATFORMS = {'LINUX': sys.platform.startswith("linux"), - 'OPENBSD': sys.platform.startswith("openbsd"), - 'FREEBSD': sys.platform.startswith("freebsd"), - 'NETBSD': sys.platform.startswith("netbsd"), - 'DARWIN': sys.platform.startswith("darwin"), - 'SOLARIS': sys.platform.startswith("sunos"), - 'WINDOWS': sys.platform.startswith("win32")} - userAgents = [ ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6", "Firefox 2.0, Windows XP"), ("Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)", "Internet Explorer 7, Windows Vista"), @@ -48,13 +26,23 @@ userAgents = [ ("Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20060127 Netscape/8.1", "Netscape 8.1, Windows XP") ] - class UnsupportedPlatform(Exception): """Support for this platform is not currently available.""" class IfaceError(Exception): """Could not find default network interface.""" +class PermissionsError(SystemExit): + """This test requires admin or root privileges to run. Exiting...""" + +PLATFORMS = {'LINUX': sys.platform.startswith("linux"), + 'OPENBSD': sys.platform.startswith("openbsd"), + 'FREEBSD': sys.platform.startswith("freebsd"), + 'NETBSD': sys.platform.startswith("netbsd"), + 'DARWIN': sys.platform.startswith("darwin"), + 'SOLARIS': sys.platform.startswith("sunos"), + 'WINDOWS': sys.platform.startswith("win32")} + class StringProducer(object): implements(IBodyProducer) @@ -112,10 +100,6 @@ def getWindowsIfaces(): return ifup def getIfaces(platform_name=None): - """ - Determines if the client's OS is Windows or Posix based, and then calls - the appropriate function for retrieving interfaces. - """ client, test = getClientPlatform(platform_name) if client: if client == ('LINUX' or 'DARWIN') or client[-3:] == 'BSD': @@ -154,16 +138,8 @@ def randomFreePort(addr="127.0.0.1"): def checkInterfaces(ifaces=None, timeout=1): """ - Check given network interfaces to see that they can send and receive - packets. This is similar to :func:`getDefaultIface`, except that function - only retrieves the name of the interface which is associated with the LAN, - whereas this function validates tx/rx capabilities. - @param ifaces: - (optional) A dictionary in the form of ifaces['if_name'] = 'if_addr'. - @param timeout: - An integer specifying the number of seconds to timeout if - no reply is received for our pings. + A dictionary in the form of ifaces['if_name'] = 'if_addr'. """ try: from scapy.all import IP, ICMP @@ -200,15 +176,6 @@ def checkInterfaces(ifaces=None, timeout=1): raise IfaceError def getNonLoopbackIfaces(platform_name=None): - """ - Get the iface names of all client network interfaces which are not - the loopback interface, regardless of whether they route to internal - or external networks. - - @param platform_name: (optional) The client interface, if known. Should - be given precisely as listed in ooni.utils.net.PLATFORMS. - @return: A list of strings of non-loopback iface names. - """ try: ifaces = getIfaces(platform_name) except UnsupportedPlatform, up: @@ -229,75 +196,8 @@ def getNonLoopbackIfaces(platform_name=None): else: return interfaces -def getNetworksFromRoutes(): - """ - Get the networks this client is current on from the kernel routing table. - Each network is returned as a :class:`ipaddr.IPNetwork`, with the - network range as the name of the network, i.e.: - - network.compressed = '127.0.0.1/32' - network.netmask = IPv4Address('255.0.0.0') - network.ipaddr = IPv4Address('127.0.0.1') - network.gateway = IPv4Address('0.0.0.0') - network.iface = 'lo' - - This is mostly useful for retrieving the default network interface in a - portable manner, though it could be used to conduct local network checks - for things like rogue DHCP servers, or perhaps test that the clients NAT - router is not the mistakenly the source of a perceived censorship event. - - @return: A list of :class:`ipaddr.IPNetwork` objects with routing table - information. - """ - from scapy.all import conf, ltoa, read_routes - from ipaddr import IPNetwork, IPAddress - - conf.verb = 0 # Hide the warnings - networks = [] - for nw, nm, gw, iface, addr in read_routes(): - n = IPNetwork( ltoa(nw) ) - (n.netmask, n.gateway, n.ipaddr) = [ IPAddress(x) for x in - [nm, gw, addr] ] - n.iface = iface - if not n.compressed in networks: - networks.append(n) - return networks - -def getDefaultIface(): - """ - Get the client's default network interface. - - @return: A string containing the name of the default working interface. - @raise IfaceError: If no working interface is found. - """ - networks = getNetworksFromRoutes() - for net in networks: - if net.is_private: - return net.iface - raise IfaceError def getLocalAddress(): - """ - Get the rfc1918 IP address of the default working network interface. - - @return: The properly-formatted, validated, local IPv4/6 address of the - client's default working network interface. - """ default_iface = getDefaultIface() return default_iface.ipaddr -def checkIPandPort(raw_ip, raw_port): - """ - Check that IP and Port are a legitimate address and portnumber. - - @return: The validated ip and port, else None. - """ - try: - port = int(raw_port) - assert port in xrange(1, 65535), "Port out of range." - ip = IPAddress(raw_ip) ## either IPv4 or IPv6 - except Exception, e: - log.err(e) - return - else: - return ip.compressed, port diff --git a/ooni/utils/txscapy.py b/ooni/utils/txscapy.py index 7fa31fa..62bde94 100644 --- a/ooni/utils/txscapy.py +++ b/ooni/utils/txscapy.py @@ -1,12 +1,3 @@ -# -*- coding: utf-8 -*- -# -# txscapy -# ******* -# Here shall go functions related to using scapy with twisted. -# -# This software has been written to be part of OONI, the Open Observatory of -# Network Interference. More information on that here: http://ooni.nu/ - import struct import socket import os @@ -18,21 +9,25 @@ from twisted.internet import reactor, threads, error from twisted.internet import defer, abstract from zope.interface import implements -from scapy.all import BasePacketList, conf, PcapReader -from scapy.all import Gen, SetGen, MTU -from scapy.error import Scapy_Exception +from scapy.config import conf -from ooni import config from ooni.utils import log +from ooni import config try: + conf.use_pcap = True + conf.use_dnet = True + from scapy.all import PcapWriter from scapy.arch import pcapdnet + config.pcap_dnet = True - conf.use_pcap = True - conf.use_dnet = True -except ImportError: - log.err("pypcap or dnet not installed. Certain tests may not work.") + from scapy.all import Gen, SetGen, MTU + +except ImportError, e: + log.err("pypcap or dnet not installed. " + "Certain tests may not work.") + config.pcap_dnet = False conf.use_pcap = False conf.use_dnet = False @@ -40,16 +35,12 @@ except ImportError: class DummyPcapWriter: def __init__(self, pcap_filename, *arg, **kw): log.err("Initializing DummyPcapWriter. We will not actually write to a pcapfile") + def write(self): pass - PcapWriter = DummyPcapWriter + PcapWriter = DummyPcapWriter -class ProtocolNotRegistered(Exception): - pass - -class ProtocolAlreadyRegistered(Exception): - pass def getNetworksFromRoutes(): @@ -57,6 +48,9 @@ def getNetworksFromRoutes(): from scapy.all import conf, ltoa, read_routes from ipaddr import IPNetwork, IPAddress + ## Hide the 'no routes' warnings + conf.verb = 0 + networks = [] for nw, nm, gw, iface, addr in read_routes(): n = IPNetwork( ltoa(nw) ) @@ -81,6 +75,11 @@ def getDefaultIface(): return net.iface raise IfaceError +class ProtocolNotRegistered(Exception): + pass + +class ProtocolAlreadyRegistered(Exception): + pass class ScapyFactory(abstract.FileDescriptor): """ @@ -88,27 +87,14 @@ class ScapyFactory(abstract.FileDescriptor): https://github.com/enki/muXTCP/blob/master/scapyLink.py """ def __init__(self, interface, super_socket=None, timeout=5): + abstract.FileDescriptor.__init__(self, reactor) if interface == 'auto': interface = getDefaultIface() if not super_socket: - try: - # scapy is missing an import in /scapy/arch/linux.py - # see /ooni/lib/000-scapy-missing-exc.patch - super_socket = conf.L3socket(iface=interface, - promisc=True, filter='') - #super_socket = conf.L2socket(iface=interface) - except NameError, ne: - raise Scapy_Exception("Filter parse error") - except Scapy_Exception, se: - log.err("txscapy: %s" % se.message) - log.debug("txscapy: Trying socket setup again without filter") - try: - super_socket = conf.L3socket(iface=interface, - promisc=True) - except: - log.err("txscapy: Socket setup failed, giving up...") - raise sys.exit(1) + super_socket = conf.L3socket(iface=interface, + promisc=True, filter='') + #super_socket = conf.L2socket(iface=interface) self.protocols = [] fdesc._setCloseOnExec(super_socket.ins.fileno()) @@ -167,13 +153,9 @@ class ScapyProtocol(object): raise NotImplementedError class ScapySender(ScapyProtocol): - - # This deferred will fire when we have finished sending and receiving - # packets. timeout = 5 - if config.advanced.default_timeout: - timeout = int(config.advanced.default_timeout) + # This deferred will fire when we have finished sending a receiving packets. # Should we look for multiple answers for the same sent packet? multi = False @@ -197,7 +179,7 @@ class ScapySender(ScapyProtocol): break if len(self.answered_packets) == len(self.sent_packets): - # All of our questions have been answered. + log.debug("All of our questions have been answered.") self.stopSending() return @@ -254,10 +236,7 @@ class ScapySender(ScapyProtocol): class ScapySniffer(ScapyProtocol): def __init__(self, pcap_filename, *arg, **kw): - # The "str(pcap_filename)" explicit typing is due to an error where - # scapy.utils.PcapWriter expects strings, and it's getting unicode - # due to the "# -*- coding: utf-8 -*-"...this might be a problem... - self.pcapwriter = PcapWriter(str(pcap_filename), *arg, **kw) + self.pcapwriter = PcapWriter(pcap_filename, *arg, **kw) def packetReceived(self, packet): self.pcapwriter.write(packet) diff --git a/oonib/README.md b/oonib/README.md index d11f876..27c1163 100644 --- a/oonib/README.md +++ b/oonib/README.md @@ -5,12 +5,8 @@ The extra dependencies necessary to run OONIB are: * twisted-names * cyclone: https://github.com/fiorix/cyclone -We recommend that you use a python virtualenv. See OONI's README.md. - # Generate self signed certs for OONIB -If you want to use the HTTPS test helper, you will need to create a certificate: - openssl genrsa -des3 -out private.key 4096 openssl req -new -key private.key -out server.csr cp private.key private.key.org @@ -19,99 +15,9 @@ If you want to use the HTTPS test helper, you will need to create a certificate: openssl x509 -req -days 365 -in server.csr -signkey private.key -out certificate.crt rm private.key.org -Don't forget to update oonib/config.py options helpers.ssl.private_key and -helpers.ssl.certificate - # Redirect low ports with iptables -The following iptables commands will map connections on low ports to those bound by oonib - # Map port 80 to config.helpers.http_return_request.port (default: 57001) iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 57001 # Map port 443 to config.helpers.ssl.port (default: 57006) iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 57006 - # Map port 53 udp to config.helpers.dns.udp_port (default: 57004) - iptables -t nat -A PREROUTING -p tcp -m udp --dport 53 -j REDIRECT --tor-ports - # Map port 53 tcp to config.helpers.dns.tcp_port (default: 57005) - iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --tor-ports - -# Install Tor (Debian). - -You will need a Tor binary on your system. For complete instructions, see also: - - https://www.torproject.org/docs/tor-doc-unix.html.en - https://www.torproject.org/docs/rpms.html.en - -Add this line to your /etc/apt/sources.list, replacing <DISTRIBUTION> -where appropriate: - - deb http://deb.torproject.org/torproject.org <DISTRIBUTION> main - -Add the Tor Project gpg key to apt: - - gpg --keyserver keys.gnupg.net --recv 886DDD89 - gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - - # Update apt and install the torproject keyring, tor, and geoipdb - apt-get update - apt-get install deb.torproject.org-keyring tor tor-geoipdb - -# Update ooni-probe/oonib/config.py - - Set config.main.tor_binary to your Tor path - Set config.main.tor2webmode = False - -# (For Experts Only) Tor2webmode: - -WARNING: provides no anonymity! Use only if you know what you are doing! -Tor2webmode will improve the performance of the collector Hidden Service -by discarding server-side anonymity. - -You will need to build Tor from source. At the time of writing, the latest stable Tor is tor-0.2.3.25. You should use the most recent stable Tor. - -Example: - - git clone https://git.torproject.org/tor.git - git checkout tor-0.2.3.25 - git verify-tag -v tor-0.2.3.25 - -You should see: - - object 17c24b3118224d6536c41fa4e1493a831fb29f0a - type commit - tag tor-0.2.3.25 - tagger Roger Dingledine <arma(a)torproject.org> 1353399116 -0500 - - tag 0.2.3.25 - gpg: Signature made Tue 20 Nov 2012 08:11:59 AM UTC using RSA key ID 19F78451 - gpg: Good signature from "Roger Dingledine <arma(a)mit.edu>" - gpg: aka "Roger Dingledine <arma(a)freehaven.net>" - gpg: aka "Roger Dingledine <arma(a)torproject.org>" - -It is always good idea to verify. - - gpg --fingerprint 19F78451 - pub 4096R/19F78451 2010-05-07 - Key fingerprint = F65C E37F 04BA 5B36 0AE6 EE17 C218 5258 19F7 8451 - uid Roger Dingledine <arma(a)mit.edu> - uid Roger Dingledine <arma(a)freehaven.net> - uid Roger Dingledine <arma(a)torproject.org> - sub 4096R/9B11185C 2012-05-02 [expires: 2013-05-02] - -Build Tor with enable-tor2web-mode - - ./autogen.sh ; ./configure --enable-tor2web-mode ; make - -Copy the tor binary from src/or/tor somewhere and set the corresponding -options in oonib/config.py - -# To launch oonib on system boot - -To launch oonib on startup, you may want to use supervisord (www.supervisord.org) -The following supervisord config will use the virtual environment in -/home/ooni/venv_oonib and start oonib on boot: - - [program:oonib] - command=/home/ooni/venv_oonib/bin/python /home/ooni/ooni-probe/bin/oonib - autostart=true - user=oonib - directory=/home/oonib/ diff --git a/ooniprobe.conf.sample b/ooniprobe.conf.sample index 99faa4b..ed77dfe 100644 --- a/ooniprobe.conf.sample +++ b/ooniprobe.conf.sample @@ -17,21 +17,15 @@ privacy: # Should we collect a full packet capture on the client? includepcap: false advanced: - # XXX change this to point to the directory where you have stored the - # GeoIP database file. This should be the directory in which OONI is - # installed /path/to/ooni-probe/data/ + # XXX change this to point to the directory where you have stored the GeoIP + # database file. This should be the directory in which OONI is installed + # /path/to/ooni-probe/data/ geoip_data_dir: /usr/share/GeoIP/ - # Should we display debug messages?: debug: true - # Should we display warning messages?: - show_warnings: true - # How many seconds should we wait for connections before timing out?: - default_timeout: 30 - # Location where Tor is installed: tor_binary: '/usr/sbin/tor' - # To automatically detect the system default networking interface, use: + # For auto detection interface: auto - # Or specify a specific interface: + # Of specify a specific interface #interface: wlan0 # If you do not specify start_tor, you will have to have Tor running and # explicitly set the control port and orport. diff --git a/scripts/before_i_commit.sh b/scripts/before_i_commit.sh index a504ad8..918b137 100755 --- a/scripts/before_i_commit.sh +++ b/scripts/before_i_commit.sh @@ -1,4 +1,4 @@ -#!/bin/sh +#!/bin/bash # This script should be run before you commit to verify that the basic tests # are working as they should # Once you have run it you can inspect the log file via @@ -32,12 +32,11 @@ fi echo "Below you should not see anything" echo "---------------------------------" -[ -f before_i_commit.log ] && grep "Error: " before_i_commit.log +grep "Error: " before_i_commit.log echo "---------------------------------" echo "If you do, it means something is wrong." echo "Read through the log file and fix it." echo "If you are having some problems fixing some things that have to do with" echo "the core of OONI, let's first discuss it on IRC, or open a ticket" read -cat *yamloo | less -rm -f *yamloo +#cat *.yamloo | less

1 0

[ooni-probe/master] Add draft ooniprobe API specification
by art＠torproject.org 22 Dec '12

22 Dec '12

commit 15b7c8593a1551b22923452aab87256ccb9687be Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Dec 21 20:39:13 2012 +0100 Add draft ooniprobe API specification (cherry picked from commit 24d36abfaec176afb28620bc36b93cdac8058ba0) --- docs/source/architecture.rst | 74 +++++++++++++++++++++++++++++++++++++++-- 1 files changed, 70 insertions(+), 4 deletions(-) diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index 7091c29..af92878 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -181,20 +181,86 @@ Draft API specification Through the ooniprobe API it will be possible to `start tests`_, `stop tests`_ and `monitor test progress`_. +List tests +.......... + +`GET /test` + +Shall return the list of available tests as an array. + +This is how a response looks like +:: + + [{'id': 'http_requests', + 'name': 'HTTP Requests Test', + 'description': 'This test perform a HTTP GET request for the / resource over the test network and over Tor', + 'type': [ 'blocking' ], + 'version': '0.1', + 'arguments': { + 'urllist': 'Specify the list of URLs to be used for the test' + } + }] + +*type* may be either **blocking** or **manipulation**. + Start tests ........... -.. TODO + +`POST /test/<test_id>/start` + +Is used to start a test with the specified test_id. + +Inside of the request you will specify the arguments supported by the test + +This is how a request could look like +:: + { + 'urllist': + ['http://google.com/', 'http://torproject.org/'] + } + +The server will then respond with the test object +:: + { + 'status': 'running', + 'percentage': 0, + 'current_input': 'http://google.com/', + 'urllist': + ['http://google.com/', 'http://torproject.org/'] + } + Stop tests ........... -.. TODO +`POST /test/<test_id>/stop` + +This will terminate the execution of the test with the specified test_id. + +The request may optionally contain a reason for stopping the test such as +:: + { + 'reason': 'some reason' + } Monitor test progress -...................... +..................... + +`GET /test/<test_id>` + +Will return the status of a test + +Like so for example +:: + { + 'status': 'running', + 'percentage': 0, + 'current_input': 'http://google.com/', + 'urllist': + ['http://google.com/', 'http://torproject.org/'] + } -.. TODO Implementation status =====================

1 0

[ooni-probe/master] Add birdseye view of the ooniprobe architecture
by art＠torproject.org 22 Dec '12

22 Dec '12

commit 7f7d8a63e499ccb7e4ca7b6576d5a6f83eb78574 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Dec 21 20:38:38 2012 +0100 Add birdseye view of the ooniprobe architecture (cherry picked from commit 07329ef9e2ad5a54af8d66aa97669bfeeb359916) --- .../_static/images/ooniprobe-architecture.png | Bin 0 -> 96574 bytes docs/source/architecture.rst | 5 +++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/_static/images/ooniprobe-architecture.png b/docs/source/_static/images/ooniprobe-architecture.png new file mode 100644 index 0000000..06065fc Binary files /dev/null and b/docs/source/_static/images/ooniprobe-architecture.png differ diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index f0736ce..cb50402 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -4,8 +4,6 @@ Architecture :Contact: art(a)torproject.org :Copyright: This document has been placed in the public domain. -.. [[Image(https://github.com/hellais/ooni-docsnspecs/raw/master/graphics/CnCVie…, width=550px, align=right)]] - The goal of this document is provide an overview of how ooni works, what are it's pieces and how they interact with one another. @@ -16,6 +14,9 @@ To get an idea of what is implemented and with what sort of quality see the The two main components of ooni are `oonib`_ and `ooniprobe`_. +.. image:: _static/images/ooniprobe-architecture.png + :width: 700px + ooniprobe ---------

1 0

[ooni-probe/master] Add details on the current implementation status of ooni*
by art＠torproject.org 22 Dec '12

22 Dec '12

commit 8d26efcff80f595556782c05effbd6e1fc715c83 Author: Arturo Filastò <art(a)fuffa.org> Date: Fri Dec 21 20:39:24 2012 +0100 Add details on the current implementation status of ooni* (cherry picked from commit 303788d7289ba3cf6a47f4de0a1fad9fa04048ee) --- docs/source/architecture.rst | 65 +++++++++++++++++++++++++++++++++++++++++- 1 files changed, 64 insertions(+), 1 deletions(-) diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst index af92878..f0736ce 100644 --- a/docs/source/architecture.rst +++ b/docs/source/architecture.rst @@ -265,6 +265,69 @@ Like so for example Implementation status ===================== -.. TODO +ooniprobe +......... + +**Reporting** + + * To flat YAML file: *alpha* + + * To remote httpo backend: *alpha* + +**Test templates** + + * HTTP test template: *alpha* + + * Scapy test template: *alpha* + + * DNS test template: *alpha* + + * TCP test template: *prototype* + +**Tests** + +To see the list of implemented tests see: +https://ooni.torproject.org/docs/#core-ooniprobe-tests + +**ooniprobe API** + + * Specification: *draft* + + * HTTP API: *not implemented* + +**ooniprobe HTML5/JS user interface** + + Not implemented. + +**ooniprobe build system** + + Not implemented. + +**ooniprobe command line interface** + + Implemented in alpha quality, though needs to be ported to use the HTTP based + API. + +oonib +..... + +**Collector** + + * collection of YAML reports to flat file: *alpha* + + * collection of pcap reports: *not implemented* + + * association of reports with test helpers: *not implemented* + +**Test helpers** + + * HTTP Return JSON Helper: *alpha* + + * DNS Test helper: *prototype* + + * Test Helper - collector mapping: *Not implemented* + + * TCP Test helper: *prototype* + * Daphn3 Test helper: *prototype*

1 0

[ooni-probe/master] Update oonib README.md
by art＠torproject.org 22 Dec '12

22 Dec '12

commit 2b7112eb06c285545b654ba5ec0a455b403a20f8 Author: aagbsn <aagbsn(a)extc.org> Date: Wed Dec 12 16:34:41 2012 +0000 Update oonib README.md Add insructions for configuring oonib, additional iptables rules for redirecting tcp and udp dns, and building Tor with tor2webmode. (cherry picked from commit 41fb481e55a353f72779590b38a76e34cb25960c) --- oonib/README.md | 79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 79 insertions(+), 0 deletions(-) diff --git a/oonib/README.md b/oonib/README.md index 27c1163..3192e52 100644 --- a/oonib/README.md +++ b/oonib/README.md @@ -5,6 +5,11 @@ The extra dependencies necessary to run OONIB are: * twisted-names * cyclone: https://github.com/fiorix/cyclone +We recommend that you use a python virtualenv. See OONI's README.md. + +#XXX: add instructions for isolating the python environment, sandboxing +#XXX: see pypy-sandbox + # Generate self signed certs for OONIB openssl genrsa -des3 -out private.key 4096 @@ -21,3 +26,77 @@ The extra dependencies necessary to run OONIB are: iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 57001 # Map port 443 to config.helpers.ssl.port (default: 57006) iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 57006 + # Map port 53 udp to config.helpers.dns.udp_port (default: 57004) + iptables -t nat -A PREROUTING -p tcp -m udp --dport 53 -j REDIRECT --tor-ports + # Map port 53 tcp to config.helpers.dns.tcp_port (default: 57005) + iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --tor-ports + +# Install Tor (Debian). + +See also: https://www.torproject.org/docs/tor-doc-unix.html.en, +https://www.torproject.org/docs/rpms.html.en + +Add this line to your /etc/apt/sources.list, replacing <DISTRIBUTION> +where appropriate: + + deb http://deb.torproject.org/torproject.org <DISTRIBUTION> main + +Add the Tor Project gpg key to apt: + + gpg --keyserver keys.gnupg.net --recv 886DDD89 + gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - + # Update apt and install the torproject keyring, tor, and geoipdb + apt-get update + apt-get install deb.torproject.org-keyring tor tor-geoipdb + +# Edit ooni-probe/oonib/config.py and configure + + Set config.main.tor_binary to your Tor path + Set config.main.tor2webmode = False + +# (For Experts Only) To use Tor2webmode: + +WARNING: provides no anonymity! Use only if you know what you are doing! +Tor2webmode will improve the performance of the collector Hidden Service +by discarding server-side anonymity. + +You will need to build Tor from source. At the time of writing, the latest stable Tor is tor-0.2.3.25. You should use the most recent stable Tor. + +Example: + + git clone https://git.torproject.org/tor.git + git checkout tor-0.2.3.25 + git verify-tag -v tor-0.2.3.25 + +You should see: + + object 17c24b3118224d6536c41fa4e1493a831fb29f0a + type commit + tag tor-0.2.3.25 + tagger Roger Dingledine <arma(a)torproject.org> 1353399116 -0500 + + tag 0.2.3.25 + gpg: Signature made Tue 20 Nov 2012 08:11:59 AM UTC using RSA key ID 19F78451 + gpg: Good signature from "Roger Dingledine <arma(a)mit.edu>" + gpg: aka "Roger Dingledine <arma(a)freehaven.net>" + gpg: aka "Roger Dingledine <arma(a)torproject.org>" + +It is always good idea to verify. + + gpg --fingerprint 19F78451 + pub 4096R/19F78451 2010-05-07 + Key fingerprint = F65C E37F 04BA 5B36 0AE6 EE17 C218 5258 19F7 8451 + uid Roger Dingledine <arma(a)mit.edu> + uid Roger Dingledine <arma(a)freehaven.net> + uid Roger Dingledine <arma(a)torproject.org> + sub 4096R/9B11185C 2012-05-02 [expires: 2013-05-02] + +Build Tor with enable-tor2web-mode + + ./autogen.sh ; ./configure --enable-tor2web-mode ; make + +Copy the tor binary from src/or/tor somewhere and set the corresponding +options in oonib/config.py + +#XXX: add instructions for launching on boot +#XXX: see: supervisord, crontab @restart, etc

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits December 2012