June 2013 - tor-commits - lists.torproject.org

[translation/tsum] Update translations for tsum
by translation＠torproject.org 19 Jun '13

19 Jun '13

commit 7cfad3a841eff730f27b8243fb504a8a8691e569 Author: Translation commit bot <translation(a)torproject.org> Date: Wed Jun 19 15:15:19 2013 +0000 Update translations for tsum --- ka/short-user-manual_ka_noimg.xhtml | 132 +++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) diff --git a/ka/short-user-manual_ka_noimg.xhtml b/ka/short-user-manual_ka_noimg.xhtml new file mode 100644 index 0000000..681e5b7 --- /dev/null +++ b/ka/short-user-manual_ka_noimg.xhtml @@ -0,0 +1,132 @@ +<html xmlns="http://www.w3.org/1999/xhtml"> + <head> + <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> + <title/> + </head> + <body> + <h1 id="the-short-user-manual">The Short User Manual</h1> + <p>This user manual contains information about how to download Tor, how to use it, and what to do if Tor is unable to connect to the network. If you can't find the answer to your question in this document, email help(a)rt.torproject.org.</p> + <p>Please note that we are providing support on a voluntary basis, and we get a lot of emails every single day. There is no need to worry if we don't get back to you straight away.</p> + <h2 id="how-tor-works">How Tor works</h2> + <p>Tor is a network of virtual tunnels that allows you to improve your privacy and security on the Internet. Tor works by sending your traffic through three random servers (also known as <em>relays</em>) in the Tor network, before the traffic is sent out onto the public Internet.</p> + <p>The image above illustrates a user browsing to different websites over Tor. The green monitors represent relays in the Tor network, while the three keys represent the layers of encryption between the user and each relay.</p> + <p>Tor will anonymize the origin of your traffic, and it will encrypt everything between you and the Tor network. Tor will also encrypt your traffic inside the Tor network, but it cannot encrypt your traffic between the Tor network and its final destination.</p> + <p>If you are communicating sensitive information, for example when logging on to a website with a username and password, make sure that you are using HTTPS (e.g. <strong>https</strong>://torproject.org/, not <strong>http</strong>://torproject.org/).</p> + <h2 id="how-to-download-tor">How to download Tor</h2> + <p>The bundle we recommend to most users is the <a href="https://www.torproject.org/projects/torbrowser.html">Tor Browser Bundle</a>. This bundle contains a browser preconfigured to safely browse the Internet through Tor, and requires no installation. You download the bundle, extract the archive, and start Tor.</p> + <p>There are two different ways to get hold of the Tor software. You can either browse to the <a href="https://www.torproject.org/">Tor Project website</a> and download it there, or you can use GetTor, the email autoresponder.</p> + <h3 id="how-to-get-tor-via-email">How to get Tor via email</h3> + <p>To receive the English Tor Browser Bundle for Windows, send an email to gettor(a)torproject.org with <strong>windows</strong> in the body of the message. You can leave the subject blank.</p> + <p>You can also request the Tor Browser Bundle for Mac OS X (write <strong>macos-i386</strong>), and Linux (write <strong>linux-i386</strong> for 32-bit systems or <strong>linux-x86_64</strong> for 64-bit systems).</p> + <p>If you want a translated version of Tor, write <strong>help</strong> instead. You will then receive an email with instructions and a list of available languages.</p> + <p><strong>Note</strong>: The Tor Browser Bundles for Linux and Mac OS X are rather large, and you will not be able to receive any of these bundles with a Gmail, Hotmail or Yahoo account. If you cannot receive the bundle you want, send an email to help(a)rt.torproject.org and we will give you a list of website mirrors to use.</p> + <h3 id="tor-for-smartphones">Tor for smartphones</h3> + <p>You can get Tor on your Android device by installing the package named <em>Orbot</em>. For information about how to download and install Orbot, please see the <a href="https://www.torproject.org/docs/android.html.en">Tor Project website</a>.</p> + <p>We also have experimental packages for <a href="https://www.torproject.org/docs/N900.html.en">Nokia Maemo/N900</a> and <a href="http://sid77.slackware.it/iphone/">Apple iOS</a>.</p> + <h3 id="how-to-verify-that-you-have-the-right-version">How to verify that you have the right version</h3> + <p>Before running the Tor Browser Bundle, you should make sure that you have the right version.</p> + <p>The software you receive is accompanied by a file with the same name as the bundle and the extension <strong>.asc</strong>. This .asc file is a GPG signature, and will allow you to verify the file you've downloaded is exactly the one that we intended you to get.</p> + <p>Before you can verify the signature, you will have to download and install GnuPG:</p> + <p><strong>Windows</strong>: <a href="http://gpg4win.org/download.html">http://gpg4win.org/download.html</a><br/><strong>Mac OS X</strong>: <a href="http://macgpg.sourceforge.net/">http://macgpg.sourceforge.net/</a><br/><strong>Linux</strong>: Most Linux distributions come with GnuPG preinstalled.</p> + <p>Please note that you may need to edit the paths and the commands used below to get it to work on your system.</p> + <p>Erinn Clark signs the Tor Browser Bundles with key 0x63FEE659. To import Erinn's key, run:</p> + <pre> + <code>gpg --keyserver hkp://keys.gnupg.net --recv-keys 0x63FEE659 +</code> + </pre> + <p>After importing the key, verify that the fingerprint is correct:</p> + <pre> + <code>gpg --fingerprint 0x63FEE659 +</code> + </pre> + <p>You should see:</p> + <pre> + <code>pub 2048R/63FEE659 2003-10-16 + Key fingerprint = 8738 A680 B84B 3031 A630 F2DB 416F 0610 63FE E659 +uid Erinn Clark <erinn(a)torproject.org> +uid Erinn Clark <erinn(a)debian.org> + +uid Erinn Clark <erinn(a)double-helix.org> +sub 2048R/EB399FD7 2003-10-16 +</code> + </pre> + <p>To verify the signature of the package you downloaded, run the following command:</p> + <pre> + <code>gpg --verify tor-browser-2.2.33-2_en-US.exe.asc tor-browser-2.2.33-2_en-US.exe +</code> + </pre> + <p>The output should say <em>"Good signature"</em>. A bad signature means that the file may have been tampered with. If you see a bad signature, send details about where you downloaded the package from, how you verified the signature, and the output from GnuPG in an email to help(a)rt.torproject.org.</p> + <p>Once you have verified the signature and seen the <em>"Good signature"</em> output, go ahead and extract the package archive. You should then see a directory similar to <strong>tor-browser_en-US</strong>. Inside that directory is another directory called <strong>Docs</strong>, which contains a file called <strong>changelog</strong>. You want to make sure that the version number on the top line of the changelog file matches the version number in the filename.</p> + <h3 id="how-to-use-the-tor-browser-bundle">How to use the Tor Browser Bundle</h3> + <p>After downloading the Tor Browser Bundle and extracting the package, you should have a directory with a few files in it. One of the files is an executable called "Start Tor Browser" (or "start-tor-browser", depending on your operating system).</p> + <p>When you start the Tor Browser Bundle, you will first see Vidalia start up and connect you to the Tor network. After that, you will see a browser confirming that you are now using Tor. This is done by displaying <a href="https://check.torproject.org/">https://check.torproject.org/</a>. You can now browse the Internet through Tor.</p> + <p> + <em>Please note that it is important that you use the browser that comes with the bundle, and not your own browser.</em> + </p> + <h3 id="what-to-do-when-tor-does-not-connect">What to do when Tor does not connect</h3> + <p>Some users will notice that Vidalia gets stuck when trying to connect to the Tor network. If this happens, make sure that you are connected to the Internet. If you need to connect to a proxy server, see <em>How to use an open proxy</em> below.</p> + <p>If your normal Internet connection is working, but Tor still can't connect to the network, try the following; open the Vidalia control panel, click on <em>Message Log</em> and select the <em>Advanced</em> tab. It may be that Tor won't connect because:</p> + <p><strong>Your system clock is off</strong>: Make sure that the date and time on your system is correct, and restart Tor. You may need to synchronize your system clock with an Internet time server.</p> + <p><strong>You are behind a restrictive firewall</strong>: To tell Tor to only try port 80 and port 443, open the Vidalia control panel, click on <em>Settings</em> and <em>Network</em>, and tick the box that says <em>My firewall only lets me connect to certain ports</em>.</p> + <p><strong>Your anti-virus program is blocking Tor</strong>: Make sure that your anti-virus program is not preventing Tor from making network connections.</p> + <p>If Tor still doesn't work, it's likely that your Internet Service Provider (ISP) is blocking Tor. Very often this can be worked around with <strong>Tor bridges</strong>, hidden relays that aren't as easy to block.</p> + <p>If you need help with figuring out why Tor can't connect, send an email to help(a)rt.torproject.org and include the relevant parts from the log file.</p> + <h3 id="how-to-find-a-bridge">How to find a bridge</h3> + <p>To use a bridge, you will first have to locate one; you can either browse to <a href="https://bridges.torproject.org/">bridges.torproject.org</a>, or you can send an email to bridges(a)torproject.org. If you do send an email, please make sure that you write <strong>get bridges</strong> in the body of the email. Without this, you will not get a reply. Note that you need to send this email from either a gmail.com or a yahoo.com address.</p> + <p>Configuring more than one bridge address will make your Tor connection more stable, in case some of the bridges become unreachable. There is no guarantee that the bridge you are using now will work tomorrow, so you should make a habit of updating your list of bridges every so often.</p> + <h3 id="how-to-use-a-bridge">How to use a bridge</h3> + <p>Once you have a set of bridges to use, open the Vidalia control panel, click on <em>Settings</em>, <em>Network</em> and tick the box that says <em>My ISP blocks connections to the Tor network</em>. Enter the bridges in the box below, hit <em>OK</em> and start Tor again.</p> + <h3 id="how-to-use-an-open-proxy">How to use an open proxy</h3> + <p>If using a bridge does not work, try configuring Tor to use any HTTPS or SOCKS proxy to get access to the Tor network. This means even if Tor is blocked by your local network, open proxies can be safely used to connect to the Tor Network and on to the uncensored Internet.</p> + <p>The steps below assume you have a functional Tor/Vidalia configuration, and you have found a list of HTTPS, SOCKS4, or SOCKS5 proxies.</p> + <ol style="list-style-type: decimal"> + <li>Open the Vidalia control panel, click on <em>Settings</em>.</li> + <li>Click <em>Network</em>. Select <em>I use a proxy to access the Internet</em>.</li> + <li>On the <em>Address</em> line, enter the open proxy address. This can be a hostname or an IP Address.</li> + <li>Enter the port for the proxy.</li> + <li>Generally, you do not need a username and password. If you do, enter the information in the proper fields.</li> + <li>Choose the <em>Type</em> of proxy you are using, whether HTTP/HTTPS, SOCKS4, or SOCKS5.</li> + <li>Push the <em>OK</em> button. Vidalia and Tor are now configured to use a proxy to access the rest of the Tor network.</li> + </ol> + <h2 id="frequently-asked-questions">Frequently Asked Questions</h2> + <p>This section will answer some of the most common questions. If your question is not mentioned here, please send an email to help(a)rt.torproject.org.</p> + <h3 id="unable-to-extract-the-archive">Unable to extract the archive</h3> + <p>If you are using Windows and find that you cannot extract the archive, download and install <a href="http://www.7-zip.org/">7-Zip</a>.</p> + <p>If you are unable to download 7-Zip, try to rename the file from .z to .zip and use winzip to extract the archive. Before renaming the file, tell Windows to show file extensions:</p> + <h4 id="windows-xp">Windows XP</h4> + <ol style="list-style-type: decimal"> + <li>Open <em>My Computer</em></li> + <li>Click on <em>Tools</em> and choose <em>Folder Options...</em> in the menu</li> + <li>Click on the <em>View</em> tab</li> + <li>Uncheck <em>Hide extensions for known file types</em> and click <em>OK</em></li> + </ol> + <h4 id="windows-vista">Windows Vista</h4> + <ol style="list-style-type: decimal"> + <li>Open <em>Computer</em></li> + <li>Click on <em>Organize</em> and choose <em>Folder and search options</em> in the menu</li> + <li>Click on the <em>View</em> tab</li> + <li>Uncheck <em>Hide extensions for known file types</em> and click <em>OK</em></li> + </ol> + <h4 id="windows-7">Windows 7</h4> + <ol style="list-style-type: decimal"> + <li>Open <em>Computer</em></li> + <li>Click on <em>Organize</em> and choose <em>Folder and search options</em> in the menu</li> + <li>Click on the <em>View</em> tab</li> + <li>Uncheck <em>Hide extensions for known file types</em> and click <em>OK</em></li> + </ol> + <h3 id="vidalia-asks-for-a-password">Vidalia asks for a password</h3> + <p>You should not have to enter a password when starting Vidalia. If you are prompted for one, you are likely affected by one of these problems:</p> + <p><strong>You are already running Vidalia and Tor</strong>: For example, this situation can happen if you installed the Vidalia bundle and now you're trying to run the Tor Browser Bundle. In that case, you will need to close the old Vidalia and Tor before you can run this one.</p> + <p><strong>Vidalia crashed, but left Tor running</strong>: If the dialog that prompts you for a control password has a Reset button, you can click the button and Vidalia will restart Tor with a new random control password. If you do not see a Reset button, or if Vidalia is unable to restart Tor for you; go into your process or task manager, and terminate the Tor process. Then use Vidalia to restart Tor.</p> + <p>For more information, see the <a href="https://torproject.org/docs/faq.html#VidaliaPassword">FAQ</a> on the Tor Project website.</p> + <h3 id="flash-does-not-work">Flash does not work</h3> + <p>For security reasons, Flash, Java, and other plugins are currently disabled for Tor. Plugins operate independently from Firefox and can perform activity on your computer that ruins your anonymity.</p> + <p>Most YouTube videos work with HTML5, and it is possible to view these videos over Tor. You need to join the <a href="https://www.youtube.com/html5">HTML5 trial</a> on the YouTube website before you can use the HTML5 player.</p> + <p>Note that the browser will not remember that you joined the trial once you close it, so you will need to re-join the trial the next time you run the Tor Browser Bundle.</p> + <p>Please see the <a href="https://www.torproject.org/torbutton/torbutton-faq.html#noflash">Torbutton FAQ</a> for more information.</p> + <h3 id="i-want-to-use-another-browser">I want to use another browser</h3> + <p>For security reasons, we recommend that you only browse the web through Tor using the Tor Browser Bundle. It is technically possible to use Tor with other browsers, but by doing so you open yourself up to potential attacks.</p> + <h3 id="why-tor-is-slow">Why Tor is slow</h3> + <p>Tor can sometimes be a bit slower than your normal Internet connection. After all, your traffic is sent through many different countries, sometimes across oceans around the world!</p> + </body> +</html>

1 0

[translation/orbot] Update translations for orbot
by translation＠torproject.org 19 Jun '13

19 Jun '13

commit 4dbcc1c4f63b8631b169aa106ad9cc0cf5a71cc1 Author: Translation commit bot <translation(a)torproject.org> Date: Wed Jun 19 15:15:15 2013 +0000 Update translations for orbot --- values-ka/strings.xml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/values-ka/strings.xml b/values-ka/strings.xml new file mode 100644 index 0000000..0f46066 --- /dev/null +++ b/values-ka/strings.xml @@ -0,0 +1,18 @@ +<?xml version='1.0' encoding='UTF-8'?> +<resources> +  +  +  +  +  +  +  +  +  +  + <string name="default_bridges"></string> +</resources>

1 0

[translation/gettor] Update translations for gettor
by translation＠torproject.org 19 Jun '13

19 Jun '13

commit 209432de357c977876a7ac0b00d0e409ac2743c6 Author: Translation commit bot <translation(a)torproject.org> Date: Wed Jun 19 15:15:10 2013 +0000 Update translations for gettor --- ka/gettor.po | 503 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 503 insertions(+) diff --git a/ka/gettor.po b/ka/gettor.po new file mode 100644 index 0000000..4bb2c0b --- /dev/null +++ b/ka/gettor.po @@ -0,0 +1,503 @@ +# SOME DESCRIPTIVE TITLE. +# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER +# This file is distributed under the same license as the PACKAGE package. +# +# Translators: +# Jacob Appelbaum <jacob(a)appelbaum.net>, 2009 +msgid "" +msgstr "" +"Project-Id-Version: The Tor Project\n" +"Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" +"POT-Creation-Date: 2013-01-19 13:40+0100\n" +"PO-Revision-Date: 2013-06-19 14:50+0000\n" +"Last-Translator: runasand <runa.sandvik(a)gmail.com>\n" +"Language-Team: Georgian (http://www.transifex.com/projects/p/torproject/language/ka/)\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Language: ka\n" +"Plural-Forms: nplurals=1; plural=0;\n" + +#: lib/gettor/i18n.py:27 +msgid "Hello, This is the \"GetTor\" robot." +msgstr "" + +#: lib/gettor/i18n.py:29 +msgid "Thank you for your request." +msgstr "" + +#: lib/gettor/i18n.py:31 +msgid "" +"Unfortunately, we won't answer you at this address. You should make\n" +"an account with GMAIL.COM, YAHOO.COM or YAHOO.CN and send the mail from\n" +"one of those." +msgstr "" + +#: lib/gettor/i18n.py:35 +msgid "" +"We only process requests from email services that support \"DKIM\",\n" +"which is an email feature that lets us verify that the address in the\n" +"\"From\" line is actually the one who sent the mail." +msgstr "" + +#: lib/gettor/i18n.py:39 +msgid "" +"(We apologize if you didn't ask for this mail. Since your email is from\n" +"a service that doesn't use DKIM, we're sending a short explanation,\n" +"and then we'll ignore this email address for the next day or so.)" +msgstr "" + +#: lib/gettor/i18n.py:43 lib/gettor/i18n.py:135 +msgid "" +"If you have any questions or it doesn't work, you can contact a\n" +"human at this support email address: help(a)rt.torproject.org" +msgstr "" + +#: lib/gettor/i18n.py:46 +msgid "" +"I will mail you a Tor package, if you tell me which one you want.\n" +"Please select one of the following package names:\n" +"\n" +" windows\n" +" macos-i386\n" +" macos-ppc\n" +" linux-i386\n" +" linux-x86_64\n" +" obfs-windows\n" +" obfs-macos-i386\n" +" obfs-macos-x86_64\n" +" obfs-linux-i386\n" +" obfs-linux-x86_64\n" +" source" +msgstr "" + +#: lib/gettor/i18n.py:61 +msgid "" +"Please reply to this mail, and tell me a single package name anywhere \n" +"in the body of your email." +msgstr "" + +#: lib/gettor/i18n.py:64 +msgid "" +"OBTAINING LOCALIZED VERSIONS OF TOR\n" +"===================================" +msgstr "" + +#: lib/gettor/i18n.py:67 +msgid "" +"To get a version of Tor translated into your language, specify the\n" +"language you want in the address you send the mail to:\n" +"\n" +" gettor+fa(a)torproject.org" +msgstr "" + +#: lib/gettor/i18n.py:72 +msgid "" +"This example will give you the requested package in a localized\n" +"version for Farsi (Persian). Check below for a list of supported language\n" +"codes. " +msgstr "" + +#: lib/gettor/i18n.py:76 +msgid " List of supported locales:" +msgstr "" + +#: lib/gettor/i18n.py:78 +msgid "Here is a list of all available languages:" +msgstr "" + +#: lib/gettor/i18n.py:80 +msgid "" +" gettor+ar(a)torproject.org: Arabic\n" +" gettor+de(a)torproject.org: German\n" +" gettor+en(a)torproject.org: English\n" +" gettor+es(a)torproject.org: Spanish\n" +" gettor+fa(a)torproject.org: Farsi (Iran)\n" +" gettor+fr(a)torproject.org: French\n" +" gettor+it(a)torproject.org: Italian\n" +" gettor+nl(a)torproject.org: Dutch\n" +" gettor+pl(a)torproject.org: Polish\n" +" gettor+ru(a)torproject.org: Russian\n" +" gettor+zh(a)torproject.org: Chinese" +msgstr "" + +#: lib/gettor/i18n.py:92 +msgid "If you select no language, you will receive the English version." +msgstr "" + +#: lib/gettor/i18n.py:94 +msgid "" +"SMALLER SIZED PACKAGES\n" +"======================" +msgstr "" + +#: lib/gettor/i18n.py:97 +msgid "" +"If your bandwith is low or your provider doesn't allow you to\n" +"receive large attachments in your email, GetTor can send you several\n" +"small packages instead of one big one." +msgstr "" + +#: lib/gettor/i18n.py:101 +msgid "" +"Simply include the keyword 'split' in a new line on its own (this part\n" +"is important!) like so: \n" +" \n" +" windows\n" +" split" +msgstr "" + +#: lib/gettor/i18n.py:107 +msgid "" +"Sending this text in an email to GetTor will cause it to send you \n" +"the Tor Browser Bundle in a number of 1,4MB attachments." +msgstr "" + +#: lib/gettor/i18n.py:110 +msgid "" +"After having received all parts, you need to re-assemble them to \n" +"one package again. This is done as follows:" +msgstr "" + +#: lib/gettor/i18n.py:113 +msgid "1.) Save all received attachments into one folder on your disk." +msgstr "" + +#: lib/gettor/i18n.py:115 +msgid "" +"2.) Unzip all files ending in \".z\". If you saved all attachments to\n" +"a fresh folder before, simply unzip all files in that folder. If you don't\n" +"know how to unzip the .z files, please see the UNPACKING THE FILES section." +msgstr "" + +#: lib/gettor/i18n.py:119 +msgid "" +"3.) Verify all files as described in the mail you received with \n" +"each package. (gpg --verify)" +msgstr "" + +#: lib/gettor/i18n.py:122 +msgid "" +"4.) Now unpack the multi-volume archive into one file by double-\n" +"clicking the file ending in \"..split.part01.exe\". This should start the \n" +"process automatically." +msgstr "" + +#: lib/gettor/i18n.py:126 +msgid "" +"5.) After unpacking is finished, you should find a newly created \n" +"\".exe\" file in your destination folder. Simply doubleclick\n" +"that and Tor Browser Bundle should start within a few seconds." +msgstr "" + +#: lib/gettor/i18n.py:130 +msgid "6.) That's it. You're done. Thanks for using Tor and have fun!" +msgstr "" + +#: lib/gettor/i18n.py:132 +msgid "" +"SUPPORT\n" +"=======" +msgstr "" + +#: lib/gettor/i18n.py:138 +msgid "" +"Here's your requested software as a zip file. Please unzip the\n" +"package and verify the signature." +msgstr "" + +#: lib/gettor/i18n.py:141 +msgid "" +"VERIFY SIGNATURE\n" +"================\n" +"If your computer has GnuPG installed, use the gpg commandline \n" +"tool as follows after unpacking the zip file:\n" +"\n" +" gpg --verify tor-browser-1.3.24_en-US.exe.asc tor-browser-1.3.24_en-US.exe" +msgstr "" + +#: lib/gettor/i18n.py:148 +msgid "" +"The output should look somewhat like this:\n" +"\n" +" gpg: Good signature from 'Erinn Clark <...>'" +msgstr "" + +#: lib/gettor/i18n.py:152 +msgid "" +"If you're not familiar with commandline tools, try looking for\n" +"a graphical user interface for GnuPG on this website:\n" +"\n" +" http://www.gnupg.org/related_software/frontends.html" +msgstr "" + +#: lib/gettor/i18n.py:157 +msgid "" +"BLOCKED ACCESS / CENSORSHIP\n" +"===========================" +msgstr "" + +#: lib/gettor/i18n.py:160 +msgid "" +"If your Internet connection blocks access to the Tor network, you\n" +"may need a bridge relay. Bridge relays (or \"bridges\" for short)\n" +"are Tor relays that aren't listed in the main directory. Since there\n" +"is no complete public list of them, even if your ISP is filtering\n" +"connections to all the known Tor relays, they probably won't be able\n" +"to block all the bridges." +msgstr "" + +#: lib/gettor/i18n.py:167 +msgid "" +"You can acquire a bridge by sending an email that contains \"get bridges\"\n" +"in the body of the email to the following email address:\n" +"\n" +" bridges(a)torproject.org" +msgstr "" + +#: lib/gettor/i18n.py:172 +msgid "" +"It is also possible to fetch bridges with a web browser at the following\n" +"url: https://bridges.torproject.org/" +msgstr "" + +#: lib/gettor/i18n.py:175 +msgid "" +"Another censorship circumvention tool you can request from GetTor is\n" +"the Tor Obfsproxy Browser Bundle. Please read the package descriptions for\n" +"which package you should request to receive this." +msgstr "" + +#: lib/gettor/i18n.py:179 +msgid "" +"IMPORTANT NOTE:\n" +"Since this is part of a split-file request, you need to wait for\n" +"all split files to be received by you before you can save them all\n" +"into the same directory and unpack them by double-clicking the\n" +"first file." +msgstr "" + +#: lib/gettor/i18n.py:185 +msgid "" +"Packages might arrive out of order! Please make sure you received\n" +"all packages before you attempt to unpack them!" +msgstr "" + +#: lib/gettor/i18n.py:188 +#, python-format +msgid "" +"It was successfully understood. Your request is currently being processed.\n" +"Your package (%s) should arrive within the next ten minutes." +msgstr "" + +#: lib/gettor/i18n.py:191 +msgid "" +"If it doesn't arrive, the package might be too big for your mail provider.\n" +"Try resending the mail from a GMAIL.COM, YAHOO.CN or YAHOO.COM account." +msgstr "" + +#: lib/gettor/i18n.py:194 +msgid "" +"Unfortunately we are currently experiencing problems and we can't fulfill\n" +"your request right now. Please be patient as we try to resolve this issue." +msgstr "" + +#: lib/gettor/i18n.py:197 +msgid "" +"Unfortunately there is no split package available for the package you\n" +"requested. Please send us another package name or request the same package \n" +"again, but remove the 'split' keyword. In that case we'll send you the whole \n" +"package. Make sure this is what you want." +msgstr "" + +#: lib/gettor/i18n.py:202 +msgid "" +"UNPACKING THE FILES\n" +"===================" +msgstr "" + +#: lib/gettor/i18n.py:205 +msgid "" +"The easiest way to unpack the files you received is to install 7-Zip,\n" +"a free file compression/uncompression tool. If it isn't installed on\n" +"your computer yet, you can download it here:\n" +"\n" +" http://www.7-zip.org/" +msgstr "" + +#: lib/gettor/i18n.py:211 +msgid "" +"When 7-Zip is installed, you can open the .z archive you received from\n" +"us by double-clicking on it." +msgstr "" + +#: lib/gettor/i18n.py:214 +msgid "" +"An alternative way to get the .z files extraced is to rename them to\n" +".zip. For example, if you recevied a file called \"windows.z\", rename it to \n" +"\"windows.zip\". You should then be able to extract the archive with common \n" +"file archiver programs that probably are already installed on your computer." +msgstr "" + +#: lib/gettor/i18n.py:219 +msgid "" +"Please reply to this mail, and tell me a single package name anywhere\n" +"in your reply. Here's a short explanation of what these packages are:" +msgstr "" + +#: lib/gettor/i18n.py:222 +msgid "" +"windows:\n" +"The Tor Browser Bundle package for Windows operating systems. If you're \n" +"running some version of Windows, like Windows XP, Windows Vista or \n" +"Windows 7, this is the package you should get." +msgstr "" + +#: lib/gettor/i18n.py:227 +msgid "" +"macos-i386:\n" +"The Tor Browser Bundle package for OS X, Intel CPU architecture. In \n" +"general, newer Mac hardware will require you to use this package." +msgstr "" + +#: lib/gettor/i18n.py:231 +msgid "" +"macos-ppc:\n" +"This is an older installer (the \"Vidalia bundle\") for older Macs running\n" +"OS X on PowerPC CPUs. Note that this package will be deprecated soon." +msgstr "" + +#: lib/gettor/i18n.py:235 +msgid "" +"linux-i386:\n" +"The Tor Browser Bundle package for Linux, 32bit versions." +msgstr "" + +#: lib/gettor/i18n.py:238 +msgid "" +"Note that this package is rather large and needs your email provider to \n" +"allow for attachments of about 30MB in size." +msgstr "" + +#: lib/gettor/i18n.py:241 +msgid "" +"linux-x86_64:\n" +"The Tor Browser Bundle package for Linux, 64bit versions." +msgstr "" + +#: lib/gettor/i18n.py:244 +msgid "" +"obfs-windows:\n" +"The Tor Obfsproxy Browser Bundle for Windows operating systems. If you need\n" +"strong censorship circumvention and you are running some version of the \n" +"Windows, like Windows XP, Windows Vista or Windows 7, this is the package\n" +"you should get." +msgstr "" + +#: lib/gettor/i18n.py:250 +msgid "" +"obfs-macos-i386:\n" +"The Tor Obfsproxy Browser Bundle package for OS X, 32bit Intel CPU \n" +"architecture." +msgstr "" + +#: lib/gettor/i18n.py:254 +msgid "" +"obfs-macos-x86_64:\n" +"The Tor Obfsproxy Browser Bundle package for OS X, 64bit Intel CPU \n" +"architecture." +msgstr "" + +#: lib/gettor/i18n.py:258 +msgid "" +"obfs-linux-i386:\n" +"The Tor Obfsproxy Browser Bundle package for Linux, 32bit Intel CPU \n" +"architecture." +msgstr "" + +#: lib/gettor/i18n.py:262 +msgid "" +"obfs-linux-x86_64:\n" +"The Tor Obfsproxy Browser Bundle package for Linux, 64bit Intel CPU \n" +"architecture." +msgstr "" + +#: lib/gettor/i18n.py:266 +msgid "" +"source:\n" +"The Tor source code, for experts. Most users do not want this package." +msgstr "" + +#: lib/gettor/i18n.py:269 +msgid "" +"FREQUENTLY ASKED QUESTIONS\n" +"==========================" +msgstr "" + +#: lib/gettor/i18n.py:272 +msgid "What is Tor?" +msgstr "" + +#: lib/gettor/i18n.py:274 +msgid "The name \"Tor\" can refer to several different components." +msgstr "" + +#: lib/gettor/i18n.py:276 +msgid "" +"The Tor software is a program you can run on your computer that helps \n" +"keep you safe on the Internet. Tor protects you by bouncing your \n" +"communications around a distributed network of relays run by volunteers \n" +"all around the world: it prevents somebody watching your Internet connection \n" +"from learning what sites you visit, and it prevents the sites you visit from \n" +"learning your physical location. This set of volunteer relays is called the \n" +"Tor network. You can read more about how Tor works here:\n" +"\n" +" https://www.torproject.org/about/overview.html.en" +msgstr "" + +#: lib/gettor/i18n.py:286 +msgid "What is the Tor Browser Bundle?" +msgstr "" + +#: lib/gettor/i18n.py:288 +msgid "" +"The Browser Bundle (TBB) is the package we recommend to most users. \n" +"The bundle comes with everything you need to safely browse the Internet.\n" +"Just extract it and run." +msgstr "" + +#: lib/gettor/i18n.py:292 +msgid "What package should I request?" +msgstr "" + +#: lib/gettor/i18n.py:294 +msgid "" +"This depends on the operating system you use. For instance, if your\n" +"operating system is Microsoft Windows, you should request \"windows\". Here\n" +"is a short explanation of all packages to request and what operating \n" +"systems there are suitable for:" +msgstr "" + +#: lib/gettor/i18n.py:299 +msgid "How do I extract the file(s) you sent me?" +msgstr "" + +#: lib/gettor/i18n.py:301 +msgid "QUESTION:" +msgstr "" + +#: lib/gettor/i18n.py:303 +msgid "ANSWER:" +msgstr "" + +#: lib/gettor/i18n.py:305 +#, python-format +msgid "" +"Sorry, but the package you requested (%s) is too large for your \n" +"provider to accept as an attachment. Try using another provider that allows \n" +"for larger email attachments. Or try one of the following mirrors:\n" +"\n" +" https://www.oignon.net/dist/torbrowser/\n" +" https://tor.beme-it.de/dist/torbrowser/\n" +" https://www.torservers.net/mirrors/torproject.org/dist/torbrowser/" +msgstr ""

1 0

[translation/bridgedb] Update translations for bridgedb
by translation＠torproject.org 19 Jun '13

19 Jun '13

commit 26e24cf17d11c82dbce63838cc656682f81ee736 Author: Translation commit bot <translation(a)torproject.org> Date: Wed Jun 19 15:15:03 2013 +0000 Update translations for bridgedb --- ka/bridgedb.po | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/ka/bridgedb.po b/ka/bridgedb.po new file mode 100644 index 0000000..c8c2f29 --- /dev/null +++ b/ka/bridgedb.po @@ -0,0 +1,100 @@ +# Translations template for BridgeDB. +# Copyright (C) 2013 ORGANIZATION +# This file is distributed under the same license as the BridgeDB project. +# +# Translators: +# x0rmia <x0rmia(a)gmail.com>, 2013 +msgid "" +msgstr "" +"Project-Id-Version: The Tor Project\n" +"Report-Msgid-Bugs-To: https://trac.torproject.org/projects/tor\n" +"POT-Creation-Date: 2013-03-27 21:41+0000\n" +"PO-Revision-Date: 2013-06-19 15:10+0000\n" +"Last-Translator: x0rmia <x0rmia(a)gmail.com>\n" +"Language-Team: Georgian (http://www.transifex.com/projects/p/torproject/language/ka/)\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Generated-By: Babel 0.9.6\n" +"Language: ka\n" +"Plural-Forms: nplurals=1; plural=0;\n" + +#: lib/bridgedb/templates/base.html:33 +msgid "What are bridges?" +msgstr "" + +#: lib/bridgedb/templates/base.html:34 +#, python-format +msgid "" +"%s Bridge relays %s are Tor relays that help you circumvent censorship." +msgstr "" + +#: lib/bridgedb/templates/base.html:39 +msgid "I need an alternative way of getting bridges!" +msgstr "" + +#: lib/bridgedb/templates/base.html:40 +#, python-format +msgid "" +"Another way to find public bridge addresses is to send an email (from a %s " +"or a %s address) to %s with the line 'get bridges' by itself in the body of " +"the mail." +msgstr "" + +#: lib/bridgedb/templates/base.html:48 +msgid "My bridges don't work! I need help!" +msgstr "" + +#: lib/bridgedb/templates/base.html:49 +#, python-format +msgid "" +"If your Tor doesn't work, you should email %s. Try including as much info " +"about your case as you can, including the list of bridges you used, the " +"bundle filename/version you used, the messages that Tor gave out, etc." +msgstr "" + +#: lib/bridgedb/templates/bridges.html:10 +msgid "" +"To use the above lines, go to Vidalia's Network settings page, and click " +"\"My ISP blocks connections to the Tor network\". Then add each bridge " +"address one at a time." +msgstr "" + +#: lib/bridgedb/templates/bridges.html:13 +msgid "No bridges currently available" +msgstr "" + +#: lib/bridgedb/templates/captcha.html:6 +msgid "Upgrade your browser to Firefox" +msgstr "" + +#: lib/bridgedb/templates/captcha.html:8 +msgid "Type the two words" +msgstr "" + +#: lib/bridgedb/templates/index.html:6 +msgid "Step 1" +msgstr "ნაბიჯი 1" + +#: lib/bridgedb/templates/index.html:8 +#, python-format +msgid "Get %s Tor Browser Bundle %s" +msgstr "მიიღეთ %s Tor ბრაუზერის ნუსხა %s" + +#: lib/bridgedb/templates/index.html:13 +msgid "Step 2" +msgstr "ნაბიჯი 2" + +#: lib/bridgedb/templates/index.html:15 +#, python-format +msgid "Get %s bridges %s" +msgstr "" + +#: lib/bridgedb/templates/index.html:19 +msgid "Step 3" +msgstr "ნაბიჯი 3" + +#: lib/bridgedb/templates/index.html:21 +#, python-format +msgid "Now %s add the bridges to Tor %s" +msgstr ""

1 0

[onionoo/master] Extract GeoIP lookup code and test it.
by karsten＠torproject.org 19 Jun '13

19 Jun '13

commit 9748712da484111087b7268e2246829178f2a612 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jun 19 13:39:22 2013 +0200 Extract GeoIP lookup code and test it. --- src/org/torproject/onionoo/CurrentNodes.java | 360 +------------- src/org/torproject/onionoo/LookupService.java | 396 ++++++++++++++++ src/org/torproject/onionoo/Main.java | 6 +- test/org/torproject/onionoo/LookupServiceTest.java | 501 ++++++++++++++++++++ 4 files changed, 923 insertions(+), 340 deletions(-) diff --git a/src/org/torproject/onionoo/CurrentNodes.java b/src/org/torproject/onionoo/CurrentNodes.java index 64db6a1..9e27f5b 100644 --- a/src/org/torproject/onionoo/CurrentNodes.java +++ b/src/org/torproject/onionoo/CurrentNodes.java @@ -2,16 +2,11 @@ * See LICENSE for licensing information */ package org.torproject.onionoo; -import java.io.BufferedReader; -import java.io.File; -import java.io.FileReader; -import java.io.IOException; import java.text.ParseException; import java.text.SimpleDateFormat; import java.util.Arrays; import java.util.Collection; import java.util.Collections; -import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Scanner; @@ -21,12 +16,12 @@ import java.util.SortedSet; import java.util.TimeZone; import java.util.TreeMap; import java.util.TreeSet; -import java.util.regex.Pattern; import org.torproject.descriptor.BridgeNetworkStatus; import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.NetworkStatusEntry; import org.torproject.descriptor.RelayNetworkStatusConsensus; +import org.torproject.onionoo.LookupService.LookupResult; /* Store relays and bridges that have been running in the past seven * days. */ @@ -34,17 +29,20 @@ public class CurrentNodes { private DescriptorSource descriptorSource; + private LookupService lookupService; + private DocumentStore documentStore; /* Initialize an instance for the back-end that is read-only and doesn't * support parsing new descriptor contents. */ public CurrentNodes(DocumentStore documentStore) { - this(null, documentStore); + this(null, null, documentStore); } public CurrentNodes(DescriptorSource descriptorSource, - DocumentStore documentStore) { + LookupService lookupService, DocumentStore documentStore) { this.descriptorSource = descriptorSource; + this.lookupService = lookupService; this.documentStore = documentStore; } @@ -411,345 +409,29 @@ public class CurrentNodes { } public void lookUpCitiesAndASes() { - - /* Make sure we have all required .csv files. */ - // TODO Make paths configurable or allow passing file contents as - // strings in order to facilitate testing. - // TODO Move look-up code to new LookupService class that is - // initialized with geoip files, receives a sorted set of addresses, - // performs lookups, and returns results to CurrentNodes. - File[] geoLiteCityBlocksCsvFiles = new File[] { - new File("geoip/Manual-GeoLiteCity-Blocks.csv"), - new File("geoip/Automatic-GeoLiteCity-Blocks.csv"), - new File("geoip/GeoLiteCity-Blocks.csv") - }; - File geoLiteCityBlocksCsvFile = null; - for (File file : geoLiteCityBlocksCsvFiles) { - if (file.exists()) { - geoLiteCityBlocksCsvFile = file; - break; - } - } - if (geoLiteCityBlocksCsvFile == null) { - System.err.println("No *GeoLiteCity-Blocks.csv file in geoip/."); - return; - } - File geoLiteCityLocationCsvFile = - new File("geoip/GeoLiteCity-Location.csv"); - if (!geoLiteCityLocationCsvFile.exists()) { - System.err.println("No GeoLiteCity-Location.csv file in geoip/."); - return; - } - File iso3166CsvFile = new File("geoip/iso3166.csv"); - if (!iso3166CsvFile.exists()) { - System.err.println("No iso3166.csv file in geoip/."); - return; - } - File regionCsvFile = new File("geoip/region.csv"); - if (!regionCsvFile.exists()) { - System.err.println("No region.csv file in geoip/."); - return; - } - File geoIPASNum2CsvFile = new File("geoip/GeoIPASNum2.csv"); - if (!geoIPASNum2CsvFile.exists()) { - System.err.println("No GeoIPASNum2.csv file in geoip/."); - return; - } - - /* Obtain a map from relay IP address strings to numbers. */ - Map<String, Long> addressStringNumbers = new HashMap<String, Long>(); - Pattern ipv4Pattern = Pattern.compile("^[0-9\\.]{7,15}$"); + SortedSet<String> addressStrings = new TreeSet<String>(); for (Node relay : this.knownRelays.values()) { - String addressString = relay.getAddress(); - long addressNumber = -1L; - if (ipv4Pattern.matcher(addressString).matches()) { - String[] parts = addressString.split("\\.", 4); - if (parts.length == 4) { - addressNumber = 0L; - for (int i = 0; i < 4; i++) { - addressNumber *= 256L; - int octetValue = -1; - try { - octetValue = Integer.parseInt(parts[i]); - } catch (NumberFormatException e) { - } - if (octetValue < 0 || octetValue > 255) { - addressNumber = -1L; - break; - } - addressNumber += octetValue; - } - } - } - if (addressNumber >= 0L) { - addressStringNumbers.put(addressString, addressNumber); - } + addressStrings.add(relay.getAddress()); } - if (addressStringNumbers.isEmpty()) { + if (addressStrings.isEmpty()) { System.err.println("No relay IP addresses to resolve to cities or " + "ASN."); return; } - - /* Obtain a map from IP address numbers to blocks. */ - Map<Long, Long> addressNumberBlocks = new HashMap<Long, Long>(); - try { - SortedSet<Long> sortedAddressNumbers = new TreeSet<Long>( - addressStringNumbers.values()); - long firstAddressNumber = sortedAddressNumbers.first(); - BufferedReader br = new BufferedReader(new FileReader( - geoLiteCityBlocksCsvFile)); - String line; - long previousStartIpNum = -1L; - while ((line = br.readLine()) != null) { - if (!line.startsWith("\"")) { - continue; - } - String[] parts = line.replaceAll("\"", "").split(",", 3); - if (parts.length != 3) { - System.err.println("Illegal line '" + line + "' in " - + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - try { - long startIpNum = Long.parseLong(parts[0]); - if (startIpNum <= previousStartIpNum) { - System.err.println("Line '" + line + "' not sorted in " - + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - previousStartIpNum = startIpNum; - while (firstAddressNumber < startIpNum && - firstAddressNumber != -1L) { - sortedAddressNumbers.remove(firstAddressNumber); - if (sortedAddressNumbers.isEmpty()) { - firstAddressNumber = -1L; - } else { - firstAddressNumber = sortedAddressNumbers.first(); - } - } - long endIpNum = Long.parseLong(parts[1]); - while (firstAddressNumber <= endIpNum && - firstAddressNumber != -1L) { - long blockNumber = Long.parseLong(parts[2]); - addressNumberBlocks.put(firstAddressNumber, blockNumber); - sortedAddressNumbers.remove(firstAddressNumber); - if (sortedAddressNumbers.isEmpty()) { - firstAddressNumber = -1L; - } else { - firstAddressNumber = sortedAddressNumbers.first(); - } - } - if (firstAddressNumber == -1L) { - break; - } - } - catch (NumberFormatException e) { - System.err.println("Number format exception while parsing line " - + "'" + line + "' in " - + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - } - br.close(); - } catch (IOException e) { - System.err.println("I/O exception while reading " - + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); - return; - } - - /* Obtain a map from relevant blocks to location lines. */ - Map<Long, String> blockLocations = new HashMap<Long, String>(); - try { - Set<Long> blockNumbers = new HashSet<Long>( - addressNumberBlocks.values()); - BufferedReader br = new BufferedReader(new FileReader( - geoLiteCityLocationCsvFile)); - String line; - while ((line = br.readLine()) != null) { - if (line.startsWith("C") || line.startsWith("l")) { - continue; - } - String[] parts = line.replaceAll("\"", "").split(",", 9); - if (parts.length != 9) { - System.err.println("Illegal line '" + line + "' in " - + geoLiteCityLocationCsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - try { - long locId = Long.parseLong(parts[0]); - if (blockNumbers.contains(locId)) { - blockLocations.put(locId, line); - } - } - catch (NumberFormatException e) { - System.err.println("Number format exception while parsing line " - + "'" + line + "' in " - + geoLiteCityLocationCsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - } - br.close(); - } catch (IOException e) { - System.err.println("I/O exception while reading " - + geoLiteCityLocationCsvFile.getAbsolutePath() + "."); - return; - } - - /* Read country names to memory. */ - Map<String, String> countryNames = new HashMap<String, String>(); - try { - BufferedReader br = new BufferedReader(new FileReader( - iso3166CsvFile)); - String line; - while ((line = br.readLine()) != null) { - String[] parts = line.replaceAll("\"", "").split(",", 2); - if (parts.length != 2) { - System.err.println("Illegal line '" + line + "' in " - + iso3166CsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - countryNames.put(parts[0].toLowerCase(), parts[1]); - } - br.close(); - } catch (IOException e) { - System.err.println("I/O exception while reading " - + iso3166CsvFile.getAbsolutePath() + "."); - return; - } - - /* Read region names to memory. */ - Map<String, String> regionNames = new HashMap<String, String>(); - try { - BufferedReader br = new BufferedReader(new FileReader( - regionCsvFile)); - String line; - while ((line = br.readLine()) != null) { - String[] parts = line.replaceAll("\"", "").split(",", 3); - if (parts.length != 3) { - System.err.println("Illegal line '" + line + "' in " - + regionCsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - regionNames.put(parts[0].toLowerCase() + "," - + parts[1].toLowerCase(), parts[2]); - } - br.close(); - } catch (IOException e) { - System.err.println("I/O exception while reading " - + regionCsvFile.getAbsolutePath() + "."); - return; - } - - /* Obtain a map from IP address numbers to ASN. */ - Map<Long, String> addressNumberASN = new HashMap<Long, String>(); - try { - SortedSet<Long> sortedAddressNumbers = new TreeSet<Long>( - addressStringNumbers.values()); - long firstAddressNumber = sortedAddressNumbers.first(); - BufferedReader br = new BufferedReader(new FileReader( - geoIPASNum2CsvFile)); - String line; - long previousStartIpNum = -1L; - while ((line = br.readLine()) != null) { - String[] parts = line.replaceAll("\"", "").split(",", 3); - if (parts.length != 3) { - System.err.println("Illegal line '" + line + "' in " - + geoIPASNum2CsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - try { - long startIpNum = Long.parseLong(parts[0]); - if (startIpNum <= previousStartIpNum) { - System.err.println("Line '" + line + "' not sorted in " - + geoIPASNum2CsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - previousStartIpNum = startIpNum; - while (firstAddressNumber < startIpNum && - firstAddressNumber != -1L) { - sortedAddressNumbers.remove(firstAddressNumber); - if (sortedAddressNumbers.isEmpty()) { - firstAddressNumber = -1L; - } else { - firstAddressNumber = sortedAddressNumbers.first(); - } - } - long endIpNum = Long.parseLong(parts[1]); - while (firstAddressNumber <= endIpNum && - firstAddressNumber != -1L) { - if (parts[2].startsWith("AS") && - parts[2].split(" ", 2).length == 2) { - addressNumberASN.put(firstAddressNumber, parts[2]); - } - sortedAddressNumbers.remove(firstAddressNumber); - if (sortedAddressNumbers.isEmpty()) { - firstAddressNumber = -1L; - } else { - firstAddressNumber = sortedAddressNumbers.first(); - } - } - if (firstAddressNumber == -1L) { - break; - } - } - catch (NumberFormatException e) { - System.err.println("Number format exception while parsing line " - + "'" + line + "' in " - + geoIPASNum2CsvFile.getAbsolutePath() + "."); - br.close(); - return; - } - } - br.close(); - } catch (IOException e) { - System.err.println("I/O exception while reading " - + geoIPASNum2CsvFile.getAbsolutePath() + "."); - return; - } - - /* Finally, set relays' city and ASN information. */ + SortedMap<String, LookupResult> lookupResults = + this.lookupService.lookup(addressStrings); for (Node relay : knownRelays.values()) { String addressString = relay.getAddress(); - if (addressStringNumbers.containsKey(addressString)) { - long addressNumber = addressStringNumbers.get(addressString); - if (addressNumberBlocks.containsKey(addressNumber)) { - long blockNumber = addressNumberBlocks.get(addressNumber); - if (blockLocations.containsKey(blockNumber)) { - String[] parts = blockLocations.get(blockNumber). - replaceAll("\"", "").split(",", -1); - String countryCode = parts[1].toLowerCase(); - relay.setCountryCode(countryCode); - if (countryNames.containsKey(countryCode)) { - relay.setCountryName(countryNames.get(countryCode)); - } - String regionCode = countryCode + "," - + parts[2].toLowerCase(); - if (regionNames.containsKey(regionCode)) { - relay.setRegionName(regionNames.get(regionCode)); - } - if (parts[3].length() > 0) { - relay.setCityName(parts[3]); - } - relay.setLatitude(parts[5]); - relay.setLongitude(parts[6]); - } - } - if (addressNumberASN.containsKey(addressNumber)) { - String[] parts = addressNumberASN.get(addressNumber).split(" ", - 2); - relay.setASNumber(parts[0]); - relay.setASName(parts[1]); - } + if (lookupResults.containsKey(addressString)) { + LookupResult lookupResult = lookupResults.get(addressString); + relay.setCountryCode(lookupResult.countryCode); + relay.setCountryName(lookupResult.countryName); + relay.setRegionName(lookupResult.regionName); + relay.setCityName(lookupResult.cityName); + relay.setLatitude(lookupResult.latitude); + relay.setLongitude(lookupResult.longitude); + relay.setASNumber(lookupResult.aSNumber); + relay.setASName(lookupResult.aSName); } } } diff --git a/src/org/torproject/onionoo/LookupService.java b/src/org/torproject/onionoo/LookupService.java new file mode 100644 index 0000000..bf3131e --- /dev/null +++ b/src/org/torproject/onionoo/LookupService.java @@ -0,0 +1,396 @@ +/* Copyright 2013 The Tor Project + * See LICENSE for licensing information */ + +package org.torproject.onionoo; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; +import java.util.regex.Pattern; + +public class LookupService { + + File geoipDir; + File geoLiteCityBlocksCsvFile; + File geoLiteCityLocationCsvFile; + File iso3166CsvFile; + File regionCsvFile; + File geoIPASNum2CsvFile; + private boolean hasAllFiles = false; + public LookupService(File geoipDir) { + this.geoipDir = geoipDir; + this.findRequiredCsvFiles(); + } + + /* Make sure we have all required .csv files. */ + private void findRequiredCsvFiles() { + File[] geoLiteCityBlocksCsvFiles = new File[] { + new File(this.geoipDir, "Manual-GeoLiteCity-Blocks.csv"), + new File(this.geoipDir, "Automatic-GeoLiteCity-Blocks.csv"), + new File(this.geoipDir, "GeoLiteCity-Blocks.csv") }; + for (File file : geoLiteCityBlocksCsvFiles) { + if (file.exists()) { + this.geoLiteCityBlocksCsvFile = file; + break; + } + } + if (this.geoLiteCityBlocksCsvFile == null) { + System.err.println("No *GeoLiteCity-Blocks.csv file in geoip/."); + return; + } + this.geoLiteCityLocationCsvFile = new File(this.geoipDir, + "GeoLiteCity-Location.csv"); + if (!this.geoLiteCityLocationCsvFile.exists()) { + System.err.println("No GeoLiteCity-Location.csv file in geoip/."); + return; + } + this.iso3166CsvFile = new File(this.geoipDir, "iso3166.csv"); + if (!this.iso3166CsvFile.exists()) { + System.err.println("No iso3166.csv file in geoip/."); + return; + } + this.regionCsvFile = new File(this.geoipDir, "region.csv"); + if (!this.regionCsvFile.exists()) { + System.err.println("No region.csv file in geoip/."); + return; + } + this.geoIPASNum2CsvFile = new File(this.geoipDir, "GeoIPASNum2.csv"); + if (!this.geoIPASNum2CsvFile.exists()) { + System.err.println("No GeoIPASNum2.csv file in geoip/."); + return; + } + this.hasAllFiles = true; + } + + public SortedMap<String, LookupResult> lookup( + SortedSet<String> addressStrings) { + + SortedMap<String, LookupResult> lookupResults = + new TreeMap<String, LookupResult>(); + + if (!this.hasAllFiles) { + return lookupResults; + } + + /* Obtain a map from relay IP address strings to numbers. */ + Map<String, Long> addressStringNumbers = new HashMap<String, Long>(); + Pattern ipv4Pattern = Pattern.compile("^[0-9\\.]{7,15}$"); + for (String addressString : addressStrings) { + long addressNumber = -1L; + if (ipv4Pattern.matcher(addressString).matches()) { + String[] parts = addressString.split("\\.", 4); + if (parts.length == 4) { + addressNumber = 0L; + for (int i = 0; i < 4; i++) { + addressNumber *= 256L; + int octetValue = -1; + try { + octetValue = Integer.parseInt(parts[i]); + } catch (NumberFormatException e) { + } + if (octetValue < 0 || octetValue > 255) { + addressNumber = -1L; + break; + } + addressNumber += octetValue; + } + } + } + if (addressNumber >= 0L) { + addressStringNumbers.put(addressString, addressNumber); + } + } + if (addressStringNumbers.isEmpty()) { + return lookupResults; + } + + /* Obtain a map from IP address numbers to blocks. */ + Map<Long, Long> addressNumberBlocks = new HashMap<Long, Long>(); + try { + SortedSet<Long> sortedAddressNumbers = new TreeSet<Long>( + addressStringNumbers.values()); + long firstAddressNumber = sortedAddressNumbers.first(); + BufferedReader br = new BufferedReader(new FileReader( + geoLiteCityBlocksCsvFile)); + String line; + long previousStartIpNum = -1L; + while ((line = br.readLine()) != null) { + if (!line.startsWith("\"")) { + continue; + } + String[] parts = line.replaceAll("\"", "").split(",", 3); + if (parts.length != 3) { + System.err.println("Illegal line '" + line + "' in " + + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + try { + long startIpNum = Long.parseLong(parts[0]); + if (startIpNum <= previousStartIpNum) { + System.err.println("Line '" + line + "' not sorted in " + + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + previousStartIpNum = startIpNum; + while (firstAddressNumber < startIpNum && + firstAddressNumber != -1L) { + sortedAddressNumbers.remove(firstAddressNumber); + if (sortedAddressNumbers.isEmpty()) { + firstAddressNumber = -1L; + } else { + firstAddressNumber = sortedAddressNumbers.first(); + } + } + long endIpNum = Long.parseLong(parts[1]); + while (firstAddressNumber <= endIpNum && + firstAddressNumber != -1L) { + long blockNumber = Long.parseLong(parts[2]); + addressNumberBlocks.put(firstAddressNumber, blockNumber); + sortedAddressNumbers.remove(firstAddressNumber); + if (sortedAddressNumbers.isEmpty()) { + firstAddressNumber = -1L; + } else { + firstAddressNumber = sortedAddressNumbers.first(); + } + } + if (firstAddressNumber == -1L) { + break; + } + } + catch (NumberFormatException e) { + System.err.println("Number format exception while parsing line " + + "'" + line + "' in " + + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + } + br.close(); + } catch (IOException e) { + System.err.println("I/O exception while reading " + + geoLiteCityBlocksCsvFile.getAbsolutePath() + "."); + return lookupResults; + } + + /* Obtain a map from relevant blocks to location lines. */ + Map<Long, String> blockLocations = new HashMap<Long, String>(); + try { + Set<Long> blockNumbers = new HashSet<Long>( + addressNumberBlocks.values()); + BufferedReader br = new BufferedReader(new FileReader( + geoLiteCityLocationCsvFile)); + String line; + while ((line = br.readLine()) != null) { + if (line.startsWith("C") || line.startsWith("l")) { + continue; + } + String[] parts = line.replaceAll("\"", "").split(",", 9); + if (parts.length != 9) { + System.err.println("Illegal line '" + line + "' in " + + geoLiteCityLocationCsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + try { + long locId = Long.parseLong(parts[0]); + if (blockNumbers.contains(locId)) { + blockLocations.put(locId, line); + } + } + catch (NumberFormatException e) { + System.err.println("Number format exception while parsing line " + + "'" + line + "' in " + + geoLiteCityLocationCsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + } + br.close(); + } catch (IOException e) { + System.err.println("I/O exception while reading " + + geoLiteCityLocationCsvFile.getAbsolutePath() + "."); + return lookupResults; + } + + /* Read country names to memory. */ + Map<String, String> countryNames = new HashMap<String, String>(); + try { + BufferedReader br = new BufferedReader(new FileReader( + iso3166CsvFile)); + String line; + while ((line = br.readLine()) != null) { + String[] parts = line.replaceAll("\"", "").split(",", 2); + if (parts.length != 2) { + System.err.println("Illegal line '" + line + "' in " + + iso3166CsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + countryNames.put(parts[0].toLowerCase(), parts[1]); + } + br.close(); + } catch (IOException e) { + System.err.println("I/O exception while reading " + + iso3166CsvFile.getAbsolutePath() + "."); + return lookupResults; + } + + /* Read region names to memory. */ + Map<String, String> regionNames = new HashMap<String, String>(); + try { + BufferedReader br = new BufferedReader(new FileReader( + regionCsvFile)); + String line; + while ((line = br.readLine()) != null) { + String[] parts = line.replaceAll("\"", "").split(",", 3); + if (parts.length != 3) { + System.err.println("Illegal line '" + line + "' in " + + regionCsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + regionNames.put(parts[0].toLowerCase() + "," + + parts[1].toLowerCase(), parts[2]); + } + br.close(); + } catch (IOException e) { + System.err.println("I/O exception while reading " + + regionCsvFile.getAbsolutePath() + "."); + return lookupResults; + } + + /* Obtain a map from IP address numbers to ASN. */ + Map<Long, String> addressNumberASN = new HashMap<Long, String>(); + try { + SortedSet<Long> sortedAddressNumbers = new TreeSet<Long>( + addressStringNumbers.values()); + long firstAddressNumber = sortedAddressNumbers.first(); + BufferedReader br = new BufferedReader(new FileReader( + geoIPASNum2CsvFile)); + String line; + long previousStartIpNum = -1L; + while ((line = br.readLine()) != null) { + String[] parts = line.replaceAll("\"", "").split(",", 3); + if (parts.length != 3) { + System.err.println("Illegal line '" + line + "' in " + + geoIPASNum2CsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + try { + long startIpNum = Long.parseLong(parts[0]); + if (startIpNum <= previousStartIpNum) { + System.err.println("Line '" + line + "' not sorted in " + + geoIPASNum2CsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + previousStartIpNum = startIpNum; + while (firstAddressNumber < startIpNum && + firstAddressNumber != -1L) { + sortedAddressNumbers.remove(firstAddressNumber); + if (sortedAddressNumbers.isEmpty()) { + firstAddressNumber = -1L; + } else { + firstAddressNumber = sortedAddressNumbers.first(); + } + } + long endIpNum = Long.parseLong(parts[1]); + while (firstAddressNumber <= endIpNum && + firstAddressNumber != -1L) { + if (parts[2].startsWith("AS") && + parts[2].split(" ", 2).length == 2) { + addressNumberASN.put(firstAddressNumber, parts[2]); + } + sortedAddressNumbers.remove(firstAddressNumber); + if (sortedAddressNumbers.isEmpty()) { + firstAddressNumber = -1L; + } else { + firstAddressNumber = sortedAddressNumbers.first(); + } + } + if (firstAddressNumber == -1L) { + break; + } + } + catch (NumberFormatException e) { + System.err.println("Number format exception while parsing line " + + "'" + line + "' in " + + geoIPASNum2CsvFile.getAbsolutePath() + "."); + br.close(); + return lookupResults; + } + } + br.close(); + } catch (IOException e) { + System.err.println("I/O exception while reading " + + geoIPASNum2CsvFile.getAbsolutePath() + "."); + return lookupResults; + } + + /* Finally, put together lookup results. */ + for (String addressString : addressStrings) { + if (!addressStringNumbers.containsKey(addressString)) { + continue; + } + long addressNumber = addressStringNumbers.get(addressString); + if (!addressNumberBlocks.containsKey(addressNumber) && + !addressNumberASN.containsKey(addressNumber)) { + continue; + } + LookupResult lookupResult = new LookupResult(); + if (addressNumberBlocks.containsKey(addressNumber)) { + long blockNumber = addressNumberBlocks.get(addressNumber); + if (blockLocations.containsKey(blockNumber)) { + String[] parts = blockLocations.get(blockNumber). + replaceAll("\"", "").split(",", -1); + String countryCode = parts[1].toLowerCase(); + lookupResult.countryCode = countryCode; + if (countryNames.containsKey(countryCode)) { + lookupResult.countryName = countryNames.get(countryCode); + } + String regionCode = countryCode + "," + parts[2].toLowerCase(); + if (regionNames.containsKey(regionCode)) { + lookupResult.regionName = regionNames.get(regionCode); + } + if (parts[3].length() > 0) { + lookupResult.cityName = parts[3]; + } + lookupResult.latitude = parts[5]; + lookupResult.longitude = parts[6]; + } + } + if (addressNumberASN.containsKey(addressNumber)) { + String[] parts = addressNumberASN.get(addressNumber).split(" ", + 2); + lookupResult.aSNumber = parts[0]; + lookupResult.aSName = parts[1]; + } + lookupResults.put(addressString, lookupResult); + } + + return lookupResults; + } + + class LookupResult { + String countryCode; + String countryName; + String regionName; + String cityName; + String latitude; + String longitude; + String aSNumber; + String aSName; + } +} diff --git a/src/org/torproject/onionoo/Main.java b/src/org/torproject/onionoo/Main.java index f636b5e..3086054 100644 --- a/src/org/torproject/onionoo/Main.java +++ b/src/org/torproject/onionoo/Main.java @@ -19,8 +19,12 @@ public class Main { new File("out")); printStatusTime("Initialized document store"); + printStatus("Initializing lookup service."); + LookupService ls = new LookupService(new File("geoip")); + printStatusTime("Initialized Geoip lookup service"); + printStatus("Updating internal node list."); - CurrentNodes cn = new CurrentNodes(dso, ds); + CurrentNodes cn = new CurrentNodes(dso, ls, ds); cn.readStatusSummary(); printStatusTime("Read status summary"); cn.readRelayNetworkConsensuses(); diff --git a/test/org/torproject/onionoo/LookupServiceTest.java b/test/org/torproject/onionoo/LookupServiceTest.java new file mode 100644 index 0000000..42df61f --- /dev/null +++ b/test/org/torproject/onionoo/LookupServiceTest.java @@ -0,0 +1,501 @@ +/* Copyright 2013 The Tor Project + * See LICENSE for licensing information */ + +package org.torproject.onionoo; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +import java.io.BufferedWriter; +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; +import java.io.OutputStream; +import java.io.PrintStream; +import java.util.ArrayList; +import java.util.List; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeSet; + +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.torproject.onionoo.LookupService.LookupResult; + +public class LookupServiceTest { + + private static class LookupServiceTestHelper { + + private File tempGeoipDir; + + private List<String> manualGeoLiteCityBlocksLines, + automaticGeoLiteCityBlocksLines, geoLiteCityBlocksLines, + geoLiteCityLocationLines, iso3166Lines, regionLines, + geoipASNum2Lines; + + private LookupService lookupService; + + private SortedSet<String> addressStrings; + + private SortedMap<String, LookupResult> lookupResults; + + private LookupServiceTestHelper(File tempGeoipDir, + SortedSet<String> addressStrings) { + this.tempGeoipDir = tempGeoipDir; + this.addressStrings = addressStrings; + } + + private void populateLines() { + this.manualGeoLiteCityBlocksLines = new ArrayList<String>(); + this.manualGeoLiteCityBlocksLines.add( + "Copyright (c) 2011 MaxMind Inc. All Rights Reserved."); + this.manualGeoLiteCityBlocksLines.add("startIpNum,endIpNum,locId"); + this.manualGeoLiteCityBlocksLines.add("\"134739200\",\"134744063\"," + + "\"223\""); + this.manualGeoLiteCityBlocksLines.add("\"134744064\",\"134744319\"," + + "\"32191\""); + this.manualGeoLiteCityBlocksLines.add("\"134744320\",\"134751743\"," + + "\"223\""); + this.geoLiteCityLocationLines = new ArrayList<String>(); + this.geoLiteCityLocationLines.add("Copyright (c) 2012 MaxMind " + + "LLC. All Rights Reserved."); + this.geoLiteCityLocationLines.add("locId,country,region,city," + + "postalCode,latitude,longitude,metroCode,areaCode"); + this.geoLiteCityLocationLines.add("223,\"US\",\"\",\"\",\"\"," + + "38.0000,-97.0000,,"); + this.geoLiteCityLocationLines.add("32191,\"US\",\"CA\"," + + "\"Mountain View\",\"\",37.3860,-122.0838,807,650"); + this.iso3166Lines = new ArrayList<String>(); + this.iso3166Lines.add("US,\"United States\""); + this.regionLines = new ArrayList<String>(); + this.regionLines.add("US,CA,\"California\""); + this.geoipASNum2Lines = new ArrayList<String>(); + this.geoipASNum2Lines.add("134743296,134744063,\"AS3356 Level 3 " + + "Communications\""); + this.geoipASNum2Lines.add("134744064,134744319,\"AS15169 Google " + + "Inc.\""); + this.geoipASNum2Lines.add("134744320,134750463,\"AS3356 Level 3 " + + "Communications\""); + } + + private void writeCsvFiles() { + try { + this.writeCsvFile(this.manualGeoLiteCityBlocksLines, + "Manual-GeoLiteCity-Blocks.csv"); + this.writeCsvFile(this.automaticGeoLiteCityBlocksLines, + "Automatic-GeoLiteCity-Blocks.csv"); + this.writeCsvFile(this.geoLiteCityBlocksLines, + "GeoLiteCity-Blocks.csv"); + this.writeCsvFile(this.geoLiteCityLocationLines, + "GeoLiteCity-Location.csv"); + this.writeCsvFile(this.iso3166Lines, "iso3166.csv"); + this.writeCsvFile(this.regionLines, "region.csv"); + this.writeCsvFile(this.geoipASNum2Lines, "GeoIPASNum2.csv"); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + private void writeCsvFile(List<String> lines, String fileName) + throws IOException { + if (lines != null && !lines.isEmpty()) { + BufferedWriter bw = new BufferedWriter(new FileWriter( + new File(this.tempGeoipDir, fileName))); + for (String line : lines) { + bw.write(line + "\n"); + } + bw.close(); + } + } + + private void performLookups() { + this.lookupService = new LookupService(this.tempGeoipDir); + this.lookupResults = this.lookupService.lookup(this.addressStrings); + } + + private static void assertLookupResult(File tempGeoipDir, + List<String> manualGeoLiteCityBlocksLines, + List<String> automaticGeoLiteCityBlocksLines, + List<String> geoLiteCityBlocksLines, + List<String> geoLiteCityLocationLines, List<String> iso3166Lines, + List<String> regionLines, List<String> geoipASNum2Lines, + String addressString, String countryCode, String countryName, + String regionName, String cityName, String latitude, + String longitude, String aSNumber, String aSName) { + SortedSet<String> addressStrings = new TreeSet<String>(); + addressStrings.add(addressString); + LookupServiceTestHelper helper = new LookupServiceTestHelper( + tempGeoipDir, addressStrings); + helper.populateLines(); + if (manualGeoLiteCityBlocksLines != null) { + helper.manualGeoLiteCityBlocksLines = + manualGeoLiteCityBlocksLines; + } + if (automaticGeoLiteCityBlocksLines != null) { + helper.automaticGeoLiteCityBlocksLines = + automaticGeoLiteCityBlocksLines; + } + if (geoLiteCityBlocksLines != null) { + helper.geoLiteCityBlocksLines = geoLiteCityBlocksLines; + } + if (geoLiteCityLocationLines != null) { + helper.geoLiteCityLocationLines = geoLiteCityLocationLines; + } + if (iso3166Lines != null) { + helper.iso3166Lines = iso3166Lines; + } + if (regionLines != null) { + helper.regionLines = regionLines; + } + if (geoipASNum2Lines != null) { + helper.geoipASNum2Lines = geoipASNum2Lines; + } + helper.writeCsvFiles(); + /* Disable log messages printed to System.err. */ + System.setErr(new PrintStream(new OutputStream() { + public void write(int b) { + } + })); + helper.performLookups(); + if (countryCode == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).countryCode == null); + } else { + assertEquals(countryCode, + helper.lookupResults.get(addressString).countryCode); + } + if (countryName == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).countryName == null); + } else { + assertEquals(countryName, + helper.lookupResults.get(addressString).countryName); + } + if (regionName == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).regionName == null); + } else { + assertEquals(regionName, + helper.lookupResults.get(addressString).regionName); + } + if (cityName == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).cityName == null); + } else { + assertEquals(cityName, + helper.lookupResults.get(addressString).cityName); + } + if (latitude == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).latitude == null); + } else { + assertEquals(latitude, + helper.lookupResults.get(addressString).latitude); + } + if (longitude == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).longitude == null); + } else { + assertEquals(longitude, + helper.lookupResults.get(addressString).longitude); + } + if (aSNumber == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).aSNumber == null); + } else { + assertEquals(aSNumber, + helper.lookupResults.get(addressString).aSNumber); + } + if (aSName == null) { + assertTrue(!helper.lookupResults.containsKey(addressString) || + helper.lookupResults.get(addressString).aSName == null); + } else { + assertEquals(aSName, + helper.lookupResults.get(addressString).aSName); + } + } + } + + @Rule + public TemporaryFolder tempFolder = new TemporaryFolder(); + + private File tempGeoipDir; + + @Before + public void createTempGeoipDir() throws IOException { + this.tempGeoipDir = this.tempFolder.newFolder("geoip"); + } + + @Test() + public void testLookup8888() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, null, "8.8.8.8", "us", + "United States", "California", "Mountain View", "37.3860", + "-122.0838", "AS15169", "Google Inc."); + } + + @Test() + public void testLookup8880() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, null, "8.8.8.0", "us", + "United States", "California", "Mountain View", "37.3860", + "-122.0838", "AS15169", "Google Inc."); + } + + @Test() + public void testLookup888255() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, null, "8.8.8.255", "us", + "United States", "California", "Mountain View", "37.3860", + "-122.0838", "AS15169", "Google Inc."); + } + + @Test() + public void testLookup888256() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, null, "8.8.8.256", null, null, null, + null, null, null, null, null); + } + + @Test() + public void testLookup888Minus1() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, null, "8.8.8.-1", null, null, null, + null, null, null, null, null); + } + + @Test() + public void testLookup000() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, null, "0.0.0.0", null, null, null, + null, null, null, null, null); + } + + @Test() + public void testLookupNoBlocksLines() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, + new ArrayList<String>(), null, null, null, null, null, null, + "8.8.8.8", null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupNoLocationLines() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, new ArrayList<String>(), null, null, null, "8.8.8.8", + null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupNoIso3166Lines() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, new ArrayList<String>(), null, null, "8.8.8.8", + null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupNoRegionLines() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, new ArrayList<String>(), null, "8.8.8.8", + null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupNoGeoipASNum2Lines() { + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, new ArrayList<String>(), "8.8.8.8", + null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupNoCorrespondingLocation() { + List<String> geoLiteCityLocationLines = new ArrayList<String>(); + geoLiteCityLocationLines.add("Copyright (c) 2012 MaxMind LLC. All " + + "Rights Reserved."); + geoLiteCityLocationLines.add("locId,country,region,city,postalCode," + + "latitude,longitude,metroCode,areaCode"); + geoLiteCityLocationLines.add("223,\"US\",\"\",\"\",\"\",38.0000," + + "-97.0000,,"); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, geoLiteCityLocationLines, null, null, null, "8.8.8.8", + null, null, null, null, null, null, "AS15169", "Google Inc."); + } + + @Test() + public void testLookupNoCorrespondingCountryName() { + List<String> iso3166Lines = new ArrayList<String>(); + iso3166Lines.add("UY,\"Uruguay\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, iso3166Lines, null, null, "8.8.8.8", "us", + null, "California", "Mountain View", "37.3860", "-122.0838", + "AS15169", "Google Inc."); + } + + @Test() + public void testLookupNoCorrespondingRegionName() { + List<String> regionLines = new ArrayList<String>(); + regionLines.add("US,CO,\"Colorado\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, regionLines, null, "8.8.8.8", "us", + "United States", null, "Mountain View", "37.3860", "-122.0838", + "AS15169", "Google Inc."); + } + + @Test() + public void testLookupBlocksEndBeforeStart() { + List<String> manualGeoLiteCityBlocksLines = new ArrayList<String>(); + manualGeoLiteCityBlocksLines.add("Copyright (c) 2011 MaxMind Inc. " + + "All Rights Reserved."); + manualGeoLiteCityBlocksLines.add("startIpNum,endIpNum,locId"); + manualGeoLiteCityBlocksLines.add("\"134739200\",\"134744063\"," + + "\"223\""); + manualGeoLiteCityBlocksLines.add("\"134744319\",\"134744064\"," + + "\"32191\""); + manualGeoLiteCityBlocksLines.add("\"134744320\",\"134751743\"," + + "\"223\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, + manualGeoLiteCityBlocksLines, null, null, null, null, null, null, + "8.8.8.8", null, null, null, null, null, null, "AS15169", + "Google Inc."); + } + + @Test() + public void testLookupBlocksStartNotANumber() { + List<String> manualGeoLiteCityBlocksLines = new ArrayList<String>(); + manualGeoLiteCityBlocksLines.add("Copyright (c) 2011 MaxMind Inc. " + + "All Rights Reserved."); + manualGeoLiteCityBlocksLines.add("startIpNum,endIpNum,locId"); + manualGeoLiteCityBlocksLines.add("\"one\",\"134744319\"," + + "\"32191\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, + manualGeoLiteCityBlocksLines, null, null, null, null, null, null, + "8.8.8.8", null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupBlocksStartTooLarge() { + List<String> manualGeoLiteCityBlocksLines = new ArrayList<String>(); + manualGeoLiteCityBlocksLines.add("Copyright (c) 2011 MaxMind Inc. " + + "All Rights Reserved."); + manualGeoLiteCityBlocksLines.add("startIpNum,endIpNum,locId"); + manualGeoLiteCityBlocksLines.add("\"1" + + String.valueOf(Long.MAX_VALUE) + "\",\"134744319\",\"32191\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, + manualGeoLiteCityBlocksLines, null, null, null, null, null, null, + "8.8.8.8", null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupBlocksLocationX() { + List<String> manualGeoLiteCityBlocksLines = new ArrayList<String>(); + manualGeoLiteCityBlocksLines.add("Copyright (c) 2011 MaxMind Inc. " + + "All Rights Reserved."); + manualGeoLiteCityBlocksLines.add("startIpNum,endIpNum,locId"); + manualGeoLiteCityBlocksLines.add("\"134744064\",\"134744319\",\"X\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, + manualGeoLiteCityBlocksLines, null, null, null, null, null, null, + "8.8.8.8", null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupBlocksTooFewFields() { + List<String> manualGeoLiteCityBlocksLines = new ArrayList<String>(); + manualGeoLiteCityBlocksLines.add("Copyright (c) 2011 MaxMind Inc. " + + "All Rights Reserved."); + manualGeoLiteCityBlocksLines.add("startIpNum,endIpNum,locId"); + manualGeoLiteCityBlocksLines.add("\"134744064\",\"134744319\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, + manualGeoLiteCityBlocksLines, null, null, null, null, null, null, + "8.8.8.8", null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupLocationLocIdNotANumber() { + List<String> geoLiteCityLocationLines = new ArrayList<String>(); + geoLiteCityLocationLines.add("Copyright (c) 2012 MaxMind LLC. All " + + "Rights Reserved."); + geoLiteCityLocationLines.add("locId,country,region,city,postalCode," + + "latitude,longitude,metroCode,areaCode"); + geoLiteCityLocationLines.add("threetwoonenineone,\"US\",\"CA\"," + + "\"Mountain View\",\"\",37.3860,-122.0838,807,650"); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, geoLiteCityLocationLines, null, null, null, "8.8.8.8", + null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupLocationTooFewFields() { + List<String> geoLiteCityLocationLines = new ArrayList<String>(); + geoLiteCityLocationLines.add("Copyright (c) 2012 MaxMind LLC. All " + + "Rights Reserved."); + geoLiteCityLocationLines.add("locId,country,region,city,postalCode," + + "latitude,longitude,metroCode,areaCode"); + geoLiteCityLocationLines.add("32191,\"US\",\"CA\",\"Mountain View\"," + + "\"\",37.3860,-122.0838,807"); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, geoLiteCityLocationLines, null, null, null, "8.8.8.8", + null, null, null, null, null, null, null, null); + } + + @Test() + public void testLookupIso3166TooFewFields() { + List<String> iso3166Lines = new ArrayList<String>(); + iso3166Lines.add("US"); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, iso3166Lines, null, null, "8.8.8.8", null, null, + null, null, null, null, null, null); + } + + @Test() + public void testLookupRegionTooFewFields() { + List<String> regionLines = new ArrayList<String>(); + regionLines.add("US,CA"); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, regionLines, null, "8.8.8.8", null, null, + null, null, null, null, null, null); + } + + @Test() + public void testLookupGeoipASNum2EndBeforeStart() { + List<String> geoipASNum2Lines = new ArrayList<String>(); + geoipASNum2Lines.add("134743296,134744063,\"AS3356 Level 3 " + + "Communications\""); + geoipASNum2Lines.add("134744319,134744064,\"AS15169 Google Inc.\""); + geoipASNum2Lines.add("134744320,134750463,\"AS3356 Level 3 " + + "Communications\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, geoipASNum2Lines, "8.8.8.8", "us", + "United States", "California", "Mountain View", "37.3860", + "-122.0838", null, null); + } + + @Test() + public void testLookupGeoipASNum2StartNotANumber() { + List<String> geoipASNum2Lines = new ArrayList<String>(); + geoipASNum2Lines.add("one,134744319,\"AS15169 Google Inc.\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, geoipASNum2Lines, "8.8.8.8", null, + null, null, null, null, null, null, null); + } + + @Test() + public void testLookupGeoipASNum2StartTooLarge() { + List<String> geoipASNum2Lines = new ArrayList<String>(); + geoipASNum2Lines.add("1" + String.valueOf(Long.MAX_VALUE) + + ",134744319,\"AS15169 Google Inc.\""); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, geoipASNum2Lines, "8.8.8.8", null, + null, null, null, null, null, null, null); + } + + @Test() + public void testLookupGeoipASNum2TooFewFields() { + List<String> geoipASNum2Lines = new ArrayList<String>(); + geoipASNum2Lines.add("134744064,134744319"); + LookupServiceTestHelper.assertLookupResult(this.tempGeoipDir, null, + null, null, null, null, null, geoipASNum2Lines, "8.8.8.8", null, + null, null, null, null, null, null, null); + } +} +

1 0

[ooni-probe/master] Update changelog
by art＠torproject.org 19 Jun '13

19 Jun '13

commit 514b707df4c701a1e4c58f726c0261e26fabdb32 Author: Arturo Filastò <art(a)fuffa.org> Date: Wed Jun 19 14:44:35 2013 +0200 Update changelog --- ChangeLog.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/ChangeLog.md b/ChangeLog.md index 7d22967..93379ce 100644 --- a/ChangeLog.md +++ b/ChangeLog.md @@ -1,3 +1,33 @@ +## v0.1.0 + +Improvements to HTML/JS based user interface: + + * XSRF protection + + * user supplied input specification + +Bugfixing and improvements to scheduler. + +## v0.0.12 + +Implement JS/HTML based user interface. + +Supports: + + * Starting and stopping of tests + + * Monitoring of test progress + +## v0.0.11 + +* Parametrize task timeout and retry count + +* Set the default collector via the command line option + +* Add option to disable the default collector + +* Add continuous integration with travis + ## v0.0.10 ### ooniprobe

1 0

[ooni-probe/master] Point to data directory in short-not-root deck
by art＠torproject.org 19 Jun '13

19 Jun '13

commit 673236cf5e872d13aff403260b70d711b2fa6ad1 Author: Arturo Filastò <art(a)fuffa.org> Date: Tue Apr 30 09:18:07 2013 +0200 Point to data directory in short-not-root deck --- decks/short_no_root.deck | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/decks/short_no_root.deck b/decks/short_no_root.deck index 18117f5..fbc8f7b 100644 --- a/decks/short_no_root.deck +++ b/decks/short_no_root.deck @@ -8,7 +8,7 @@ reportfile: null resume: 0 subargs: [] - test_file: nettests/manipulation/captiveportal.py + test_file: data/nettests/manipulation/captiveportal.py testdeck: null - options: collector: null @@ -20,7 +20,7 @@ reportfile: null resume: 0 subargs: [-f, example_inputs/url_lists_file.txt] - test_file: nettests/blocking/http_requests.py + test_file: data/nettests/blocking/http_requests.py testdeck: null - options: collector: null @@ -32,7 +32,7 @@ reportfile: null resume: 0 subargs: [-f, example_inputs/tcpconnect_host_file.txt] - test_file: nettests/blocking/tcpconnect.py + test_file: data/nettests/blocking/tcpconnect.py testdeck: null - options: collector: null @@ -44,7 +44,7 @@ reportfile: null resume: 0 subargs: [-f, example_inputs/http_host_file.txt, -T, example_inputs/dns_tamper_test_resolvers.txt] - test_file: nettests/blocking/dnsconsistency.py + test_file: data/nettests/blocking/dnsconsistency.py testdeck: null - options: collector: null @@ -56,5 +56,5 @@ reportfile: null resume: 0 subargs: [-b, 'http://93.95.227.200'] - test_file: nettests/manipulation/http_header_field_manipulation.py + test_file: data/nettests/manipulation/http_header_field_manipulation.py testdeck: null

1 0

[ooni-probe/master] Improve setup.py, fixup paths for reporting.
by art＠torproject.org 19 Jun '13

19 Jun '13

commit 3b889359329f6a90fd6d5538f6bf4e417cdf9b00 Author: Arturo Filastò <art(a)fuffa.org> Date: Mon Apr 29 13:17:37 2013 +0200 Improve setup.py, fixup paths for reporting. --- data/ooniprobe.conf.sample | 13 +++++----- decks/before_i_commit.testdeck | 8 +++--- ooni/__init__.py | 2 +- ooni/api/spec.py | 55 +++++++++++++++++++++++++++++++++++++--- ooni/nettest.py | 8 ++++-- ooni/reporter.py | 6 +++++ setup.py | 17 ++++++++++--- 7 files changed, 89 insertions(+), 20 deletions(-) diff --git a/data/ooniprobe.conf.sample b/data/ooniprobe.conf.sample index 5528199..2b79d3e 100644 --- a/data/ooniprobe.conf.sample +++ b/data/ooniprobe.conf.sample @@ -4,7 +4,7 @@ basic: # Where OONIProbe should be writing it's log file - logfile: ooniprobe.log + logfile: /var/log/ooniprobe.log privacy: # Should we include the IP address of the probe in the report? includeip: false @@ -18,14 +18,13 @@ privacy: includepcap: false reports: # This is a packet capture file (.pcap) to load as a test: - pcap: Null + pcap: null + collector: 'httpo://nkvphnp3p6agi5qq.onion' advanced: - # XXX change this to point to the directory where you have stored the GeoIP - # database file. This should be the directory in which OONI is installed - # /path/to/ooni-probe/data/ - geoip_data_dir: /usr/share/GeoIP/ + geoip_data_dir: /usr/share/ooni/ debug: true - tor_binary: /usr/sbin/tor + # enable if auto detection fails + #tor_binary: /usr/sbin/tor # For auto detection interface: auto # Of specify a specific interface diff --git a/decks/before_i_commit.testdeck b/decks/before_i_commit.testdeck index d1b4062..1159a0d 100644 --- a/decks/before_i_commit.testdeck +++ b/decks/before_i_commit.testdeck @@ -5,7 +5,7 @@ pcapfile: null reportfile: reports/captive_portal_test.yamloo subargs: [] - test_file: nettests/manipulation/captiveportal.py + test_file: data/nettests/manipulation/captiveportal.py - options: collector: null help: 0 @@ -13,7 +13,7 @@ pcapfile: null reportfile: reports/dns_tamper_test.yamloo subargs: [-T, example_inputs/dns_tamper_test_resolvers.txt, -f, example_inputs/dns_tamper_file.txt] - test_file: nettests/blocking/dnsconsistency.py + test_file: data/nettests/blocking/dnsconsistency.py - options: collector: null help: 0 @@ -21,7 +21,7 @@ pcapfile: null reportfile: reports/http_host.yamloo subargs: [-b, 'http://93.95.227.200', -f, example_inputs/http_host_file.txt] - test_file: nettests/manipulation/http_host.py + test_file: data/nettests/manipulation/http_host.py - options: collector: null help: 0 @@ -29,4 +29,4 @@ pcapfile: null reportfile: reports/header_field_manipulation.yamloo subargs: [-b, 'http://93.95.227.200'] - test_file: nettests/manipulation/http_header_field_manipulation.py + test_file: data/nettests/manipulation/http_header_field_manipulation.py diff --git a/ooni/__init__.py b/ooni/__init__.py index 815e16e..1810a7f 100644 --- a/ooni/__init__.py +++ b/ooni/__init__.py @@ -7,7 +7,7 @@ from . import templates from . import utils __author__ = "Arturo Filastò" -__version__ = "0.0.11" +__version__ = "0.0.12" __all__ = ['config', 'inputunit', 'kit', 'lib', 'nettest', 'oonicli', 'reporter', diff --git a/ooni/api/spec.py b/ooni/api/spec.py index 39df2fd..071beaf 100644 --- a/ooni/api/spec.py +++ b/ooni/api/spec.py @@ -7,7 +7,7 @@ import types from twisted.python import usage from cyclone import web, escape -from ooni.reporter import YAMLReporter, OONIBReporter +from ooni.reporter import YAMLReporter, OONIBReporter, collector_supported from ooni import errors from ooni.nettest import NetTestLoader, MissingRequiredOption from ooni.settings import config @@ -43,11 +43,21 @@ def list_inputs(): return input_list class Inputs(ORequestHandler): + """ + This handler is responsible for listing and adding new inputs. + """ def get(self): + """ + Obtain the list of currently installed inputs. Inputs are stored inside + of $OONI_HOME/inputs/. + """ input_list = list_inputs() self.write(input_list) def post(self): + """ + Add a new input to the currently installed inputs. + """ input_file = self.request.files.get("file")[0] filename = input_file['filename'] @@ -72,6 +82,15 @@ class ListTests(ORequestHandler): self.write(test_list) def get_net_test_loader(test_options, test_file): + """ + Args: + test_options: (dict) containing as keys the option names. + + test_file: (string) the path to the test_file to be run. + Returns: + an instance of :class:`ooni.nettest.NetTestLoader` with the specified + test_file and the specified options. + """ options = [] for k, v in test_options.items(): options.append('--'+k) @@ -82,10 +101,26 @@ def get_net_test_loader(test_options, test_file): return net_test_loader def get_reporters(net_test_loader): + """ + Determines which reports are able to run and returns an instance of them. + + We always report to flat file via the :class:`ooni.reporters.YAMLReporter` + and the :class:`ooni.reporters.OONIBReporter`. + + The later will be used only if we determine that Tor is running. + + Returns: + a list of reporter instances + """ test_details = net_test_loader.testDetails + reporters = [] yaml_reporter = YAMLReporter(test_details, config.reports_directory) - #oonib_reporter = OONIBReporter(test_details, collector) - return [yaml_reporter] + reporters.append(yaml_reporter) + + if config.reports.collector and collector_supported(config.reports.collector): + oonib_reporter = OONIBReporter(test_details, collector) + reporters.append(oonib_reporter) + return reporters class StartTest(ORequestHandler): def post(self, test_name): @@ -114,6 +149,16 @@ class StopTest(ORequestHandler): pass def get_test_results(test_id): + """ + Returns: + a list of test dicts that correspond to the test results for the given + test_id. + The dict is made like so: + { + 'name': The name of the report, + 'content': The content of the report + } + """ test_results = [] for test_result in os.listdir(config.reports_directory): if test_result.startswith('report-'+test_id): @@ -126,6 +171,10 @@ def get_test_results(test_id): class TestStatus(ORequestHandler): def get(self, test_id): + """ + Returns the requested test_id details and the stored results for such + test. + """ try: test = copy.deepcopy(oonidApplication.director.netTests[test_id]) test.pop('path') diff --git a/ooni/nettest.py b/ooni/nettest.py index 964bee3..66639f8 100644 --- a/ooni/nettest.py +++ b/ooni/nettest.py @@ -238,6 +238,10 @@ class NetTestLoader(object): if not hasattr(usage_options, 'optParameters'): usage_options.optParameters = [] + else: + for parameter in usage_options.optParameters: + if len(parameter) == 5: + parameter.pop() if klass.inputFile: usage_options.optParameters.append(klass.inputFile) @@ -267,8 +271,8 @@ class NetTestLoader(object): def loadNetTestString(self, net_test_string): """ Load NetTest from a string. - WARNING input to this function *MUST* be sanitized and *NEVER* be - untrusted. + WARNING input to this function *MUST* be sanitized and *NEVER* take + untrusted input. Failure to do so will result in code exec. net_test_string: diff --git a/ooni/reporter.py b/ooni/reporter.py index 109eccf..1095cb1 100644 --- a/ooni/reporter.py +++ b/ooni/reporter.py @@ -227,6 +227,12 @@ class YAMLReporter(OReporter): def finish(self): self._stream.close() +def collector_supported(collector_address): + if collector_address.startswith('httpo') \ + and (not (config.tor_state or config.tor.socks_port)): + return False + return True + class OONIBReporter(OReporter): def __init__(self, test_details, collector_address): self.collectorAddress = collector_address diff --git a/setup.py b/setup.py index 41b9050..454984d 100644 --- a/setup.py +++ b/setup.py @@ -1,7 +1,9 @@ #!/usr/bin/env python #-*- coding: utf-8 -*- -from setuptools import setup +import os +import sys +from distutils.core import setup install_requires = [ 'txsocksx>=0.0.2', @@ -16,6 +18,14 @@ dependency_links = [ 'https://github.com/hellais/pypcap/archive/v1.1.1.tar.gz#egg=pypcap-1.1.1' ] +files = [] +for root, dirs, file_names in os.walk('data/'): + for file_name in file_names: + if not file_name.endswith('.pyc'): + files.append(os.path.join(root, file_name)) + +data_files = [('/usr/share/ooni/', files)] + with open('requirements.txt') as f: for line in f: if line.startswith("#") or line.startswith('http'): @@ -24,12 +34,13 @@ with open('requirements.txt') as f: setup( name="ooni-probe", - version="0.0.11", + version="0.0.12", author="Arturo Filastò", author_email = "art(a)torproject.org", url="https://ooni.torproject.org/", package_dir={'ooni': 'ooni'}, - packages=['ooni', 'ooni.templates', 'ooni.utils'], + data_files=data_files, + packages=['ooni', 'ooni.api', 'ooni.templates', 'ooni.tests', 'ooni.utils'], scripts=["bin/ooniprobe"], dependency_links=dependency_links, install_requires=install_requires,

1 0

[ooni-probe/master] Add support for viewing test results and uploading inputs
by art＠torproject.org 19 Jun '13

19 Jun '13

commit 4f51cd3107bc1f9e65be587aeb3b41f7a584e3d7 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Apr 25 13:33:46 2013 +0200 Add support for viewing test results and uploading inputs --- data/ui/app/index.html | 1 + data/ui/app/libs/ng-upload/ng-upload.js | 107 +++++++++++++++++++++++++++++++ data/ui/app/scripts/app.js | 21 +++--- data/ui/app/scripts/controllers.js | 36 ++++++++--- data/ui/app/scripts/services.js | 9 ++- data/ui/app/styles/app.css | 5 ++ data/ui/app/views/inputs.html | 31 +++++++++ data/ui/app/views/sidebar.html | 4 ++ data/ui/app/views/test-list.html | 38 ----------- data/ui/app/views/test-status.html | 7 -- data/ui/app/views/test.html | 42 ++++++++++++ ooni/api/spec.py | 6 +- 12 files changed, 239 insertions(+), 68 deletions(-) diff --git a/data/ui/app/index.html b/data/ui/app/index.html index a903ceb..c304066 100644 --- a/data/ui/app/index.html +++ b/data/ui/app/index.html @@ -28,6 +28,7 @@ <script src="libs/angular/angular.js"></script> <script src="libs/angular-resource/angular-resource.js"></script> + <script src="libs/ng-upload/ng-upload.js"></script> <script src="scripts/app.js"></script> <script src="scripts/services.js"></script> <script src="scripts/controllers.js"></script> diff --git a/data/ui/app/libs/ng-upload/ng-upload.js b/data/ui/app/libs/ng-upload/ng-upload.js new file mode 100644 index 0000000..fae7a44 --- /dev/null +++ b/data/ui/app/libs/ng-upload/ng-upload.js @@ -0,0 +1,107 @@ +// Version 0.3.2 +// AngularJS simple file upload directive +// this directive uses an iframe as a target +// to enable the uploading of files without +// losing focus in the ng-app. +// +// <div ng-app="app"> +// <div ng-controller="mainCtrl"> +// <form action="/uploads" ng-upload> +// <input type="file" name="avatar"></input> +// <input type="submit" value="Upload" +// upload-submit="submited(content, completed)"></input> +// </form> +// </div> +// </div> +// +// angular.module('app', ['ngUpload']) +// .controller('mainCtrl', function($scope) { +// $scope.submited = function(content, completed) { +// if (completed) { +// console.log(content); +// } +// } +// }); +// +angular.module('ngUpload', []) + .directive('uploadSubmit', ['$parse', function($parse) { + return { + restrict: 'AC', + link: function(scope, element, attrs) { + // Options (just 1 for now) + // Each option should be prefixed with 'upload-options-' or 'uploadOptions' + // { + // // specify whether to enable the submit button when uploading forms + // enableControls: bool + // } + var options = {}; + options.enableControls = attrs.uploadOptionsEnableControls; + + // submit the form - requires jQuery + var form = element.parents('form[ng-upload]') || element.parents('form.ng-upload'); + + // Retrieve the callback function + var fn = $parse(attrs.uploadSubmit); + + if (!angular.isFunction(fn)) { + var message = "The expression on the ngUpload directive does not point to a valid function."; + throw message + "\n"; + } + + element.bind('click', function($event) { + // prevent default behavior of click + $event.preventDefault = true; + // create a new iframe + var iframe = angular.element("<iframe id='upload_iframe' name='upload_iframe' border='0' width='0' height='0' style='width: 0px; height: 0px; border: none; display: none' />"); + + // attach function to load event of the iframe + iframe.bind('load', function () { + // get content - requires jQuery + var content = iframe.contents().find('body').text(); + // execute the upload response function in the active scope + scope.$apply(function () { + fn(scope, { content: content, completed: true}); + }); + // remove iframe + if (content !== "") { // Fixes a bug in Google Chrome that dispose the iframe before content is ready. + setTimeout(function () { iframe.remove(); }, 250); + } + element.attr('disabled', null); + element.attr('title', 'Click to start upload.'); + }); + + // add the new iframe to application + form.parent().append(iframe); + + scope.$apply(function () { + fn(scope, {content: "Please wait...", completed: false }); + }); + + var enabled = true; + if (!options.enableControls) { + // disable the submit control on click + element.attr('disabled', 'disabled'); + enabled = false; + } + // why do we need this??? + element.attr('title', (enabled ? '[ENABLED]: ' : '[DISABLED]: ') + 'Uploading, please wait...'); + + form.submit(); + + }).attr('title', 'Click to start upload.'); + } + }; + }]) + .directive('ngUpload', ['$parse', function ($parse) { + return { + restrict: 'AC', + link: function (scope, element, attrs) { + element.attr("target", "upload_iframe"); + element.attr("method", "post"); + // Append a timestamp field to the url to prevent browser caching results + element.attr("action", element.attr("action") + "?_t=" + new Date().getTime()); + element.attr("enctype", "multipart/form-data"); + element.attr("encoding", "multipart/form-data"); + } + }; + }]); \ No newline at end of file diff --git a/data/ui/app/scripts/app.js b/data/ui/app/scripts/app.js index 5fb8fdf..d36a17c 100644 --- a/data/ui/app/scripts/app.js +++ b/data/ui/app/scripts/app.js @@ -2,28 +2,29 @@ // Declare app level module which depends on filters, and services -var ooniprobe = angular.module('ooniprobe', ['ooniprobe.services']). +var ooniprobe = angular.module('ooniprobe', ['ngUpload', 'ooniprobe.services']). config(['$routeProvider', function($routeProvider) { - $routeProvider.when('/test-status', + + $routeProvider.when('/inputs', { - templateUrl: 'views/test-status.html', - controller: 'PageCtrl' + templateUrl: 'views/inputs.html', + controller: 'InputsCtrl' } ); - $routeProvider.when('/test-list', + $routeProvider.when('/settings', { - templateUrl: 'views/test-list.html', - controller: 'TestListCtrl' + templateUrl: 'views/settings.html', + controller: 'SettingsCtrl' } ); $routeProvider.when('/test/:testID', { - templateUrl: 'views/test-list.html', - controller: 'TestListCtrl' + templateUrl: 'views/test.html', + controller: 'TestCtrl' } ); - $routeProvider.otherwise({redirectTo: '/test-status'}); + $routeProvider.otherwise({redirectTo: '/settings'}); }]); diff --git a/data/ui/app/scripts/controllers.js b/data/ui/app/scripts/controllers.js index 489dfd2..5ba24e1 100644 --- a/data/ui/app/scripts/controllers.js +++ b/data/ui/app/scripts/controllers.js @@ -3,16 +3,16 @@ ooniprobe.controller('PageCtrl', ['$scope', function($scope) { }]); -ooniprobe.controller('TestListCtrl', ['$scope', '$routeParams', 'testStatus', - function($scope, $routeParams, testStatus) { +ooniprobe.controller('SettingsCtrl', ['$scope', + function($scope) { +}]); - var testID = $routeParams['testID']; - $scope.updateTestStatus = function() { - testStatus(testID).success(function(testDetails){ - $scope.testDetails = testDetails; - }); +ooniprobe.controller('InputsCtrl', ['$scope', 'Inputs', + function($scope, Inputs) { + $scope.inputs = Inputs.query(); + $scope.uploadComplete = function(contents, completed) { + return; } - $scope.updateTestStatus(); }]); @@ -30,13 +30,31 @@ ooniprobe.controller('SideBarCtrl', ['$scope', 'listTests', '$location', }]); +ooniprobe.controller('TestCtrl', ['$scope', '$routeParams', 'testStatus', 'Inputs', + function($scope, $routeParams, testStatus, Inputs) { + + var testID = $routeParams['testID']; + + $scope.inputs = Inputs.query(); + + $scope.updateTestStatus = function() { + testStatus(testID).success(function(testDetails){ + $scope.testDetails = testDetails; + }); + } + $scope.updateTestStatus(); + + +}]); + ooniprobe.controller('TestBoxCtrl', ['$scope', 'startTest', function($scope, startTest) { $scope.startTest = function() { var options = {}; - angular.forEach($scope.testDetails.arguments, function(option, key){ + angular.forEach($scope.testDetails.arguments, + function(option, key) { options[key] = option.value; }); diff --git a/data/ui/app/scripts/services.js b/data/ui/app/scripts/services.js index 3d6721d..ea013ee 100644 --- a/data/ui/app/scripts/services.js +++ b/data/ui/app/scripts/services.js @@ -2,7 +2,7 @@ angular.module('ooniprobe.services', ['ngResource']). factory('listTests', ['$resource', - function($resource){ + function($resource) { return $resource('/test'); }]). factory('testStatus', ['$http', function($http){ @@ -10,11 +10,16 @@ angular.module('ooniprobe.services', ['ngResource']). return $http.get('/test/' + testID); } }]). - factory('startTest', ['$http', function($http){ + factory('startTest', ['$http', + function($http) { return function(testID, options) { return $http.post('/test/' + testID + '/start', options); } }]). + factory('Inputs', ['$resource', + function($resource) { + return $resource('/inputs'); +}]). factory('status', ['$resource', function($resource) { return $resource('/status'); diff --git a/data/ui/app/styles/app.css b/data/ui/app/styles/app.css index 5fe454f..b4a0fb8 100644 --- a/data/ui/app/styles/app.css +++ b/data/ui/app/styles/app.css @@ -14,3 +14,8 @@ background-color: rgb(240, 240, 240); } +.testResult { + height: 200px; + overflow-y: scroll; + overflow-x: hidden; +} diff --git a/data/ui/app/views/inputs.html b/data/ui/app/views/inputs.html new file mode 100644 index 0000000..767e00e --- /dev/null +++ b/data/ui/app/views/inputs.html @@ -0,0 +1,31 @@ +<div class="row"> + <div class="span8"> + <h2>Inputs</h2> + <ul class="unstyled"> + <li ng-repeat="input in inputs">{{input.filename}} +  + </li> + </ul> + <form ng-upload action="/inputs"> + + <h4>Add file</h4> + <label>file</label> + <input type="file" name="file" /> + <br/> + <button class="btn" + upload-submit="uploadComplete(contents, completed)">Upload</button> + +  + </form> + + + </div> +</div> + diff --git a/data/ui/app/views/settings.html b/data/ui/app/views/settings.html new file mode 100644 index 0000000..e69de29 diff --git a/data/ui/app/views/sidebar.html b/data/ui/app/views/sidebar.html index 172a51e..5ab4630 100644 --- a/data/ui/app/views/sidebar.html +++ b/data/ui/app/views/sidebar.html @@ -3,4 +3,8 @@ <li ng-repeat="test in test_list" ng-class="{'active': testSelected(test.id)}"> <a href="#/test/{{test.id}}">{{test.name}}</a> </li> + + <li class="nav-header">Configuration</li> + <li><a href="#/inputs">Inputs</a></li> + <li><a href="#/settings">Settings</a></li> </ul> diff --git a/data/ui/app/views/test-list.html b/data/ui/app/views/test-list.html deleted file mode 100644 index ffcf3fb..0000000 --- a/data/ui/app/views/test-list.html +++ /dev/null @@ -1,38 +0,0 @@ -<div class="row"> - <div class="span8"> - <h2>{{testDetails.name}}</h2> - <div class="netTest" ng-controller="TestBoxCtrl"> - version: <span class="badge badge-success">{{testDetails.version}}</span> - <p>{{testDetails.description}}</p> - <form name="testOptions"> - <div ng-repeat="(name, options) in testDetails.arguments"> - <div ng-switch on="options.type"> - - <div ng-switch-when="file"> - <label>{{name}}</label> - <input type="file" name="{{name}}"> - </div> - - <div ng-switch-default> - <label>{{name}}</label> - <input ng-model="testDetails.arguments[name].value" type="{{options.type}}" - value="{{options.default}}"> - </div> - - </div> - </div> - </form> - <button class="btn btn-primary" ng-click="startTest()">Start Test</button> - </div> - </div> -</div> - -<div class="row"> - <div class="span8"> - <h3>Test results</h3> - <div class="testResult" ng-repeat="result in testDetails.results"> - <h4>{{result.name}}</h4> - <pre>{{result.content}}</pre> - </div> - </div> -</div> diff --git a/data/ui/app/views/test-status.html b/data/ui/app/views/test-status.html deleted file mode 100644 index a457119..0000000 --- a/data/ui/app/views/test-status.html +++ /dev/null @@ -1,7 +0,0 @@ -<h2>Test Status</h2> -<p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod -tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, -quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo -consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse -cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non -proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p> diff --git a/data/ui/app/views/test.html b/data/ui/app/views/test.html new file mode 100644 index 0000000..f92461a --- /dev/null +++ b/data/ui/app/views/test.html @@ -0,0 +1,42 @@ +<div class="row"> + <div class="span8"> + <h2>{{testDetails.name}}</h2> + <div class="netTest" ng-controller="TestBoxCtrl"> + version: <span class="badge badge-success">{{testDetails.version}}</span> + <p>{{testDetails.description}}</p> + <form name="testOptions"> + <div ng-repeat="(name, options) in testDetails.arguments"> + <div ng-switch on="options.type"> + + <div ng-switch-when="file"> + <label>{{name}}</label> + <select ng-model="testDetails.arguments[name].value"> + <option ng-repeat="input in inputs" value="input.filename">{{input.filename}}</option> + </select> + </div> + + <div ng-switch-default> + <label>{{name}}</label> + <input ng-model="testDetails.arguments[name].value" type="{{options.type}}" + value="{{options.default}}"> + </div> + + </div> + </div> + </form> + <button class="btn btn-primary" ng-click="startTest()">Start Test</button> + </div> + </div> +</div> + +<div class="row"> + <div class="span8"> + <h3>Test results</h3> + <button class="btn" ng-click="updateTestStatus()"> + <i class="icon-refresh"></i>Reload</button> + <div ng-repeat="result in testDetails.results"> + <h4>{{result.name}}</h4> + <pre class="testResult">{{result.content}}</pre> + </div> + </div> +</div> diff --git a/ooni/api/spec.py b/ooni/api/spec.py index ec4cf4b..39df2fd 100644 --- a/ooni/api/spec.py +++ b/ooni/api/spec.py @@ -48,14 +48,15 @@ class Inputs(ORequestHandler): self.write(input_list) def post(self): - filename = self.get_argument("fullname", None) + input_file = self.request.files.get("file")[0] + filename = input_file['filename'] + if not filename or not re.match('(\w.*\.\w.*).*', filename): raise InvalidInputFilename if os.path.exists(filename): raise FilenameExists - input_file = self.request.files.get("input_file") content_type = input_file["content_type"] body = input_file["body"] @@ -120,6 +121,7 @@ def get_test_results(test_id): test_content = ''.join(f.readlines()) test_results.append({'name': test_result, 'content': test_content}) + test_results.reverse() return test_results class TestStatus(ORequestHandler):

1 0

[ooni-probe/master] Remove oonib related data from ooniprobe
by art＠torproject.org 19 Jun '13

19 Jun '13

commit 42b43f0a8ae841546f00f49e1b26b511e42ec1b0 Author: Arturo Filastò <art(a)fuffa.org> Date: Thu Apr 25 16:05:01 2013 +0200 Remove oonib related data from ooniprobe --- bin/oonib | 36 ------- oonib/INSTALL | 4 - oonib/README.md | 117 ---------------------- oonib/__init__.py | 24 ----- oonib/config.py | 58 ----------- oonib/db/__init__.py | 30 ------ oonib/db/tables.py | 123 ----------------------- oonib/models.py | 129 ------------------------- oonib/oonibackend.py | 74 -------------- oonib/report/__init__.py | 5 - oonib/report/api.py | 106 -------------------- oonib/report/file_collector.py | 193 ------------------------------------- oonib/requirements.txt | 24 ----- oonib/runner.py | 81 ---------------- oonib/testhelpers/__init__.py | 5 - oonib/testhelpers/dns_helpers.py | 16 --- oonib/testhelpers/http_helpers.py | 154 ----------------------------- oonib/testhelpers/ssl_helpers.py | 9 -- oonib/testhelpers/tcp_helpers.py | 72 -------------- 19 files changed, 1260 deletions(-) diff --git a/bin/oonib b/bin/oonib deleted file mode 100755 index 79885ba..0000000 --- a/bin/oonib +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env python - -import sys -import os -# Avoid the export PYTHONPATH insanity -sys.path[:] = map(os.path.abspath, sys.path) -this_directory = os.path.dirname(__file__) -root = os.path.abspath(os.path.join(this_directory, '..')) -sys.path.insert(0, root) - -backend_script = os.path.join(root, 'oonib', 'oonibackend.py') - -from twisted.python import log, usage -from twisted.internet import reactor -from twisted.application import app - -from oonib import runner -from oonib.oonibackend import application - -sys.argv[1:] = ['-ny', backend_script] - -# Uncomment this line to daemonize -#sys.argv[1:] = ['-y', backend_script] - -def runApp(config): - runner.OBaseRunner(config).run() - -config = runner.ServerOptions() -try: - config.parseOptions() -except usage.error, ue: - print config - print "%s: %s" % (sys.argv[0], ue) -else: - runApp(config) - diff --git a/oonib/INSTALL b/oonib/INSTALL deleted file mode 100644 index 1622707..0000000 --- a/oonib/INSTALL +++ /dev/null @@ -1,4 +0,0 @@ -BEWARE: This requires python 2.7.3 -storm (Storm ORM) -transaction (zope transaction) - diff --git a/oonib/README.md b/oonib/README.md deleted file mode 100644 index d11f876..0000000 --- a/oonib/README.md +++ /dev/null @@ -1,117 +0,0 @@ -# Dependencies - -The extra dependencies necessary to run OONIB are: - -* twisted-names -* cyclone: https://github.com/fiorix/cyclone - -We recommend that you use a python virtualenv. See OONI's README.md. - -# Generate self signed certs for OONIB - -If you want to use the HTTPS test helper, you will need to create a certificate: - - openssl genrsa -des3 -out private.key 4096 - openssl req -new -key private.key -out server.csr - cp private.key private.key.org - # Remove passphrase from key - openssl rsa -in private.key.org -out private.key - openssl x509 -req -days 365 -in server.csr -signkey private.key -out certificate.crt - rm private.key.org - -Don't forget to update oonib/config.py options helpers.ssl.private_key and -helpers.ssl.certificate - -# Redirect low ports with iptables - -The following iptables commands will map connections on low ports to those bound by oonib - - # Map port 80 to config.helpers.http_return_request.port (default: 57001) - iptables -t nat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 57001 - # Map port 443 to config.helpers.ssl.port (default: 57006) - iptables -t nat -A PREROUTING -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 57006 - # Map port 53 udp to config.helpers.dns.udp_port (default: 57004) - iptables -t nat -A PREROUTING -p tcp -m udp --dport 53 -j REDIRECT --tor-ports - # Map port 53 tcp to config.helpers.dns.tcp_port (default: 57005) - iptables -t nat -A PREROUTING -p tcp -m tcp --dport 53 -j REDIRECT --tor-ports - -# Install Tor (Debian). - -You will need a Tor binary on your system. For complete instructions, see also: - - https://www.torproject.org/docs/tor-doc-unix.html.en - https://www.torproject.org/docs/rpms.html.en - -Add this line to your /etc/apt/sources.list, replacing <DISTRIBUTION> -where appropriate: - - deb http://deb.torproject.org/torproject.org <DISTRIBUTION> main - -Add the Tor Project gpg key to apt: - - gpg --keyserver keys.gnupg.net --recv 886DDD89 - gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 | sudo apt-key add - - # Update apt and install the torproject keyring, tor, and geoipdb - apt-get update - apt-get install deb.torproject.org-keyring tor tor-geoipdb - -# Update ooni-probe/oonib/config.py - - Set config.main.tor_binary to your Tor path - Set config.main.tor2webmode = False - -# (For Experts Only) Tor2webmode: - -WARNING: provides no anonymity! Use only if you know what you are doing! -Tor2webmode will improve the performance of the collector Hidden Service -by discarding server-side anonymity. - -You will need to build Tor from source. At the time of writing, the latest stable Tor is tor-0.2.3.25. You should use the most recent stable Tor. - -Example: - - git clone https://git.torproject.org/tor.git - git checkout tor-0.2.3.25 - git verify-tag -v tor-0.2.3.25 - -You should see: - - object 17c24b3118224d6536c41fa4e1493a831fb29f0a - type commit - tag tor-0.2.3.25 - tagger Roger Dingledine <arma(a)torproject.org> 1353399116 -0500 - - tag 0.2.3.25 - gpg: Signature made Tue 20 Nov 2012 08:11:59 AM UTC using RSA key ID 19F78451 - gpg: Good signature from "Roger Dingledine <arma(a)mit.edu>" - gpg: aka "Roger Dingledine <arma(a)freehaven.net>" - gpg: aka "Roger Dingledine <arma(a)torproject.org>" - -It is always good idea to verify. - - gpg --fingerprint 19F78451 - pub 4096R/19F78451 2010-05-07 - Key fingerprint = F65C E37F 04BA 5B36 0AE6 EE17 C218 5258 19F7 8451 - uid Roger Dingledine <arma(a)mit.edu> - uid Roger Dingledine <arma(a)freehaven.net> - uid Roger Dingledine <arma(a)torproject.org> - sub 4096R/9B11185C 2012-05-02 [expires: 2013-05-02] - -Build Tor with enable-tor2web-mode - - ./autogen.sh ; ./configure --enable-tor2web-mode ; make - -Copy the tor binary from src/or/tor somewhere and set the corresponding -options in oonib/config.py - -# To launch oonib on system boot - -To launch oonib on startup, you may want to use supervisord (www.supervisord.org) -The following supervisord config will use the virtual environment in -/home/ooni/venv_oonib and start oonib on boot: - - [program:oonib] - command=/home/ooni/venv_oonib/bin/python /home/ooni/ooni-probe/bin/oonib - autostart=true - user=oonib - directory=/home/oonib/ diff --git a/oonib/__init__.py b/oonib/__init__.py deleted file mode 100644 index ab7419c..0000000 --- a/oonib/__init__.py +++ /dev/null @@ -1,24 +0,0 @@ -# -*- encoding: utf-8 -*- -# -# :authors: Arturo Filastò -# :licence: see LICENSE -""" -In here we shall keep track of all variables and objects that should be -instantiated only once and be common to pieces of GLBackend code. -""" -__all__ = ['database', 'db_threadpool'] - -from twisted.python.threadpool import ThreadPool - -from storm.uri import URI -from storm.twisted.transact import Transactor -from storm.databases.sqlite import SQLite - -__version__ = '0.0.1' - -from oonib import config - -database = SQLite(URI(config.main.database_uri)) -db_threadpool = ThreadPool(0, config.main.db_threadpool_size) -db_threadpool.start() -transactor = Transactor(db_threadpool) diff --git a/oonib/config.py b/oonib/config.py deleted file mode 100644 index 8598e00..0000000 --- a/oonib/config.py +++ /dev/null @@ -1,58 +0,0 @@ -from ooni.utils import Storage -import os - -def get_root_path(): - this_directory = os.path.dirname(__file__) - root = os.path.join(this_directory, '..') - root = os.path.abspath(root) - return root - -backend_version = '0.0.1' - -# XXX convert this to something that is a proper config file -main = Storage() - -# This is the location where submitted reports get stored -main.report_dir = os.path.join(get_root_path(), 'oonib', 'reports') - -# This is where tor will place it's Hidden Service hostname and Hidden service -# private key -main.tor_datadir = os.path.join(get_root_path(), 'oonib', 'data', 'tor') - -main.database_uri = "sqlite:"+get_root_path()+"oonib_test_db.db" -main.db_threadpool_size = 10 -#main.tor_binary = '/usr/sbin/tor' -main.tor_binary = '/usr/local/bin/tor' - -# This requires compiling Tor with tor2web mode enabled -# BEWARE!! THIS PROVIDES NO ANONYMITY!! -# ONLY DO IT IF YOU KNOW WHAT YOU ARE DOING!! -# HOSTING A COLLECTOR WITH TOR2WEB MODE GIVES YOU NO ANONYMITY!! -main.tor2webmode = True - -helpers = Storage() - -helpers.http_return_request = Storage() -helpers.http_return_request.port = 57001 -# XXX this actually needs to be the advertised Server HTTP header of our web -# server -helpers.http_return_request.server_version = "Apache" - -helpers.tcp_echo = Storage() -helpers.tcp_echo.port = 57002 - -helpers.daphn3 = Storage() -#helpers.daphn3.yaml_file = "/path/to/data/oonib/daphn3.yaml" -#helpers.daphn3.pcap_file = "/path/to/data/server.pcap" -helpers.daphn3.port = 57003 - -helpers.dns = Storage() -helpers.dns.udp_port = 57004 -helpers.dns.tcp_port = 57005 - -helpers.ssl = Storage() -#helpers.ssl.private_key = /path/to/data/private.key -#helpers.ssl.certificate = /path/to/data/certificate.crt -#helpers.ssl.port = 57006 - - diff --git a/oonib/db/__init__.py b/oonib/db/__init__.py deleted file mode 100644 index 8ddaff0..0000000 --- a/oonib/db/__init__.py +++ /dev/null @@ -1,30 +0,0 @@ -__all__ = ['createTables', 'database', 'transactor'] - -from twisted.python.threadpool import ThreadPool -from twisted.internet.defer import inlineCallbacks, returnValue, Deferred - -from storm.locals import Store -from storm.uri import URI -from storm.databases.sqlite import SQLite - -from oonib import database, transactor -from ooni.utils import log - -@inlineCallbacks -def createTables(): - """ - XXX this is to be refactored and only exists for experimentation. - """ - from oonib.db import models - for model_name in models.__all__: - try: - model = getattr(m, model_name) - except Exception, e: - log.err("Error in db initting") - log.err(e) - try: - log.debug("Creating %s" % model) - yield tables.runCreateTable(model, transactor, database) - except Exception, e: - log.debug(str(e)) - diff --git a/oonib/db/tables.py b/oonib/db/tables.py deleted file mode 100644 index 908a295..0000000 --- a/oonib/db/tables.py +++ /dev/null @@ -1,123 +0,0 @@ -# -*- encoding: utf-8 -*- -# -# :authors: Arturo Filastò -# :licence: see LICENSE - -from twisted.internet.defer import inlineCallbacks - -from storm.locals import Store -from storm.properties import PropertyColumn -from storm.exceptions import StormError - -from storm.variables import BoolVariable, DateTimeVariable, DateVariable -from storm.variables import DecimalVariable, EnumVariable -from storm.variables import FloatVariable, IntVariable, RawStrVariable -from storm.variables import UnicodeVariable, JSONVariable, PickleVariable - -def variableToSQLite(var_type): - """ - We take as input a storm.variable and we output the SQLite string it - represents. - """ - sqlite_type = "VARCHAR" - if isinstance(var_type, BoolVariable): - sqlite_type = "INTEGER" - elif isinstance(var_type, DateTimeVariable): - pass - sqlite_type = "" - elif isinstance(var_type, DateVariable): - pass - elif isinstance(var_type, DecimalVariable): - pass - elif isinstance(var_type, EnumVariable): - sqlite_type = "BLOB" - elif isinstance(var_type, FloatVariable): - sqlite_type = "REAL" - elif isinstance(var_type, IntVariable): - sqlite_type = "INTEGER" - elif isinstance(var_type, RawStrVariable): - sqlite_type = "BLOB" - elif isinstance(var_type, UnicodeVariable): - pass - elif isinstance(var_type, JSONVariable): - sqlite_type = "BLOB" - elif isinstance(var_type, PickleVariable): - sqlite_type = "BLOB" - return "%s" % sqlite_type - -def varsToParametersSQLite(variables, primary_keys): - """ - Takes as input a list of variables (convered to SQLite syntax and in the - form of strings) and primary_keys. - Outputs these variables converted into paramter syntax for SQLites. - - ex. - variables: ["var1 INTEGER", "var2 BOOL", "var3 INTEGER"] - primary_keys: ["var1"] - - output: "(var1 INTEGER, var2 BOOL, var3 INTEGER PRIMARY KEY (var1))" - """ - params = "(" - for var in variables[:-1]: - params += "%s %s, " % var - if len(primary_keys) > 0: - params += "%s %s, " % variables[-1] - params += "PRIMARY KEY (" - for key in primary_keys[:-1]: - params += "%s, " % key - params += "%s))" % primary_keys[-1] - else: - params += "%s %s)" % variables[-1] - return params - -def generateCreateQuery(model): - """ - This takes as input a Storm model and outputs the creation query for it. - """ - query = "CREATE TABLE "+ model.__storm_table__ + " " - - variables = [] - primary_keys = [] - - for attr in dir(model): - a = getattr(model, attr) - if isinstance(a, PropertyColumn): - var_stype = a.variable_factory() - var_type = variableToSQLite(var_stype) - name = a.name - variables.append((name, var_type)) - if a.primary: - primary_keys.append(name) - - query += varsToParametersSQLite(variables, primary_keys) - return query - -def createTable(model, transactor, database): - """ - Create the table for the specified model. - Specification of a transactor and database is useful in unittesting. - """ - if not transactor: - from oonib.db import transactor - if not database: - from oonib.db import database - store = Store(database) - create_query = generateCreateQuery(model) - try: - store.execute(create_query) - # XXX trap the specific error that is raised when the table exists - except StormError, e: - print "Failed to create table!" - print e - store.close() - store.commit() - store.close() - -@inlineCallbacks -def runCreateTable(model, transactor=None, database=None): - """ - Runs the table creation query wrapped in a transaction. - Transactions run in a separate thread. - """ - yield transactor.run(createTable, model, transactor, database) - diff --git a/oonib/models.py b/oonib/models.py deleted file mode 100644 index 22567ad..0000000 --- a/oonib/models.py +++ /dev/null @@ -1,129 +0,0 @@ -__all__ = ['Report', 'TestHelperTMP'] -from storm.twisted.transact import transact -from storm.locals import * - -from ooni.utils import randomStr -from oonib import transactor - -def generateReportID(): - """ - Generates a report ID for usage in the database backed oonib collector. - - XXX note how this function is different from the one in report/api.py - """ - report_id = randomStr(100) - return report_id - -class OModel(object): - - transactor = transactor - - def getStore(self): - return Store(database) - - @transact - def create(self, query): - store = Store(database) - store.execute(query) - store.commit() - - @transact - def save(self): - store = getStore() - store.add(self) - store.commit() - -class Report(OModel): - """ - This represents an OONI Report as stored in the database. - - report_id: this is generated by the backend and is used by the client to - reference a previous report and append to it. It should be - treated as a shared secret between the probe and backend. - - software_name: this indicates the name of the software performing the test - (this will default to ooniprobe) - - software_version: this is the version number of the software running the - test. - - test_name: the name of the test on which the report is being created. - - test_version: indicates the version of the test - - progress: what is the current progress of the report. This allows clients - to report event partial reports up to a certain percentage of - progress. Once the report is complete progress will be 100. - - content: what is the content of the report. If the current progress is less - than 100 we should append to the YAML data structure that is - currently stored in such field. - - XXX this is currently not used. - """ - __storm_table__ = 'reports' - - createQuery = "CREATE TABLE " + __storm_table__ +\ - "(id INTEGER PRIMARY KEY, report_id VARCHAR, software_name VARCHAR,"\ - "software_version VARCHAR, test_name VARCHAR, test_version VARCHAR,"\ - "progress VARCHAR, content VARCHAR)" - - - id = Int(primary=True) - - report_id = Unicode() - - software_name = Unicode() - software_version = Unicode() - test_name = Unicode() - test_version = Unicode() - progress = Int() - - content = Unicode() - - @transact - def new(report): - store = self.getStore() - - print "Creating new report %s" % report - - report_id = generateReportID() - new_report = models.Report() - new_report.report_id = unicode(report_id) - - new_report.software_name = report['software_name'] - new_report.software_version = report['software_version'] - new_report.test_name = report['test_name'] - new_report.test_version = report['test_version'] - new_report.progress = report['progress'] - - if 'content' in report: - new_report.content = report['content'] - - print "Report: %s" % report - - store.add(new_report) - try: - store.commit() - except: - store.close() - - defer.returnValue({'backend_version': backend_version, 'report_id': - report_id}) - - -class TestHelperTMP(OModel): - __storm_table__ = 'testhelpertmp' - - createQuery = "CREATE TABLE " + __storm_table__ +\ - "(id INTEGER PRIMARY KEY, report_id VARCHAR, test_helper VARCHAR,"\ - " client_ip VARCHAR, creation_time VARCHAR)" - - id = Int(primary=True) - - report_id = Unicode() - - test_helper = Unicode() - client_ip = Unicode() - - creation_time = Date() diff --git a/oonib/oonibackend.py b/oonib/oonibackend.py deleted file mode 100644 index 868d7b7..0000000 --- a/oonib/oonibackend.py +++ /dev/null @@ -1,74 +0,0 @@ -# ooni backend -# ************ -# -# This is the backend system responsible for running certain services that are -# useful for censorship detection. -# -# In here we start all the test helpers that are required by ooniprobe and -# start the report collector - -from twisted.application import internet -from twisted.internet import reactor -from twisted.application import internet, service -from twisted.application.service import Application -from twisted.names import dns - -from cyclone import web - -import txtorcon - -from oonib.testhelpers import dns_helpers, ssl_helpers -from oonib.testhelpers import http_helpers, tcp_helpers - -from ooni.utils import log - -from oonib import db_threadpool -from oonib import config - -application = service.Application('oonibackend') -serviceCollection = service.IServiceCollection(application) - -if config.helpers.ssl.port: - print "Starting SSL helper on %s" % config.helpers.ssl.port - ssl_helper = internet.SSLServer(int(config.helpers.ssl.port), - http_helpers.HTTPReturnJSONHeadersHelper(), - ssl_helpers.SSLContext(config)) - ssl_helper.setServiceParent(serviceCollection) - -# Start the DNS Server related services -if config.helpers.dns.tcp_port: - print "Starting TCP DNS Helper on %s" % config.helpers.dns.tcp_port - tcp_dns_helper = internet.TCPServer(int(config.helpers.dns.tcp_port), - dns_helpers.DNSTestHelper()) - tcp_dns_helper.setServiceParent(serviceCollection) - -if config.helpers.dns.udp_port: - print "Starting UDP DNS Helper on %s" % config.helpers.dns.udp_port - udp_dns_factory = dns.DNSDatagramProtocol(dns_helpers.DNSTestHelper()) - udp_dns_helper = internet.UDPServer(int(config.helpers.dns.udp_port), - udp_dns_factory) - udp_dns_helper.setServiceParent(serviceCollection) - -# XXX this needs to be ported -# Start the OONI daphn3 backend -if config.helpers.daphn3.port: - print "Starting Daphn3 helper on %s" % config.helpers.daphn3.port - daphn3_helper = internet.TCPServer(int(config.helpers.daphn3.port), - tcp_helpers.Daphn3Server()) - daphn3_helper.setServiceParent(serviceCollection) - - -if config.helpers.tcp_echo.port: - print "Starting TCP echo helper on %s" % config.helpers.tcp_echo.port - tcp_echo_helper = internet.TCPServer(int(config.helpers.tcp_echo.port), - tcp_helpers.TCPEchoHelper()) - tcp_echo_helper.setServiceParent(serviceCollection) - -if config.helpers.http_return_request.port: - print "Starting HTTP return request helper on %s" % config.helpers.http_return_request.port - http_return_request_helper = internet.TCPServer( - int(config.helpers.http_return_request.port), - http_helpers.HTTPReturnJSONHeadersHelper()) - http_return_request_helper.setServiceParent(serviceCollection) - -reactor.addSystemEventTrigger('after', 'shutdown', db_threadpool.stop) diff --git a/oonib/report/__init__.py b/oonib/report/__init__.py deleted file mode 100644 index fcbf220..0000000 --- a/oonib/report/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -class MissingField(Exception): - pass - -class InvalidRequestField(Exception): - pass diff --git a/oonib/report/api.py b/oonib/report/api.py deleted file mode 100644 index b3d529d..0000000 --- a/oonib/report/api.py +++ /dev/null @@ -1,106 +0,0 @@ -""" -/report - -/pcap - -This is the async pcap reporting system. It requires the client to have created -a report already, but can work independently from test progress. - -""" -import random -import string -import json -import re -import os - -from twisted.internet import reactor, defer - -from cyclone import web - -from ooni import otime -from ooni.utils import randomStr - -from oonib import models, config -from oonib.report import file_collector - -def parseUpdateReportRequest(request): - #db_report_id_regexp = re.compile("[a-zA-Z0-9]+$") - - # this is the regexp for the reports that include the timestamp - report_id_regexp = re.compile("[a-zA-Z0-9_-]+$") - - # XXX here we are actually parsing a json object that could be quite big. - # If we want this to scale properly we only want to look at the test_id - # field. - # We are also keeping in memory multiple copies of the same object. A lot - # of optimization can be done. - parsed_request = json.loads(request) - try: - report_id = parsed_request['report_id'] - except KeyError: - raise MissingField('report_id') - - if not re.match(report_id_regexp, report_id): - raise InvalidRequestField('report_id') - - return parsed_request - - -class NewReportHandlerDB(web.RequestHandler): - """ - Responsible for creating and updating reports via database. - XXX this is not yet fully implemented. - """ - - @web.asynchronous - @defer.inlineCallbacks - def post(self): - """ - Creates a new report with the input to the database. - XXX this is not yet implemented. - - * Request - - {'software_name': 'XXX', - 'software_version': 'XXX', - 'test_name': 'XXX', - 'test_version': 'XXX', - 'progress': 'XXX', - 'content': 'XXX' - } - - Optional: - 'test_helper': 'XXX' - 'client_ip': 'XXX' - - * Response - - {'backend_version': 'XXX', 'report_id': 'XXX'} - - """ - report_data = json.loads(self.request.body) - new_report = models.Report() - log.debug("Got this request %s" % report_data) - result = yield new_report.new(report_data) - self.write(result) - self.finish() - - def put(self): - """ - Update an already existing report with the database. - - XXX this is not yet implemented. - - {'report_id': 'XXX', - 'content': 'XXX' - } - """ - pass - - -reportingBackendAPI = [ - (r"/report", file_collector.NewReportHandlerFile), - (r"/pcap", file_collector.PCAPReportHandler) -] - -reportingBackend = web.Application(reportingBackendAPI, debug=True) diff --git a/oonib/report/file_collector.py b/oonib/report/file_collector.py deleted file mode 100644 index 6d5584c..0000000 --- a/oonib/report/file_collector.py +++ /dev/null @@ -1,193 +0,0 @@ -import random -import string -import json -import re -import os - -from twisted.internet import fdesc - -from cyclone import web - -from ooni.utils import randomStr -from ooni import otime - -from oonib.report import MissingField, InvalidRequestField - -from oonib import config - -def parseUpdateReportRequest(request): - #db_report_id_regexp = re.compile("[a-zA-Z0-9]+$") - - # this is the regexp for the reports that include the timestamp - report_id_regexp = re.compile("[a-zA-Z0-9_\-]+$") - - # XXX here we are actually parsing a json object that could be quite big. - # If we want this to scale properly we only want to look at the test_id - # field. - # We are also keeping in memory multiple copies of the same object. A lot - # of optimization can be done. - parsed_request = json.loads(request) - try: - report_id = parsed_request['report_id'] - except KeyError: - raise MissingField('report_id') - - if not re.match(report_id_regexp, report_id): - raise InvalidRequestField('report_id') - - return parsed_request - - - -def parseNewReportRequest(request): - """ - Here we parse a new report request. - """ - version_string = re.compile("[0-9A-Za-z_\-\.]+$") - name = re.compile("[a-zA-Z0-9_\- ]+$") - probe_asn = re.compile("AS[0-9]+$") - - expected_request = { - 'software_name': name, - 'software_version': version_string, - 'test_name': name, - 'test_version': version_string, - 'probe_asn': probe_asn - } - - parsed_request = json.loads(request) - if not parsed_request['probe_asn']: - parsed_request['probe_asn'] = 'AS0' - - for k, regexp in expected_request.items(): - try: - value_to_check = parsed_request[k] - except KeyError: - raise MissingField(k) - - print "Matching %s with %s | %s" % (regexp, value_to_check, k) - if re.match(regexp, str(value_to_check)): - continue - else: - raise InvalidRequestField(k) - - return parsed_request - -class NewReportHandlerFile(web.RequestHandler): - """ - Responsible for creating and updating reports by writing to flat file. - """ - - def post(self): - """ - Creates a new report with the input - - * Request - - {'software_name': 'XXX', - 'software_version': 'XXX', - 'test_name': 'XXX', - 'test_version': 'XXX', - 'probe_asn': 'XXX' - 'content': 'XXX' - } - - Optional: - 'test_helper': 'XXX' - 'client_ip': 'XXX' - - (not implemented, nor in client, nor in backend) - The idea behind these two fields is that it would be interesting to - also collect how the request was observed from the collectors point - of view. - - We use as a unique key the client_ip address and a time window. We - then need to tell the test_helper that is selected the client_ip - address and tell it to expect a connection from a probe in that time - window. - - Once the test_helper sees a connection from that client_ip it will - store for the testing session the data that it receives. - When the probe completes the report (or the time window is over) the - final report will include also the data collected from the - collectors view point. - - * Response - - {'backend_version': 'XXX', 'report_id': 'XXX'} - - """ - # XXX here we should validate and sanitize the request - try: - report_data = parseNewReportRequest(self.request.body) - except InvalidRequestField, e: - raise web.HTTPError(400, "Invalid Request Field %s" % e) - except MissingField, e: - raise web.HTTPError(400, "Missing Request Field %s" % e) - - print "Parsed this data %s" % report_data - software_name = report_data['software_name'] - software_version = report_data['software_version'] - test_name = report_data['test_name'] - test_version = report_data['test_version'] - probe_asn = report_data['probe_asn'] - content = report_data['content'] - - if not probe_asn: - probe_asn = "AS0" - - report_id = otime.timestamp() + '_' \ - + probe_asn + '_' \ - + randomStr(50) - - # The report filename contains the timestamp of the report plus a - # random nonce - report_filename = os.path.join(config.main.report_dir, report_id) - - response = {'backend_version': config.backend_version, - 'report_id': report_id - } - - self.writeToReport(report_filename, - report_data['content']) - - self.write(response) - - def writeToReport(self, report_filename, data): - with open(report_filename, 'w+') as fd: - fdesc.setNonBlocking(fd.fileno()) - fdesc.writeToFD(fd.fileno(), data) - - def put(self): - """ - Update an already existing report. - - { - 'report_id': 'XXX', - 'content': 'XXX' - } - """ - parsed_request = parseUpdateReportRequest(self.request.body) - - report_id = parsed_request['report_id'] - - print "Got this request %s" % parsed_request - report_filename = os.path.join(config.main.report_dir, - report_id) - - self.updateReport(report_filename, parsed_request['content']) - - def updateReport(self, report_filename, data): - try: - with open(report_filename, 'a+') as fd: - fdesc.setNonBlocking(fd.fileno()) - fdesc.writeToFD(fd.fileno(), data) - except IOError as e: - web.HTTPError(404, "Report not found") - -class PCAPReportHandler(web.RequestHandler): - def get(self): - pass - - def post(self): - pass diff --git a/oonib/requirements.txt b/oonib/requirements.txt deleted file mode 100644 index da6ec81..0000000 --- a/oonib/requirements.txt +++ /dev/null @@ -1,24 +0,0 @@ -PyYAML>=3.10 -Pygments>=1.5 -Twisted>=12.2.0 -argparse>=1.2.1 -cyclone>=1.0-rc13 -distribute>=0.6.24 -docutils>=0.9.1 -ipaddr>=2.1.10 -pyOpenSSL>=0.13 -pygeoip>=0.2.5 -# -# This is a Tor Project mirror with valid SSL/TLS certs that is stable and fast -# -# Originally fetched from the hg repo on secdev.org: -# https://hg.secdev.org/scapy/archive/tip.zip#egg=scapy -# Mirrored on Tor's webserver: -https://people.torproject.org/~ioerror/src/mirrors/ooniprobe/scapy-02-25-2013-tip.zip -storm>=0.19 -transaction>=1.3.0 -txtorcon -wsgiref>=0.1.2 -zope.component>=4.0.0 -zope.event>=4.0.0 -zope.interface>=4.0.1 diff --git a/oonib/runner.py b/oonib/runner.py deleted file mode 100644 index 407f10d..0000000 --- a/oonib/runner.py +++ /dev/null @@ -1,81 +0,0 @@ -""" -In here we define a runner for the oonib backend system. -We are just extending the - -""" - -from twisted.internet import reactor -from twisted.application import service, internet, app -from twisted.python.runtime import platformType - -import txtorcon - -from oonib.report.api import reportingBackend - -from oonib import config -from ooni.utils import log - -def txSetupFailed(failure): - log.err("Setup failed") - log.exception(failure) - -def setupCollector(tor_process_protocol): - def setup_complete(port): - print "Exposed collector Tor hidden service on httpo://%s" % port.onion_uri - - torconfig = txtorcon.TorConfig(tor_process_protocol.tor_protocol) - public_port = 80 - # XXX there is currently a bug in txtorcon that prevents data_dir from - # being passed properly. Details on the bug can be found here: - # https://github.com/meejah/txtorcon/pull/22 - hs_endpoint = txtorcon.TCPHiddenServiceEndpoint(reactor, torconfig, - public_port, data_dir=config.main.tor_datadir) - hidden_service = hs_endpoint.listen(reportingBackend) - hidden_service.addCallback(setup_complete) - hidden_service.addErrback(txSetupFailed) - -def startTor(): - def updates(prog, tag, summary): - print "%d%%: %s" % (prog, summary) - - torconfig = txtorcon.TorConfig() - torconfig.SocksPort = 9055 - if config.main.tor2webmode: - torconfig.Tor2webMode = 1 - torconfig.save() - d = txtorcon.launch_tor(torconfig, reactor, - tor_binary=config.main.tor_binary, - progress_updates=updates) - d.addCallback(setupCollector) - d.addErrback(txSetupFailed) - -class OBaseRunner(): - pass - -if platformType == "win32": - from twisted.scripts._twistw import ServerOptions, \ - WindowsApplicationRunner - - OBaseRunner = WindowsApplicationRunner - # XXX Current we don't support windows for the starting of Tor Hidden Service - -else: - from twisted.scripts._twistd_unix import ServerOptions, \ - UnixApplicationRunner - class OBaseRunner(UnixApplicationRunner): - def postApplication(self): - """ - To be called after the application is created: start the - application and run the reactor. After the reactor stops, - clean up PID files and such. - """ - self.startApplication(self.application) - # This is our addition. The rest is taken from - # twisted/scripts/_twistd_unix.py 12.2.0 - startTor() - self.startReactor(None, self.oldstdout, self.oldstderr) - self.removePID(self.config['pidfile']) - -OBaseRunner.loggerFactory = log.LoggerFactory - - diff --git a/oonib/testhelpers/__init__.py b/oonib/testhelpers/__init__.py deleted file mode 100644 index 4dbb547..0000000 --- a/oonib/testhelpers/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -from . import dns_helpers -from . import http_helpers -from . import tcp_helpers - -__all__ = ['dns_helpers', 'http_helpers', 'tcp_helpers'] diff --git a/oonib/testhelpers/dns_helpers.py b/oonib/testhelpers/dns_helpers.py deleted file mode 100644 index cb4ff9f..0000000 --- a/oonib/testhelpers/dns_helpers.py +++ /dev/null @@ -1,16 +0,0 @@ -from twisted.internet.protocol import Factory, Protocol -from twisted.internet import reactor -from twisted.names import dns -from twisted.names import client, server - -class DNSTestHelper(server.DNSServerFactory): - def __init__(self, authorities = None, - caches = None, clients = None, - verbose = 0): - resolver = client.Resolver(servers=[('8.8.8.8', 53)]) - server.DNSServerFactory.__init__(self, authorities = authorities, - caches = caches, clients = [resolver], - verbose = verbose) - def handleQuery(self, message, protocol, address): - print message, protocol, address - server.DNSServerFactory.handleQuery(self, message, protocol, address) diff --git a/oonib/testhelpers/http_helpers.py b/oonib/testhelpers/http_helpers.py deleted file mode 100644 index 3a76b9a..0000000 --- a/oonib/testhelpers/http_helpers.py +++ /dev/null @@ -1,154 +0,0 @@ -import json -import random -import string - -from twisted.application import internet, service -from twisted.internet import protocol, reactor, defer -from twisted.protocols import basic -from twisted.web import resource, server, static, http -from twisted.web.microdom import escape - -from cyclone.web import RequestHandler, Application - -from twisted.protocols import policies, basic -from twisted.web.http import Request - -class SimpleHTTPChannel(basic.LineReceiver, policies.TimeoutMixin): - """ - This is a simplified version of twisted.web.http.HTTPChannel to overcome - header lowercase normalization. It does not actually implement the HTTP - protocol, but only the subset of it that we need for testing. - - What this HTTP channel currently does is process the HTTP Request Line and - the Request Headers and returns them in a JSON datastructure in the order - we received them. - - The returned JSON dict looks like so: - - { - 'request_headers': - [['User-Agent', 'IE6'], ['Content-Length', 200]] - 'request_line': - 'GET / HTTP/1.1' - } - """ - requestFactory = Request - __first_line = 1 - __header = '' - __content = None - - length = 0 - maxHeaders = 500 - requestLine = '' - headers = [] - - timeOut = 60 * 60 * 12 - - def __init__(self): - self.requests = [] - - def connectionMade(self): - self.setTimeout(self.timeOut) - - def lineReceived(self, line): - if self.__first_line: - self.requestLine = line - self.__first_line = 0 - elif line == '': - # We have reached the end of the headers. - if self.__header: - self.headerReceived(self.__header) - self.__header = '' - self.allHeadersReceived() - self.setRawMode() - elif line[0] in ' \t': - # This is to support header field value folding over multiple lines - # as specified by rfc2616. - self.__header = self.__header+'\n'+line - else: - if self.__header: - self.headerReceived(self.__header) - self.__header = line - - def headerReceived(self, line): - try: - header, data = line.split(':', 1) - self.headers.append((header, data.strip())) - except: - log.err("Got malformed HTTP Header request field") - log.err("%s" % line) - - def allHeadersReceived(self): - headers_dict = {} - for k, v in self.headers: - if k not in headers_dict: - headers_dict[k] = [] - headers_dict[k].append(v) - - response = {'request_headers': self.headers, - 'request_line': self.requestLine, - 'headers_dict': headers_dict - } - json_response = json.dumps(response) - self.transport.write('HTTP/1.1 200 OK\r\n\r\n') - self.transport.write('%s' % json_response) - self.transport.loseConnection() - - -class HTTPReturnJSONHeadersHelper(protocol.ServerFactory): - protocol = SimpleHTTPChannel - def buildProtocol(self, addr): - p = self.protocol() - p.headers = [] - return p - -class HTTPTrapAll(RequestHandler): - def _execute(self, transforms, *args, **kwargs): - self._transforms = transforms - defer.maybeDeferred(self.prepare).addCallbacks( - self._execute_handler, - lambda f: self._handle_request_exception(f.value), - callbackArgs=(args, kwargs)) - - def _execute_handler(self, r, args, kwargs): - if not self._finished: - args = [self.decode_argument(arg) for arg in args] - kwargs = dict((k, self.decode_argument(v, name=k)) - for (k, v) in kwargs.iteritems()) - # This is where we do the patching - # XXX this is somewhat hackish - d = defer.maybeDeferred(self.all, *args, **kwargs) - d.addCallbacks(self._execute_success, self._execute_failure) - self.notifyFinish().addCallback(self.on_connection_close) - - -class HTTPRandomPage(HTTPTrapAll): - """ - This generates a random page of arbitrary length and containing the string - selected by the user. - /<length>/<keyword> - XXX this is currently disabled as it is not of use to any test. - """ - isLeaf = True - def _gen_random_string(self, length): - return ''.join(random.choice(string.letters) for x in range(length)) - - def genRandomPage(self, length=100, keyword=None): - data = self._gen_random_string(length/2) - if keyword: - data += keyword - data += self._gen_random_string(length - length/2) - data += '\n' - return data - - def all(self, length, keyword): - length = 100 - if length > 100000: - length = 100000 - return self.genRandomPage(length, keyword) - -HTTPRandomPageHelper = Application([ - # XXX add regexps here - (r"/(.*)/(.*)", HTTPRandomPage) -]) - diff --git a/oonib/testhelpers/ssl_helpers.py b/oonib/testhelpers/ssl_helpers.py deleted file mode 100644 index 5c74996..0000000 --- a/oonib/testhelpers/ssl_helpers.py +++ /dev/null @@ -1,9 +0,0 @@ -from twisted.internet import ssl -from oonib import config - -class SSLContext(ssl.DefaultOpenSSLContextFactory): - def __init__(self, *args, **kw): - ssl.DefaultOpenSSLContextFactory.__init__(self, - config.helpers.ssl.private_key, - config.helpers.ssl.certificate) - diff --git a/oonib/testhelpers/tcp_helpers.py b/oonib/testhelpers/tcp_helpers.py deleted file mode 100644 index 4d32ae0..0000000 --- a/oonib/testhelpers/tcp_helpers.py +++ /dev/null @@ -1,72 +0,0 @@ - -from twisted.internet.protocol import Protocol, Factory, ServerFactory -from twisted.internet.error import ConnectionDone - -from oonib import config -from ooni.utils import log -from ooni.kit.daphn3 import Daphn3Protocol -from ooni.kit.daphn3 import read_pcap, read_yaml - -class TCPEchoProtocol(Protocol): - def dataReceived(self, data): - self.transport.write(data) - -class TCPEchoHelper(Factory): - """ - A very simple echo protocol implementation - """ - protocol = TCPEchoProtocol - -if config.helpers.daphn3.yaml_file: - daphn3Steps = read_pcap(config.helpers.daphn3.yaml_file) - -elif config.helpers.daphn3.pcap_file: - daphn3Steps = read_yaml(config.helpers.daphn3.pcap_file) - -else: - daphn3Steps = [{'client': 'client_packet'}, - {'server': 'server_packet'}] - -class Daphn3ServerProtocol(Daphn3Protocol): - def nextStep(self): - log.debug("Moving on to next step in the state walk") - self.current_data_received = 0 - # Python why? - if self.current_step >= (len(self.steps) - 1): - log.msg("Reached the end of the state machine") - log.msg("Censorship fingerpint bisected!") - step_idx, mutation_idx = self.factory.mutation - log.msg("step_idx: %s | mutation_id: %s" % (step_idx, mutation_idx)) - #self.transport.loseConnection() - if self.report: - self.report['mutation_idx'] = mutation_idx - self.report['step_idx'] = step_idx - return - else: - self.current_step += 1 - if self._current_step_role() == self.role: - # We need to send more data because we are again responsible for - # doing so. - self.sendPayload() - -class Daphn3Server(ServerFactory): - """ - This is the main class that deals with the daphn3 server side component. - We keep track of global state of every client here. - Every client is identified by their IP address and the state of mutation is - stored by using their IP address as a key. This may lead to some bugs if - two different clients are sharing the same IP, but hopefully the - probability of such thing is not that likely. - """ - protocol = Daphn3ServerProtocol - # step_idx, mutation_idx - mutation = [0, 0] - def buildProtocol(self, addr): - p = self.protocol() - p.steps = daphn3Steps - p.role = "server" - p.factory = self - return p - - -

1 0