- tor-commits - lists.torproject.org

[torspec/master] tweak ipv6 plan more
by nickm＠torproject.org 02 Mar '11

02 Mar '11

commit 2fa1377bdc200cecd0c36ba4104f3da156ba1ea8 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 2 12:21:56 2011 -0500 tweak ipv6 plan more --- proposals/ideas/xxx-ipv6-plan.txt | 22 +++++++++++----------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/proposals/ideas/xxx-ipv6-plan.txt b/proposals/ideas/xxx-ipv6-plan.txt index 12bd4d4..73a21f1 100644 --- a/proposals/ideas/xxx-ipv6-plan.txt +++ b/proposals/ideas/xxx-ipv6-plan.txt @@ -18,17 +18,14 @@ Motivation: What needs to change: - Tor uses the Internet in many ways. There are four main ways that + Tor uses the Internet in many ways. There are three main ways that will need to change for IPv6 support, from most urgent to least urgent. - 0. An unknown laundry list of issues that will supersede all other - listed issues in this list. - 1. Tor must allow connections from IPv6-only clients. (Currently, routers and bridges do not listen on IPv6 addresses, and can't - advertise that they support IPv6 addresses, so clients can't learn that - they do.) + advertise that they support IPv6 addresses, so clients can't + learn that they do.) 2. Tor must transport IPv6 traffic and IPv6-related DNS traffic. (Currently, Tor only allows BEGIN cells to ask for connections @@ -38,11 +35,14 @@ What needs to change: 3. Tor must allow nodes to connect to one another over IPv6. Allowing IPv6-only clients is the most important, since unless we - do, these clients will be unable to connect to Tor at all. Next most - important is to support IPv6 DNS related dependencies and exiting to IPv6 - services. Finally, allowing Tor nodes to support a dual stack of both IPv4 - and IPv6 for interconnection seems like a reasonable step towards a fully - hybrid v4/v6 Tor network. + do, these clients will be unable to connect to Tor at all. Next + most important is to support IPv6 DNS related dependencies and + exiting to IPv6 services. Finally, allowing Tor nodes to support a + dual stack of both IPv4 and IPv6 for interconnection seems like a + reasonable step towards a fully hybrid v4/v6 Tor network. + + Of course, more issues may be discovered as we develop solutions for these + issues, some of which may need to take priority. Designs that we will need to do:

1 0

[torspec/master] ipv6-plan patch from ioerror
by nickm＠torproject.org 02 Mar '11

02 Mar '11

commit 2cb38a6ef43fd3b5de7508622a815d8810275e0b Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 2 11:29:41 2011 -0500 ipv6-plan patch from ioerror --- proposals/ideas/xxx-ipv6-plan.txt | 21 +++++++++++++++------ 1 files changed, 15 insertions(+), 6 deletions(-) diff --git a/proposals/ideas/xxx-ipv6-plan.txt b/proposals/ideas/xxx-ipv6-plan.txt index e9ffb10..12bd4d4 100644 --- a/proposals/ideas/xxx-ipv6-plan.txt +++ b/proposals/ideas/xxx-ipv6-plan.txt @@ -18,13 +18,16 @@ Motivation: What needs to change: - Tor uses the Internet in many ways. There four main ways that + Tor uses the Internet in many ways. There are four main ways that will need to change for IPv6 support, from most urgent to least urgent. + 0. An unknown laundry list of issues that will supersede all other + listed issues in this list. + 1. Tor must allow connections from IPv6-only clients. (Currently, - routers do not listen on IPv6 addresses, and can't advertise - that they support IPv6 addresses, so clients can't learn that + routers and bridges do not listen on IPv6 addresses, and can't + advertise that they support IPv6 addresses, so clients can't learn that they do.) 2. Tor must transport IPv6 traffic and IPv6-related DNS traffic. @@ -35,8 +38,11 @@ What needs to change: 3. Tor must allow nodes to connect to one another over IPv6. Allowing IPv6-only clients is the most important, since unless we - do, these unable to connect to Tor at all. Next most - important is to allow IPv6 XXXX + do, these clients will be unable to connect to Tor at all. Next most + important is to support IPv6 DNS related dependencies and exiting to IPv6 + services. Finally, allowing Tor nodes to support a dual stack of both IPv4 + and IPv6 for interconnection seems like a reasonable step towards a fully + hybrid v4/v6 Tor network. Designs that we will need to do: @@ -51,13 +57,16 @@ Designs that we will need to do: for places that might assume that IPs are a scarce resource. For example, clients assume that any two routers occupying an IPv4 /16 network are "too close" topologically to be used in the same - circuit, and the bridgedb https distributor assumes that hopping + circuit, and the bridgedb HTTPS distributor assumes that hopping from one /24 to another takes a little effort for most clients. The directory authorities assume that blacklisting an IP is an okay response to a bad router at that address. These and other places will needed instead more appropriate notions of "closeness" and "similarity". + We'll want to consider geographic and political boundaries rather than + purely mathematical notions such as the size of network blocks. + We'll need a way to advertise IPv6 bridges, and to use them. For transporting IPv6-only traffic, we have another accepted design

1 0

r24289: {website} updated translations for the website as wml files (in website/trunk: about/ru docs/en download/en projects/en)
by Runa Sandvik 02 Mar '11

02 Mar '11

Author: runa Date: 2011-03-02 17:16:57 +0000 (Wed, 02 Mar 2011) New Revision: 24289 Modified: website/trunk/about/ru/overview.wml website/trunk/docs/en/faq.wml website/trunk/download/en/download-easy.wml website/trunk/projects/en/torbrowser.wml website/trunk/projects/en/vidalia.wml Log: updated translations for the website as wml files Modified: website/trunk/about/ru/overview.wml =================================================================== --- website/trunk/about/ru/overview.wml 2011-03-02 16:43:14 UTC (rev 24288) +++ website/trunk/about/ru/overview.wml 2011-03-02 17:16:57 UTC (rev 24289) @@ -4,7 +4,7 @@ ## translation metadata -# Revision: $Revision: 23689 $ +# Revision: $Revision: 24254 $ # Translation-Priority: 2-medium #include "head.wmi" TITLE="Tor Project: Overview" CHARSET="UTF-8" <div id="content" class="clearfix"> @@ -154,8 +154,6 @@ определённо сказать, откуда данные пришли и куда они направляются. - <img alt="Первый шаг цепи Tor" src="$(IMGROOT)/htw1.png"> - Чтобы создать приватный путь с помощью Tor, программа пользователя или клиент последовательно строит цепочки зашифрованных соединений с серверами Modified: website/trunk/docs/en/faq.wml =================================================================== --- website/trunk/docs/en/faq.wml 2011-03-02 16:43:14 UTC (rev 24288) +++ website/trunk/docs/en/faq.wml 2011-03-02 17:16:57 UTC (rev 24289) @@ -1577,6 +1577,7 @@ problems are: +<ol> <li>IP packets reveal OS characteristics. We would still need to do IP-level packet normalization, to stop things like TCP fingerprinting attacks. Given the diversity and complexity of TCP stacks, along with <a Modified: website/trunk/download/en/download-easy.wml =================================================================== --- website/trunk/download/en/download-easy.wml 2011-03-02 16:43:14 UTC (rev 24288) +++ website/trunk/download/en/download-easy.wml 2011-03-02 17:16:57 UTC (rev 24289) @@ -19,7 +19,7 @@ <td>The Tor Browser Bundle contains everything you need to safely browse the Internet. This package requires no installation. Just extract it and run. <a href="<page projects/torbrowser>">Learn more -&amp other languages »</a></td> +& other languages »</a></td> </tr> <tr class="gray"> <td> @@ -28,6 +28,7 @@ (<a href="../dist/torbrowser/tor-browser-<version-torbrowserbundle>_en-US.exe.asc">sig</a>) </td> +</tr> <tr> <td> # Translators: please point to the version of TBB in your language, if there is one. Modified: website/trunk/projects/en/torbrowser.wml =================================================================== --- website/trunk/projects/en/torbrowser.wml 2011-03-02 16:43:14 UTC (rev 24288) +++ website/trunk/projects/en/torbrowser.wml 2011-03-02 17:16:57 UTC (rev 24289) @@ -240,7 +240,8 @@ Click Extract (3) to begin extraction. This may take a few minutes to complete. - <img src="$(IMGROOT)/tbb-screenshot1.png" alt="Screenshot of extraction process"> + <img src="$(IMGROOT)/tbb-screenshot1.png" alt="Screenshot of +extraction process" /> <a id="Usage"></a> <h3><a class="anchor" href="#Usage">Usage</a></h3> @@ -252,7 +253,8 @@ The Vidalia window will shortly appear. - <img src="$(IMGROOT)/tbb-screenshot2.png" alt="Screenshot of bundle startup"> + <img src="$(IMGROOT)/tbb-screenshot2.png" alt="Screenshot of +bundle startup" /> Once Tor is ready, Firefox will automatically be opened. Only web pages visited through the included Firefox browser will be sent via Tor. Other web @@ -264,15 +266,18 @@ If you installed the Tor IM Browser Bundle, the Pidgin instant messaging client will also be automatically opened. Once you are finished browsing, close any open Firefox windows by clicking - on the <img src="$(IMGROOT)/tbb-close-button.png" alt="Close button (×)"> (6). + on the <img src="$(IMGROOT)/tbb-close-button.png" alt="Close button +(×)" /> (6). For privacy reasons, the list of webpages you visited and any cookies will be deleted. - <img src="$(IMGROOT)/tbb-screenshot3a.png" alt="Screenshot of Firefox"> + <img src="$(IMGROOT)/tbb-screenshot3a.png" alt="Screenshot of +Firefox" /> With the Tor Browser Bundle, Vidalia and Tor will automatically close. With the Tor IM Browser Bundle you need to also close Pidgin by right-clicking on the Pidgin icon (7), and choosing Quit (8). - <img src="$(IMGROOT)/tbb-screenshot3b.png" alt="Screenshot of Pidgin"> + <img src="$(IMGROOT)/tbb-screenshot3b.png" alt="Screenshot of +Pidgin" /> To use the Tor Browser Bundle or Tor IM Browser Bundle again, repeat the steps in <a href="#Usage">"Usage"</a>. Modified: website/trunk/projects/en/vidalia.wml =================================================================== --- website/trunk/projects/en/vidalia.wml 2011-03-02 16:43:14 UTC (rev 24288) +++ website/trunk/projects/en/vidalia.wml 2011-03-02 17:16:57 UTC (rev 24289) @@ -17,7 +17,8 @@  <div class="sidebar-left"> - <img src="$(IMGROOT)/Screenshot-Vidalia-Control-Panel.png" width="261" height="255" alt="Vidalia Control Panel Screenshot"> + <img src="$(IMGROOT)/Screenshot-Vidalia-Control-Panel.png" +width="261" height="255" alt="Vidalia Control Panel Screenshot"/> </div>  @@ -114,6 +115,7 @@ channel, or #tor-dev. + If you have any doubts about any of the points in here, you can email <a href="<page about/contact>">contact us</a> about it.

1 0

r24288: {translation} more text, some nitpicking, still work in progress (translation/trunk/documentation)
by Runa Sandvik 02 Mar '11

02 Mar '11

Author: runa Date: 2011-03-02 16:43:14 +0000 (Wed, 02 Mar 2011) New Revision: 24288 Modified: translation/trunk/documentation/howto.txt Log: more text, some nitpicking, still work in progress Modified: translation/trunk/documentation/howto.txt =================================================================== --- translation/trunk/documentation/howto.txt 2011-03-02 15:46:57 UTC (rev 24287) +++ translation/trunk/documentation/howto.txt 2011-03-02 16:43:14 UTC (rev 24288) @@ -1,16 +1,16 @@ # This file documents the translation process for all projects # currently available in the Tor translation portal: # -# *. BridgeDB -# *. GetTor -# *. Vidalia Help Files -# *. Vidalia Installer -# *. Vidalia -# *. Orbot -# *. Torbutton-alpha -# *. Torbutton -# *. Torcheck -# *. The Website +# * BridgeDB +# * GetTor +# * Vidalia Help Files +# * Vidalia Installer +# * Vidalia +# * Orbot +# * Torbutton-alpha +# * Torbutton +# * Torcheck +# * The Website # * Tor Manual Pages # # ( More translation information for Tor related apps and documentation @@ -235,8 +235,8 @@ 5. Pushing Files to the Transifex Server - To push new/updated source files or translations to the Transifex, - simply run 'tx push' with either '-s' or '-t'. + To push new/updated source files or translations to the Transifex + server, simply run 'tx push' with either '-s' or '-t'. Vidalia: @@ -266,7 +266,7 @@ 5. Pushing Files to the Transifex Server To push new/updated source files or translations to the - Transifex, simply run 'tx push' with either '-s' or '-t'. + Transifex server, simply run 'tx push' with either '-s' or '-t'. Orbot: @@ -314,7 +314,7 @@ 5. Pushing Files to the Transifex Server To push new/updated source files or translations to the - Transifex, simply run 'tx push' with either '-s' or '-t'. + Transifex server, simply run 'tx push' with either '-s' or '-t'. Torbutton and Torbutton-alpha: @@ -324,15 +324,71 @@ and here: https://svn.torproject.org/svn/translation/trunk/projects/torbutton-alpha/. + You will also need: https://gitweb.torproject.org/torbutton.git/. + 1. Pulling Translations from the Transifex Server - To pull new translations from the server, run 'tx pull'. Remember - to commit the changes to SVN. + To pull new translations from the server, cd to the 'po' + directory and run 'tx pull'. Remember to commit the changes to + SVN. + 2. Converting Translations to Mozilla Files + # TODO: document how to convert translations to mozilla files. + + 3. Creating/Updating Translation Template Files + If you made changes to strings in Torbutton, you need to rebuild + the templates in the 'po/templates' directory. This is done with + the following command from within the torbutton.git checkout + directory: + $ moz2po -P -i src/chrome/locale/en/ -o /path/to/po/templates + 4. Adding New Translation Resources to Transifex + + To add new translation resources to Transifex, open up + 'po/.tx/config' and create entries for the new resources. Use an + existing entry as a template. Remember to commit the file to SVN. + + 5. Pushing Files to the Transifex Server + + To push new/updated source files or translations to the Transifex + server, cd to the 'po' directory and run 'tx push' with either + '-s' or '-t'. + +Torcheck: + + The Transifex configuration file, source file and translations can + be found here: + https://svn.torproject.org/svn/translation/trunk/projects/torcheck/. + + 1. Pulling Translations from the Transifex Server + + To pull new translations from the server, cd to the 'po' + directory and run 'tx pull'. Remember to commit the changes to + SVN. + + 2. Converting Translations to + + # TODO: document how to convert translations to a useful format. + + 3. Creating/Updating Translation Template Files + + # TODO: document this as well + + 4. Adding New Translation Resources to Transifex + + To add new translation resources to Transifex, open up + 'po/.tx/config' and create entries for the new resources. Use an + existing entry as a template. Remember to commit the file to SVN. + + 5. Pushing Files to the Transifex Server + + To push new/updated source files or translations to the Transifex + server, cd to the 'po' directory and run 'tx push' with either + '-s' or '-t'. + ---------------------------- TorCheck ------------------------------- TorCheck uses our translation portal to accept translations. Users use

1 0

[torspec/master] Sebastian confirms that 178 should be "Open"
by nickm＠torproject.org 02 Mar '11

02 Mar '11

commit 304314dc6d0ec77887125ed7e34caf3d3ca49c09 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 2 11:25:44 2011 -0500 Sebastian confirms that 178 should be "Open" --- proposals/000-index.txt | 4 ++-- proposals/178-param-voting.txt | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index 91c2f27..b642703 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -98,7 +98,7 @@ Proposals by number: 175 Automatically promoting Tor clients to nodes [DRAFT] 176 Proposed version-3 link handshake for Tor [OPEN] 177 Abstaining from votes on individual flags [OPEN] -178 Require majority of authorities to vote for consensus parameters [DRAFT] +178 Require majority of authorities to vote for consensus parameters [OPEN] Proposals by status: @@ -112,7 +112,6 @@ Proposals by status: 149 Using data from NETINFO cells [for 0.2.1.x] 170 Configuration options regarding circuit building 175 Automatically promoting Tor clients to nodes - 178 Require majority of authorities to vote for consensus parameters NEEDS-REVISION: 131 Help users to verify they are using Tor OPEN: @@ -130,6 +129,7 @@ Proposals by status: 171 Separate streams across circuits by connection metadata 176 Proposed version-3 link handshake for Tor [for 0.2.3] 177 Abstaining from votes on individual flags + 178 Require majority of authorities to vote for consensus parameters ACCEPTED: 110 Avoiding infinite length circuits [for 0.2.1.x] [in 0.2.1.3-alpha] 117 IPv6 exits [for 0.2.1.x] diff --git a/proposals/178-param-voting.txt b/proposals/178-param-voting.txt index ff3d055..b5664b5 100644 --- a/proposals/178-param-voting.txt +++ b/proposals/178-param-voting.txt @@ -2,7 +2,7 @@ Filename: 178-param-voting.txt Title: Require majority of authorities to vote for consensus parameters Author: Sebastian Hahn Created: 16-Feb-2011 -Status: Draft +Status: Open Overview:

1 0

[torspec/master] Close proposal 166 and make xxx-geoip-survey-plan obsolete
by nickm＠torproject.org 02 Mar '11

02 Mar '11

commit 6501e1e80a6eb44aa1ff089ced2870b6728865a8 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Mar 2 11:20:33 2011 -0500 Close proposal 166 and make xxx-geoip-survey-plan obsolete Karsten confirms that 166 is implemented, and xxx-geoip-survey-plan is superseded by this tech report: https://metrics.torproject.org/papers/countingusers-2010-11-30.pdf --- proposals/000-index.txt | 4 +- proposals/166-statistics-extra-info-docs.txt | 2 +- proposals/ideas/old/xxx-geoip-survey-plan.txt | 137 +++++++++++++++++++++++++ proposals/ideas/xxx-geoip-survey-plan.txt | 137 ------------------------- 4 files changed, 140 insertions(+), 140 deletions(-) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index 48ec6a8..91c2f27 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -86,7 +86,7 @@ Proposals by number: 163 Detecting whether a connection comes from a client [OPEN] 164 Reporting the status of server votes [OPEN] 165 Easy migration for voting authority sets [OPEN] -166 Including Network Statistics in Extra-Info Documents [ACCEPTED] +166 Including Network Statistics in Extra-Info Documents [CLOSED] 167 Vote on network parameters in consensus [CLOSED] 168 Reduce default circuit window [OPEN] 169 Eliminate TLS renegotiation for the Tor connection handshake [SUPERSEDED] @@ -137,7 +137,6 @@ Proposals by status: 140 Provide diffs between consensuses [for 0.2.2.x] 147 Eliminate the need for v2 directories in generating v3 directories [for 0.2.1.x] 157 Make certificate downloads specific [for 0.2.1.x] - 166 Including Network Statistics in Extra-Info Documents [for 0.2.2] 172 GETINFO controller option for circuit information 173 GETINFO Option Expansion 174 Optimistic Data for Tor: Server Side @@ -179,6 +178,7 @@ Proposals by status: 148 Stream end reasons from the client side should be uniform [in 0.2.1.9-alpha] 150 Exclude Exit Nodes from a circuit [in 0.2.1.3-alpha] 152 Optionally allow exit from single-hop circuits [in 0.2.1.6-alpha] + 166 Including Network Statistics in Extra-Info Documents [for 0.2.2] 167 Vote on network parameters in consensus [in 0.2.2] SUPERSEDED: 112 Bring Back Pathlen Coin Weight diff --git a/proposals/166-statistics-extra-info-docs.txt b/proposals/166-statistics-extra-info-docs.txt index ab2716a..8b0c6a1 100644 --- a/proposals/166-statistics-extra-info-docs.txt +++ b/proposals/166-statistics-extra-info-docs.txt @@ -3,7 +3,7 @@ Title: Including Network Statistics in Extra-Info Documents Author: Karsten Loesing Created: 21-Jul-2009 Target: 0.2.2 -Status: Accepted +Status: Closed Change history: diff --git a/proposals/ideas/old/xxx-geoip-survey-plan.txt b/proposals/ideas/old/xxx-geoip-survey-plan.txt new file mode 100644 index 0000000..49c6615 --- /dev/null +++ b/proposals/ideas/old/xxx-geoip-survey-plan.txt @@ -0,0 +1,137 @@ + + +Abstract + + This document explains how to tell about how many Tor users there + are, and how many there are in which country. Statistics are + involved. + +Motivation + + There are a few reasons we need to keep track of which countries + Tor users (in aggregate) are coming from: + + - Resource allocation. Knowing about underserved countries with + lots of users can let us know about where we need to direct + translation and outreach efforts. + + - Anticensorship. Sudden drops in usage on a national basis can + indicate the arrival of a censorious firewall. + + - Sponsor outreach and self-evalutation. Many people and + organizations who are interested in funding The Tor Project's + work want to know that we're successfully serving parts of the + world they're interested in, and that efforts to expand our + userbase are actually succeeding. So do we. + +Goals + + We want to know approximately how many Tor users there are, and which + countries they're in, even in the presence of a hypothetical + "directory guard" feature. Some uncertainty is okay, but we'd like + to be able to put a bound on the uncertainty. + + We need to make sure this information isn't exposed in a way that + helps an adversary. + +Methods for current clients: + + Every client downloads network status documents. There are + currently three methods (one hypothetical) for clients to get them. + - 0.1.2.x clients (and earlier) fetch a v2 networkstatus + document about every NETWORKSTATUS_CLIENT_DL_INTERVAL [30 + minutes]. + + - 0.2.0.x clients fetch a v3 networkstatus consensus document + at a random interval between when their current document is no + longer freshest, and when their current document is about to + expire. + + [In both of the above cases, clients choose a running + directory cache at random with odds roughly proportional to + its bandwidth. If they're just starting, they know a XXXX FIXME -NM] + + - In some future version, clients will choose directory caches + to serve as their "directory guards" to avoid profiling + attacks, similarly to how clients currently start all their + circuits at guard nodes. + + We assume that a directory cache can tell which of these three + categories a client is in by the format of its status request. + + A directory cache can be made to count distinct client IP + addresses that make a certain request of it in a given timeframe, + and total requests made to it over that timeframe. For the first + two cases, a cache can get a picture of the overall + number and countries of users in the network by dividing the IP + count by the probability with which they (as a cache) would be + chosen. Assuming that our listed bandwidth is such that we expect + to be chosen with probability P for any given request, and we've + been counting IPs for long enough that we expect the average + client to have made N requests, they will have visited us at least + once with probability P' = 1-(1-P)^N, and so we divide the IP + counts we've seen by P' for our estimate. To estimate total + number of clients of a given type, determine how many requests a + client of that type will make over that time, and assume we'll + have seen P of them. + + Both of these numbers are useful: the IP counts will give the + total number of IPs connecting to the network, and the request + counts will give the total number of users on the network at any + given time. + + Notes: + - [Over H hours, the N for V2 clients is 2*H, and the N for V3 + clients is currently around H/2 or H/3.] + + - (We should only count requests that we actually intend to answer; + 503 requests shouldn't count.) + + - These measurements should also be taken at a directory + authority if possible: their picture of the network is skewed + by clients that fetch from them directly. These clients, + however, are all the clients that are just bootstrapping + (assuming that the fallback-consensus feature isn't yet used + much). + + - These measurements also overestimate the V2 download rate if + some downloads fail and clients retry them later after backing + off. + +Methods for directory guards: + + If directory guards are in use, directory guards get a picture of + all those users who chose them as a guard when they were listed + as a good choice for a guard, and who are also on the network + now. The cleanest data here will come from nodes that were listed + as good new-guards choices for a while, and have not been so for a + while longer (to study decay rates); nodes that have been listed + as good new-guard choices consistently for a long time (to get a + sample of the network); and nodes that have been listed as good + new-guard choices only recently (to get a sample of new users and + users whose guards have died out.) + + Since directory guards are currently unspecified, we'll need to + make some guesses about how they'll turn out to work. Here are + a couple of approaches that could work. + - We could have clients pick completely new directory guards on + a rolling basis every two months or so. This would ensure + that staying as a guard for a while would be sufficient to + see a sample of users. This is potentially advantageous for + load-balancing the network as well, though it might lose some + of the benefits of directory guard. We need to quantify the + impact of this; it might not actually make stuff worse in + practice, if most guards don't stay good guards for a month + or two. + + - We could try to collect statistics at several directory + guards and combine their statisics, but we would need to make + sure that for all time, at least one of the directory guards + had been recommended as a good choice for new guards. By + looking at new-IP rates for guards, we could get an idea of + user uptake; for looking at old-IP decay rates, we could get + an idea of turnover. This approach would entail significant + complexity, and we'd probably need to record more information + than we'd really like to. + + diff --git a/proposals/ideas/xxx-geoip-survey-plan.txt b/proposals/ideas/xxx-geoip-survey-plan.txt deleted file mode 100644 index 49c6615..0000000 --- a/proposals/ideas/xxx-geoip-survey-plan.txt +++ /dev/null @@ -1,137 +0,0 @@ - - -Abstract - - This document explains how to tell about how many Tor users there - are, and how many there are in which country. Statistics are - involved. - -Motivation - - There are a few reasons we need to keep track of which countries - Tor users (in aggregate) are coming from: - - - Resource allocation. Knowing about underserved countries with - lots of users can let us know about where we need to direct - translation and outreach efforts. - - - Anticensorship. Sudden drops in usage on a national basis can - indicate the arrival of a censorious firewall. - - - Sponsor outreach and self-evalutation. Many people and - organizations who are interested in funding The Tor Project's - work want to know that we're successfully serving parts of the - world they're interested in, and that efforts to expand our - userbase are actually succeeding. So do we. - -Goals - - We want to know approximately how many Tor users there are, and which - countries they're in, even in the presence of a hypothetical - "directory guard" feature. Some uncertainty is okay, but we'd like - to be able to put a bound on the uncertainty. - - We need to make sure this information isn't exposed in a way that - helps an adversary. - -Methods for current clients: - - Every client downloads network status documents. There are - currently three methods (one hypothetical) for clients to get them. - - 0.1.2.x clients (and earlier) fetch a v2 networkstatus - document about every NETWORKSTATUS_CLIENT_DL_INTERVAL [30 - minutes]. - - - 0.2.0.x clients fetch a v3 networkstatus consensus document - at a random interval between when their current document is no - longer freshest, and when their current document is about to - expire. - - [In both of the above cases, clients choose a running - directory cache at random with odds roughly proportional to - its bandwidth. If they're just starting, they know a XXXX FIXME -NM] - - - In some future version, clients will choose directory caches - to serve as their "directory guards" to avoid profiling - attacks, similarly to how clients currently start all their - circuits at guard nodes. - - We assume that a directory cache can tell which of these three - categories a client is in by the format of its status request. - - A directory cache can be made to count distinct client IP - addresses that make a certain request of it in a given timeframe, - and total requests made to it over that timeframe. For the first - two cases, a cache can get a picture of the overall - number and countries of users in the network by dividing the IP - count by the probability with which they (as a cache) would be - chosen. Assuming that our listed bandwidth is such that we expect - to be chosen with probability P for any given request, and we've - been counting IPs for long enough that we expect the average - client to have made N requests, they will have visited us at least - once with probability P' = 1-(1-P)^N, and so we divide the IP - counts we've seen by P' for our estimate. To estimate total - number of clients of a given type, determine how many requests a - client of that type will make over that time, and assume we'll - have seen P of them. - - Both of these numbers are useful: the IP counts will give the - total number of IPs connecting to the network, and the request - counts will give the total number of users on the network at any - given time. - - Notes: - - [Over H hours, the N for V2 clients is 2*H, and the N for V3 - clients is currently around H/2 or H/3.] - - - (We should only count requests that we actually intend to answer; - 503 requests shouldn't count.) - - - These measurements should also be taken at a directory - authority if possible: their picture of the network is skewed - by clients that fetch from them directly. These clients, - however, are all the clients that are just bootstrapping - (assuming that the fallback-consensus feature isn't yet used - much). - - - These measurements also overestimate the V2 download rate if - some downloads fail and clients retry them later after backing - off. - -Methods for directory guards: - - If directory guards are in use, directory guards get a picture of - all those users who chose them as a guard when they were listed - as a good choice for a guard, and who are also on the network - now. The cleanest data here will come from nodes that were listed - as good new-guards choices for a while, and have not been so for a - while longer (to study decay rates); nodes that have been listed - as good new-guard choices consistently for a long time (to get a - sample of the network); and nodes that have been listed as good - new-guard choices only recently (to get a sample of new users and - users whose guards have died out.) - - Since directory guards are currently unspecified, we'll need to - make some guesses about how they'll turn out to work. Here are - a couple of approaches that could work. - - We could have clients pick completely new directory guards on - a rolling basis every two months or so. This would ensure - that staying as a guard for a while would be sufficient to - see a sample of users. This is potentially advantageous for - load-balancing the network as well, though it might lose some - of the benefits of directory guard. We need to quantify the - impact of this; it might not actually make stuff worse in - practice, if most guards don't stay good guards for a month - or two. - - - We could try to collect statistics at several directory - guards and combine their statisics, but we would need to make - sure that for all time, at least one of the directory guards - had been recommended as a good choice for new guards. By - looking at new-IP rates for guards, we could get an idea of - user uptake; for looking at old-IP decay rates, we could get - an idea of turnover. This approach would entail significant - complexity, and we'd probably need to record more information - than we'd really like to. - -

1 0

r24287: {translation} documenting the workflow for all of our translation projects (translation/trunk/documentation)
by Runa Sandvik 02 Mar '11

02 Mar '11

Author: runa Date: 2011-03-02 15:46:57 +0000 (Wed, 02 Mar 2011) New Revision: 24287 Modified: translation/trunk/documentation/howto.txt Log: documenting the workflow for all of our translation projects, work in progress Modified: translation/trunk/documentation/howto.txt =================================================================== --- translation/trunk/documentation/howto.txt 2011-03-02 15:26:23 UTC (rev 24286) +++ translation/trunk/documentation/howto.txt 2011-03-02 15:46:57 UTC (rev 24287) @@ -1,32 +1,338 @@ -## Instructions for helping translate text for Vidalia, TorButton -## and TorCheck -## ( More translation information for Tor related apps will accumulate here ) +# This file documents the translation process for all projects +# currently available in the Tor translation portal: +# +# *. BridgeDB +# *. GetTor +# *. Vidalia Help Files +# *. Vidalia Installer +# *. Vidalia +# *. Orbot +# *. Torbutton-alpha +# *. Torbutton +# *. Torcheck +# *. The Website +# * Tor Manual Pages +# +# ( More translation information for Tor related apps and documentation +# will accumulate here ) -Our translations are handled in one of two places. The Tor Translation Portal -handles all of the translations for Vidalia, Torbutton and TorCheck. The Tor -website itself is currently handled by hand translations using subversion. -------------------------------------------------------------------------- +About Transifex: -For the Tor website, you'll need a Tor SVN account. -If you do not have one and you need one, please run this command with your -desired username in place of 'USERNAME': - htdigest -c passwd.tmp "Tor subversion repository" USERNAME -and send us the contents of passwd.tmp. + All of the translation projects are located here: + https://www.transifex.net/projects/p/torproject/. -------------------------------------------------------------------------- + Transifex is a website that lets users contribute translations + online using their web browser. Users will have to register at + https://www.transifex.net/accounts/register/ before they can start + translating. -For the Portal-based projects, all three check in their respective .po -files into the following subversion urls: + A project may have many files that can be translated. In transifex, + these files are called resources. Transifex lists all sentences or + phrases (called "strings") used by a particular resource, and allows + interested volunteers to translate individual sentences or phrases + as they are able. - https://tor-svn.freehaven.net/svn/translation/trunk/projects/torbutton - https://tor-svn.freehaven.net/svn/translation/trunk/projects/torcheck - https://svn.vidalia-project.net/svn/vidalia/trunk/src/vidalia/i18n/ +The Transifex Client: -The current pootle configuration is checked into subversion as well: + The Transifex Command-line Client enables you to easily manage your + translations within a project without the need of a GUI. You can use + the command line client to easily create new resources, map local + files to translations and synchronize translation projects with your + local repository and vice verca. - https://tor-svn.freehaven.net/svn/translation/trunk/pootle + For more information about the Transifex client, see + http://help.transifex.net/user-guide/client/index.html#user-client + 1. Installing the client + + The Transifex Client can be found on the Python Package Index + (PYPI) under the name transifex-client. So to install it, make sure you + have a recent version of setuptools installed and then issue the + following command: + + $ easy_install -U transifex-client + + This command will install the latest version of transifex-client + and if you have an outdated version it’ll perform an update. + + 2. Init + + After installing the client, you will need to run the command + below to initialize a project and generate your own config file. + You may run this command from anywhere on your filesystem: + + $ tx init + + NOTE: You will be asked to enter the URL of the Transifex + instance you wish to connect to. Make sure it says + "https://www.transifex.net". + + The init command will create the following files for you: + ~/.tx/config and ~/.transifexrc. Each of the translation projects + have their own configuration file, so you can safely delete + ~/.tx/config. The other file, ~/.transifexrc, holds your username + and password for Transifex. + + 3. Set + + The following command is an aggregation of all functions needed + to initialize and setup a new translation resource: + + $ tx set + + For more information and other examples, see + http://help.transifex.net/user-guide/client/index.html + + As an example, here's how you would initialize and set up Orbot + as a new translation resource. + + Check out the Orbot translation directory from SVN: + https://svn.torproject.org/svn/translation/trunk/projects/orbot/. + + Inside the 'po' directory, run the following command to + initialize the new resource: + + ~/translation/projects/orbot/po$ tx init + + Once that is done, you can create the resource 'orbot-strings-pot' + and assign a source file: + + ~/translation/projects/orbot/po$ tx set --source -r torproject.orbot-strings-pot -l en strings.pot + + Once you have created a mapping between a source file and a + resource, you should add more translations to this resource. To + set a specific file as the source and auto-detect translations + for all languages, run the following command: + + ~/translation/projects/orbot/po$ tx set --auto-local -r torproject.orbot-strings-pot '<lang>/strings.po' --source-lang en --source-file strings.pot + + The Transifex client will store this information in the following + file: /translation/projects/orbot/po/.tx/config. + + To check the existing mapping of source files and translations, run the following command: + + ~/translation/projects/orbot/po$ tx status + + When you have assigned all of the source files and added all + existing translations, push the changes up to the Transifex + server (see below for information on how to push local changes). + Remember to commit the .tx/config file to SVN. + + NOTE: you can also initialize and setup new translation resources + on https://www.transifex.net/. + + 4. Push + + The following command sends local changes to the Transifex server: + + $ tx push [-s|--source or -t|--translations] + + If you have added new source files, the corresponding resources + are created on the Transifex server and if you have new + translations those will get pushed as well. This can also be used + to update existing source files or translations on the server + with new strings. Use --help for more information. + + NOTE: All translation resources have been configured to + automatically update the source files from URL. It is not + necessary to push the source files manually. + + 5. Pull + + The following command pdates your local files by pulling the + latest translations from the server as well as new translation + files that were created in the Transifex UI: + + $ tx pull + + By default, this command will check if the modification time of + your local files is more recent than that on the Transifex server + and will not overwrite newer files. Use --help for more + information. + + +BridgeDB: + + # TODO: document the translation workflow for BridgeDB when we have + # figured it out. + +GetTor: + + # TODO: document the translation workflow for GetTor when we have + # figured it out. + +Vidalia Help Files: + + The Transifex configuration file, source files and translations can + be found here: + https://svn.torproject.org/vidalia/vidalia/trunk/src/vidalia/help/content/. + + 1. Pulling Translations from the Transifex Server + + To pull new translations from the server, cd to the 'po' + directory and run 'tx pull'. Remember to commit the changes to + SVN. + + 2. Converting Translations to HTML + + Simply pulling translations from the Transifex server is not + enough. To make use of the translations, you will need to convert + the .po files to .html files. + + You can use the following script to automate the process: + https://svn.torproject.org/vidalia/vidalia/trunk/src/vidalia/help/content/p… + + NOTE: You need to have the package 'po4a' installed before running the script. + + 3. Creating/Updating Translation Template Files + + To create a .pot file from an .html file, run the following command: + + $ po4a-gettextize -f html -m file.html -p file.pot + + You will need to edit the header of the .pot file once it has + been created. Use the header from one of the existing .pot files + in 'po/templates' as a template. Remember to commit the file to + SVN. + + 4. Adding New Translation Resources to Transifex + + To add new translation resources to Transifex, open up + 'po/.tx/config' and create entries for the new resources. Use an + existing entry as a template. Remember to commit the file to SVN. + + 5. Pushing Files to the Transifex Server + + To push new/updated source files or translations to the Transifex + server, cd to the 'po' directory and run 'tx push' with either + '-s' or '-t'. + +Vidalia Installer: + + The Transifex configuration file, source file and translations can + be found here: + https://svn.torproject.org/vidalia/vidalia/trunk/pkg/win32/po/. + + 1. Pulling Translations from the Transifex Server + + To pull new translations from the server, run 'tx pull'. Remember + to commit the changes to SVN. + + 2. Converting Translations + + # TODO: document how to convert translations to a useful format. + + 3. Creating/Updating Translation Template Files + + # TODO: document how to convert from a useful format to .pot + + 4. Adding New Translation Resources to Transifex + + To add new translation resources to Transifex, open up + '.tx/config' and create entries for the new resources. Use an + existing entry as a template. Remember to commit the file to SVN. + + 5. Pushing Files to the Transifex Server + + To push new/updated source files or translations to the Transifex, + simply run 'tx push' with either '-s' or '-t'. + +Vidalia: + + The Transifex configuration file, source file and translations can + be found here: + https://svn.torproject.org/vidalia/vidalia/trunk/src/vidalia/i18n/po/. + + 1. Pulling Translations from the Transifex Server + + To pull new translations from the server, run 'tx pull'. Remember + to commit the changes to SVN. + + 2. Converting Translations + + # TODO: document how to convert translations to a useful format. + + 3. Creating/Updating Translation Template Files + + # TODO: document how to convert from a useful format to .pot + + 4. Adding New Translation Resources to Transifex + + To add new translation resources to Transifex, open up + '.tx/config' and create entries for the new resources. Use an + existing entry as a template. Remember to commit the file to SVN. + + 5. Pushing Files to the Transifex Server + + To push new/updated source files or translations to the + Transifex, simply run 'tx push' with either '-s' or '-t'. + +Orbot: + + The Transifex configuration file, source file and translations can + be found here: + https://svn.torproject.org/svn/translation/trunk/projects/orbot/po/. + + 1. Pulling Translations from the Transifex Server + + To pull new translations from the server, run 'tx pull'. Remember + to commit the changes to SVN. + + 2. Converting Translations to XML + + Simply pulling translations from the Transifex server is not + enough. To make use of the translations, you will need to convert + the .po files to .xml files. + + Start by checking out the following directory: + https://svn.torproject.org/svn/projects/android/trunk/Orbot/. + + You can use the following script to convert translated .po files to .xml files:: + https://svn.torproject.org/vidalia/vidalia/trunk/src/vidalia/help/content/p… + + NOTE: You need to have the package 'po4a' installed before running the script. + + 3. Creating/Updating Translation Template Files + + To create a .pot file from an .xml file, run the following + command: + + $ po4a-gettextize -f xml -m file.xml -p file.pot + + You will need to edit the header of the .pot file once it has + been created. Use the header from one of the existing .pot files + in 'orbot/po/templates' as a template. Remember to commit the + file to SVN. + + 4. Adding New Translation Resources to Transifex + + To add new translation resources to Transifex, open up + '.tx/config' and create entries for the new resources. Use an + existing entry as a template. Remember to commit the file to SVN. + + 5. Pushing Files to the Transifex Server + + To push new/updated source files or translations to the + Transifex, simply run 'tx push' with either '-s' or '-t'. + +Torbutton and Torbutton-alpha: + + The Transifex configuration file, source file and translations can + be found here: + https://svn.torproject.org/svn/translation/trunk/projects/torbutton/ + and here: + https://svn.torproject.org/svn/translation/trunk/projects/torbutton-alpha/. + + 1. Pulling Translations from the Transifex Server + + To pull new translations from the server, run 'tx pull'. Remember + to commit the changes to SVN. + + + + + ---------------------------- TorCheck ------------------------------- TorCheck uses our translation portal to accept translations. Users use

1 0

[metrics-web/master] Import relay descriptors as part of metrics-web.
by karsten＠torproject.org 02 Mar '11

02 Mar '11

commit dac73329a421cb3b13978c56be88ad36abd1c159 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Mar 1 16:40:55 2011 +0100 Import relay descriptors as part of metrics-web. --- build.xml | 1 + config.template | 14 + src/org/torproject/ernie/cron/Configuration.java | 27 + src/org/torproject/ernie/cron/Main.java | 19 +- .../cron/RelayDescriptorDatabaseImporter.java | 1213 ++++++++++++++++++++ .../ernie/cron/RelayDescriptorParser.java | 241 ++++- 6 files changed, 1509 insertions(+), 6 deletions(-) diff --git a/build.xml b/build.xml index 86ce310..446572b 100644 --- a/build.xml +++ b/build.xml @@ -13,6 +13,7 @@ <path id="classpath"> <pathelement path="${classes}"/> <pathelement location="lib/commons-codec-1.4.jar"/> + <pathelement location="lib/postgresql-8.4-702.jdbc3.jar"/> </path> <target name="init"> diff --git a/config.template b/config.template index 479d78c..1d0701c 100644 --- a/config.template +++ b/config.template @@ -10,6 +10,20 @@ ## again, but it can be confusing to users who don't know about it. #KeepDirectoryArchiveImportHistory 0 # +## Write relay descriptors to the database +#WriteRelayDescriptorDatabase 0 +# +## JDBC string for relay descriptor database +#RelayDescriptorDatabaseJDBC jdbc:postgresql://localhost/tordir?user=metrics&password=password +# +## Write relay descriptors to raw text files for importing them into the +## database using PostgreSQL's \copy command +#WriteRelayDescriptorsRawFiles 0 +# +## Relative path to directory to write raw text files; note that existing +## files will be overwritten! +#RelayDescriptorRawFilesDirectory pg-import/ +# ## Write statistics about the current consensus and votes to the ## website #WriteConsensusHealth 0 diff --git a/src/org/torproject/ernie/cron/Configuration.java b/src/org/torproject/ernie/cron/Configuration.java index 6b76dc7..66ad778 100644 --- a/src/org/torproject/ernie/cron/Configuration.java +++ b/src/org/torproject/ernie/cron/Configuration.java @@ -16,6 +16,11 @@ public class Configuration { private boolean importDirectoryArchives = false; private String directoryArchivesDirectory = "archives/"; private boolean keepDirectoryArchiveImportHistory = false; + private boolean writeRelayDescriptorDatabase = false; + private String relayDescriptorDatabaseJdbc = + "jdbc:postgresql://localhost/tordir?user=metrics&password=password"; + private boolean writeRelayDescriptorsRawFiles = false; + private String relayDescriptorRawFilesDirectory = "pg-import/"; private boolean writeConsensusHealth = false; public Configuration() { @@ -42,6 +47,16 @@ public class Configuration { } else if (line.startsWith("KeepDirectoryArchiveImportHistory")) { this.keepDirectoryArchiveImportHistory = Integer.parseInt( line.split(" ")[1]) != 0; + } else if (line.startsWith("WriteRelayDescriptorDatabase")) { + this.writeRelayDescriptorDatabase = Integer.parseInt( + line.split(" ")[1]) != 0; + } else if (line.startsWith("RelayDescriptorDatabaseJDBC")) { + this.relayDescriptorDatabaseJdbc = line.split(" ")[1]; + } else if (line.startsWith("WriteRelayDescriptorsRawFiles")) { + this.writeRelayDescriptorsRawFiles = Integer.parseInt( + line.split(" ")[1]) != 0; + } else if (line.startsWith("RelayDescriptorRawFilesDirectory")) { + this.relayDescriptorRawFilesDirectory = line.split(" ")[1]; } else if (line.startsWith("WriteConsensusHealth")) { this.writeConsensusHealth = Integer.parseInt( line.split(" ")[1]) != 0; @@ -79,6 +94,18 @@ public class Configuration { public boolean getKeepDirectoryArchiveImportHistory() { return this.keepDirectoryArchiveImportHistory; } + public boolean getWriteRelayDescriptorDatabase() { + return this.writeRelayDescriptorDatabase; + } + public String getRelayDescriptorDatabaseJDBC() { + return this.relayDescriptorDatabaseJdbc; + } + public boolean getWriteRelayDescriptorsRawFiles() { + return this.writeRelayDescriptorsRawFiles; + } + public String getRelayDescriptorRawFilesDirectory() { + return this.relayDescriptorRawFilesDirectory; + } public boolean getWriteConsensusHealth() { return this.writeConsensusHealth; } diff --git a/src/org/torproject/ernie/cron/Main.java b/src/org/torproject/ernie/cron/Main.java index b95a133..1a125bc 100644 --- a/src/org/torproject/ernie/cron/Main.java +++ b/src/org/torproject/ernie/cron/Main.java @@ -37,10 +37,20 @@ public class Main { ConsensusHealthChecker chc = config.getWriteConsensusHealth() ? new ConsensusHealthChecker() : null; + // Prepare writing relay descriptors to database + RelayDescriptorDatabaseImporter rddi = + config.getWriteRelayDescriptorDatabase() || + config.getWriteRelayDescriptorsRawFiles() ? + new RelayDescriptorDatabaseImporter( + config.getWriteRelayDescriptorDatabase() ? + config.getRelayDescriptorDatabaseJDBC() : null, + config.getWriteRelayDescriptorsRawFiles() ? + config.getRelayDescriptorRawFilesDirectory() : null) : null; + // Prepare relay descriptor parser (only if we are writing the // consensus-health page to disk) - RelayDescriptorParser rdp = config.getWriteConsensusHealth() ? - new RelayDescriptorParser(chc) : null; + RelayDescriptorParser rdp = chc != null || rddi != null ? + new RelayDescriptorParser(chc, rddi) : null; // Import relay descriptors if (rdp != null) { @@ -52,6 +62,11 @@ public class Main { } } + // Close database connection (if active) + if (rddi != null) { + rddi.closeConnection(); + } + // Write consensus health website if (chc != null) { chc.writeStatusWebsite(); diff --git a/src/org/torproject/ernie/cron/RelayDescriptorDatabaseImporter.java b/src/org/torproject/ernie/cron/RelayDescriptorDatabaseImporter.java new file mode 100644 index 0000000..f8f1e9a --- /dev/null +++ b/src/org/torproject/ernie/cron/RelayDescriptorDatabaseImporter.java @@ -0,0 +1,1213 @@ +/* Copyright 2011 The Tor Project + * See LICENSE for licensing information */ +package org.torproject.ernie.cron; + +import java.io.*; +import java.sql.*; +import java.text.*; +import java.util.*; +import java.util.logging.*; +import org.postgresql.util.*; + +/** + * Parse directory data. + */ + +public final class RelayDescriptorDatabaseImporter { + + /** + * How many records to commit with each database transaction. + */ + private final long autoCommitCount = 500; + + /** + * Keep track of the number of records committed before each transaction + */ + private int rdsCount = 0; + private int resCount = 0; + private int rhsCount = 0; + private int rrsCount = 0; + private int rcsCount = 0; + private int rvsCount = 0; + private int rbsCount = 0; + private int rqsCount = 0; + + /** + * Relay descriptor database connection. + */ + private Connection conn; + + /** + * Prepared statement to check whether any network status consensus + * entries matching a given valid-after time have been imported into the + * database before. + */ + private PreparedStatement psSs; + + /** + * Prepared statement to check whether a given network status consensus + * entry has been imported into the database before. + */ + private PreparedStatement psRs; + + /** + * Prepared statement to check whether a given extra-info descriptor has + * been imported into the database before. + */ + private PreparedStatement psEs; + + /** + * Prepared statement to check whether a given server descriptor has + * been imported into the database before. + */ + private PreparedStatement psDs; + + /** + * Prepared statement to check whether a given network status consensus + * has been imported into the database before. + */ + private PreparedStatement psCs; + + /** + * Prepared statement to check whether a given network status vote has + * been imported into the database before. + */ + private PreparedStatement psVs; + + /** + * Prepared statement to check whether a given conn-bi-direct stats + * string has been imported into the database before. + */ + private PreparedStatement psBs; + + /** + * Prepared statement to check whether a given dirreq stats string has + * been imported into the database before. + */ + private PreparedStatement psQs; + + /** + * Set of dates that have been inserted into the database for being + * included in the next refresh run. + */ + private Set<Long> scheduledUpdates; + + /** + * Prepared statement to insert a date into the database that shall be + * included in the next refresh run. + */ + private PreparedStatement psU; + + /** + * Prepared statement to insert a network status consensus entry into + * the database. + */ + private PreparedStatement psR; + + /** + * Prepared statement to insert a server descriptor into the database. + */ + private PreparedStatement psD; + + /** + * Prepared statement to insert an extra-info descriptor into the + * database. + */ + private PreparedStatement psE; + + /** + * Callable statement to insert the bandwidth history of an extra-info + * descriptor into the database. + */ + private CallableStatement csH; + + /** + * Prepared statement to insert a network status consensus into the + * database. + */ + private PreparedStatement psC; + + /** + * Prepared statement to insert a network status vote into the + * database. + */ + private PreparedStatement psV; + + /** + * Prepared statement to insert a conn-bi-direct stats string into the + * database. + */ + private PreparedStatement psB; + + /** + * Prepared statement to insert a given dirreq stats string into the + * database. + */ + private PreparedStatement psQ; + + /** + * Logger for this class. + */ + private Logger logger; + + /** + * Directory for writing raw import files. + */ + private String rawFilesDirectory; + + /** + * Raw import file containing status entries. + */ + private BufferedWriter statusentryOut; + + /** + * Raw import file containing server descriptors. + */ + private BufferedWriter descriptorOut; + + /** + * Raw import file containing extra-info descriptors. + */ + private BufferedWriter extrainfoOut; + + /** + * Raw import file containing bandwidth histories. + */ + private BufferedWriter bwhistOut; + + /** + * Raw import file containing consensuses. + */ + private BufferedWriter consensusOut; + + /** + * Raw import file containing votes. + */ + private BufferedWriter voteOut; + + /** + * Raw import file containing conn-bi-direct stats strings. + */ + private BufferedWriter connBiDirectOut; + + /** + * Raw import file containing dirreq stats. + */ + private BufferedWriter dirReqOut; + + /** + * Date format to parse timestamps. + */ + private SimpleDateFormat dateTimeFormat; + + /** + * The last valid-after time for which we checked whether they have been + * any network status entries in the database. + */ + private long lastCheckedStatusEntries; + + /** + * Set of fingerprints that we imported for the valid-after time in + * <code>lastCheckedStatusEntries</code>. + */ + private Set<String> insertedStatusEntries; + + /** + * Flag that tells us whether we need to check whether a network status + * entry is already contained in the database or not. + */ + private boolean separateStatusEntryCheckNecessary; + + private boolean importIntoDatabase; + private boolean writeRawImportFiles; + + /** + * Initialize database importer by connecting to the database and + * preparing statements. + */ + public RelayDescriptorDatabaseImporter(String connectionURL, + String rawFilesDirectory) { + + /* Initialize logger. */ + this.logger = Logger.getLogger( + RelayDescriptorDatabaseImporter.class.getName()); + + if (connectionURL != null) { + try { + /* Connect to database. */ + this.conn = DriverManager.getConnection(connectionURL); + + /* Turn autocommit off */ + this.conn.setAutoCommit(false); + + /* Prepare statements. */ + this.psSs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM statusentry WHERE validafter = ?"); + this.psRs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM statusentry WHERE validafter = ? AND descriptor = ?"); + this.psDs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM descriptor WHERE descriptor = ?"); + this.psEs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM extrainfo WHERE extrainfo = ?"); + this.psCs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM consensus WHERE validafter = ?"); + this.psVs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM vote WHERE validafter = ? AND dirsource = ?"); + this.psBs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM connbidirect WHERE source = ? AND statsend = ?"); + this.psQs = conn.prepareStatement("SELECT COUNT(*) " + + "FROM dirreq_stats WHERE source = ? AND statsend = ?"); + this.psR = conn.prepareStatement("INSERT INTO statusentry " + + "(validafter, nickname, fingerprint, descriptor, " + + "published, address, orport, dirport, isauthority, " + + "isbadexit, isbaddirectory, isexit, isfast, isguard, " + + "ishsdir, isnamed, isstable, isrunning, isunnamed, " + + "isvalid, isv2dir, isv3dir, version, bandwidth, ports, " + + "rawdesc) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, " + + "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"); + this.psD = conn.prepareStatement("INSERT INTO descriptor " + + "(descriptor, nickname, address, orport, dirport, " + + "fingerprint, bandwidthavg, bandwidthburst, " + + "bandwidthobserved, platform, published, uptime, " + + "extrainfo, rawdesc) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, " + + "?, ?, ?, ?)"); + this.psE = conn.prepareStatement("INSERT INTO extrainfo " + + "(extrainfo, nickname, fingerprint, published, rawdesc) " + + "VALUES (?, ?, ?, ?, ?)"); + this.csH = conn.prepareCall("{call insert_bwhist(?, ?, ?, ?, ?, " + + "?)}"); + this.psC = conn.prepareStatement("INSERT INTO consensus " + + "(validafter, rawdesc) VALUES (?, ?)"); + this.psV = conn.prepareStatement("INSERT INTO vote " + + "(validafter, dirsource, rawdesc) VALUES (?, ?, ?)"); + this.psB = conn.prepareStatement("INSERT INTO connbidirect " + + "(source, statsend, seconds, belownum, readnum, writenum, " + + "bothnum) VALUES (?, ?, ?, ?, ?, ?, ?)"); + this.psQ = conn.prepareStatement("INSERT INTO dirreq_stats " + + "(source, statsend, seconds, country, requests) VALUES " + + "(?, ?, ?, ?, ?)"); + this.psU = conn.prepareStatement("INSERT INTO scheduled_updates " + + "(date) VALUES (?)"); + this.scheduledUpdates = new HashSet<Long>(); + this.importIntoDatabase = true; + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not connect to database or " + + "prepare statements.", e); + } + + /* Initialize set of fingerprints to remember which status entries + * we already imported. */ + this.insertedStatusEntries = new HashSet<String>(); + } + + /* Remember where we want to write raw import files. */ + if (rawFilesDirectory != null) { + this.rawFilesDirectory = rawFilesDirectory; + this.writeRawImportFiles = true; + } + + /* Initialize date format, so that we can format timestamps. */ + this.dateTimeFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); + this.dateTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); + } + + private void addDateToScheduledUpdates(long timestamp) + throws SQLException { + if (!this.importIntoDatabase) { + return; + } + long dateMillis = 0L; + try { + dateMillis = this.dateTimeFormat.parse( + this.dateTimeFormat.format(timestamp).substring(0, 10) + + " 00:00:00").getTime(); + } catch (ParseException e) { + this.logger.log(Level.WARNING, "Internal parsing error.", e); + return; + } + if (!this.scheduledUpdates.contains(dateMillis)) { + this.psU.setDate(1, new java.sql.Date(dateMillis)); + this.psU.execute(); + this.scheduledUpdates.add(dateMillis); + } + } + + /** + * Insert network status consensus entry into database. + */ + public void addStatusEntry(long validAfter, String nickname, + String fingerprint, String descriptor, long published, + String address, long orPort, long dirPort, + SortedSet<String> flags, String version, long bandwidth, + String ports, byte[] rawDescriptor) { + if (this.importIntoDatabase) { + try { + this.addDateToScheduledUpdates(validAfter); + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + Timestamp validAfterTimestamp = new Timestamp(validAfter); + if (lastCheckedStatusEntries != validAfter) { + this.psSs.setTimestamp(1, validAfterTimestamp, cal); + ResultSet rs = psSs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + separateStatusEntryCheckNecessary = false; + insertedStatusEntries.clear(); + } else { + separateStatusEntryCheckNecessary = true; + } + rs.close(); + lastCheckedStatusEntries = validAfter; + } + boolean alreadyContained = false; + if (separateStatusEntryCheckNecessary || + insertedStatusEntries.contains(fingerprint)) { + this.psRs.setTimestamp(1, validAfterTimestamp, cal); + this.psRs.setString(2, descriptor); + ResultSet rs = psRs.executeQuery(); + rs.next(); + if (rs.getInt(1) > 0) { + alreadyContained = true; + } + rs.close(); + } else { + insertedStatusEntries.add(fingerprint); + } + if (!alreadyContained) { + this.psR.clearParameters(); + this.psR.setTimestamp(1, validAfterTimestamp, cal); + this.psR.setString(2, nickname); + this.psR.setString(3, fingerprint); + this.psR.setString(4, descriptor); + this.psR.setTimestamp(5, new Timestamp(published), cal); + this.psR.setString(6, address); + this.psR.setLong(7, orPort); + this.psR.setLong(8, dirPort); + this.psR.setBoolean(9, flags.contains("Authority")); + this.psR.setBoolean(10, flags.contains("BadExit")); + this.psR.setBoolean(11, flags.contains("BadDirectory")); + this.psR.setBoolean(12, flags.contains("Exit")); + this.psR.setBoolean(13, flags.contains("Fast")); + this.psR.setBoolean(14, flags.contains("Guard")); + this.psR.setBoolean(15, flags.contains("HSDir")); + this.psR.setBoolean(16, flags.contains("Named")); + this.psR.setBoolean(17, flags.contains("Stable")); + this.psR.setBoolean(18, flags.contains("Running")); + this.psR.setBoolean(19, flags.contains("Unnamed")); + this.psR.setBoolean(20, flags.contains("Valid")); + this.psR.setBoolean(21, flags.contains("V2Dir")); + this.psR.setBoolean(22, flags.contains("V3Dir")); + this.psR.setString(23, version); + this.psR.setLong(24, bandwidth); + this.psR.setString(25, ports); + this.psR.setBytes(26, rawDescriptor); + this.psR.executeUpdate(); + rrsCount++; + if (rrsCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add network status " + + "consensus entry. We won't make any further SQL requests " + + "in this execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.statusentryOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.statusentryOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/statusentry.sql")); + this.statusentryOut.write(" COPY statusentry (validafter, " + + "nickname, fingerprint, descriptor, published, address, " + + "orport, dirport, isauthority, isbadExit, " + + "isbaddirectory, isexit, isfast, isguard, ishsdir, " + + "isnamed, isstable, isrunning, isunnamed, isvalid, " + + "isv2dir, isv3dir, version, bandwidth, ports, rawdesc) " + + "FROM stdin;\n"); + } + this.statusentryOut.write( + this.dateTimeFormat.format(validAfter) + "\t" + nickname + + "\t" + fingerprint.toLowerCase() + "\t" + + descriptor.toLowerCase() + "\t" + + this.dateTimeFormat.format(published) + "\t" + address + + "\t" + orPort + "\t" + dirPort + "\t" + + (flags.contains("Authority") ? "t" : "f") + "\t" + + (flags.contains("BadExit") ? "t" : "f") + "\t" + + (flags.contains("BadDirectory") ? "t" : "f") + "\t" + + (flags.contains("Exit") ? "t" : "f") + "\t" + + (flags.contains("Fast") ? "t" : "f") + "\t" + + (flags.contains("Guard") ? "t" : "f") + "\t" + + (flags.contains("HSDir") ? "t" : "f") + "\t" + + (flags.contains("Named") ? "t" : "f") + "\t" + + (flags.contains("Stable") ? "t" : "f") + "\t" + + (flags.contains("Running") ? "t" : "f") + "\t" + + (flags.contains("Unnamed") ? "t" : "f") + "\t" + + (flags.contains("Valid") ? "t" : "f") + "\t" + + (flags.contains("V2Dir") ? "t" : "f") + "\t" + + (flags.contains("V3Dir") ? "t" : "f") + "\t" + + (version != null ? version : "\\N") + "\t" + + (bandwidth >= 0 ? bandwidth : "\\N") + "\t" + + (ports != null ? ports : "\\N") + "\t"); + this.statusentryOut.write(PGbytea.toPGString(rawDescriptor). + replaceAll("\\\\", "\\\\\\\\") + "\n"); + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not write network status " + + "consensus entry to raw database import file. We won't " + + "make any further attempts to write raw import files in " + + "this execution.", e); + this.writeRawImportFiles = false; + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write network status " + + "consensus entry to raw database import file. We won't " + + "make any further attempts to write raw import files in " + + "this execution.", e); + this.writeRawImportFiles = false; + } + } + } + + /** + * Insert server descriptor into database. + */ + public void addServerDescriptor(String descriptor, String nickname, + String address, int orPort, int dirPort, String relayIdentifier, + long bandwidthAvg, long bandwidthBurst, long bandwidthObserved, + String platform, long published, long uptime, + String extraInfoDigest, byte[] rawDescriptor) { + if (this.importIntoDatabase) { + try { + this.addDateToScheduledUpdates(published); + this.addDateToScheduledUpdates( + published + 24L * 60L * 60L * 1000L); + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + this.psDs.setString(1, descriptor); + ResultSet rs = psDs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + this.psD.clearParameters(); + this.psD.setString(1, descriptor); + this.psD.setString(2, nickname); + this.psD.setString(3, address); + this.psD.setInt(4, orPort); + this.psD.setInt(5, dirPort); + this.psD.setString(6, relayIdentifier); + this.psD.setLong(7, bandwidthAvg); + this.psD.setLong(8, bandwidthBurst); + this.psD.setLong(9, bandwidthObserved); + this.psD.setString(10, new String(platform.getBytes(), + "US-ASCII")); + this.psD.setTimestamp(11, new Timestamp(published), cal); + this.psD.setLong(12, uptime); + this.psD.setString(13, extraInfoDigest); + this.psD.setBytes(14, rawDescriptor); + this.psD.executeUpdate(); + rdsCount++; + if (rdsCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } catch (UnsupportedEncodingException e) { + // US-ASCII is supported for sure + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add server " + + "descriptor. We won't make any further SQL requests in " + + "this execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.descriptorOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.descriptorOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/descriptor.sql")); + this.descriptorOut.write(" COPY descriptor (descriptor, " + + "nickname, address, orport, dirport, fingerprint, " + + "bandwidthavg, bandwidthburst, bandwidthobserved, " + + "platform, published, uptime, extrainfo, rawdesc) FROM " + + "stdin;\n"); + } + this.descriptorOut.write(descriptor.toLowerCase() + "\t" + + nickname + "\t" + address + "\t" + orPort + "\t" + dirPort + + "\t" + relayIdentifier + "\t" + bandwidthAvg + "\t" + + bandwidthBurst + "\t" + bandwidthObserved + "\t" + + (platform != null && platform.length() > 0 + ? new String(platform.getBytes(), "US-ASCII") : "\\N") + + "\t" + this.dateTimeFormat.format(published) + "\t" + + (uptime >= 0 ? uptime : "\\N") + "\t" + + (extraInfoDigest != null ? extraInfoDigest : "\\N") + + "\t"); + this.descriptorOut.write(PGbytea.toPGString(rawDescriptor). + replaceAll("\\\\", "\\\\\\\\") + "\n"); + } catch (UnsupportedEncodingException e) { + // US-ASCII is supported for sure + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not write server " + + "descriptor to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write server " + + "descriptor to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } + } + } + + /** + * Insert extra-info descriptor into database. + */ + public void addExtraInfoDescriptor(String extraInfoDigest, + String nickname, String fingerprint, long published, + byte[] rawDescriptor, List<String> bandwidthHistoryLines) { + if (this.importIntoDatabase) { + try { + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + this.psEs.setString(1, extraInfoDigest); + ResultSet rs = psEs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + this.psE.clearParameters(); + this.psE.setString(1, extraInfoDigest); + this.psE.setString(2, nickname); + this.psE.setString(3, fingerprint); + this.psE.setTimestamp(4, new Timestamp(published), cal); + this.psE.setBytes(5, rawDescriptor); + this.psE.executeUpdate(); + resCount++; + if (resCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add extra-info " + + "descriptor. We won't make any further SQL requests in " + + "this execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.extrainfoOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.extrainfoOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/extrainfo.sql")); + this.extrainfoOut.write(" COPY extrainfo (extrainfo, nickname, " + + "fingerprint, published, rawdesc) FROM stdin;\n"); + } + this.extrainfoOut.write(extraInfoDigest.toLowerCase() + "\t" + + nickname + "\t" + fingerprint.toLowerCase() + "\t" + + this.dateTimeFormat.format(published) + "\t"); + this.extrainfoOut.write(PGbytea.toPGString(rawDescriptor). + replaceAll("\\\\", "\\\\\\\\") + "\n"); + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write extra-info " + + "descriptor to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not write extra-info " + + "descriptor to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } + } + if (!bandwidthHistoryLines.isEmpty()) { + this.addBandwidthHistory(fingerprint.toLowerCase(), published, + bandwidthHistoryLines); + } + } + + private static class BigIntArray implements java.sql.Array { + + private final String stringValue; + + public BigIntArray(long[] array, int offset) { + if (array == null) { + this.stringValue = "[-1:-1]={0}"; + } else { + StringBuilder sb = new StringBuilder("[" + offset + ":" + + (offset + array.length - 1) + "]={"); + for (int i = 0; i < array.length; i++) { + sb.append((i > 0 ? "," : "") + array[i]); + } + sb.append('}'); + this.stringValue = sb.toString(); + } + } + + public String toString() { + return stringValue; + } + + public String getBaseTypeName() { + return "int8"; + } + + /* The other methods are never called; no need to implement them. */ + public void free() { + throw new UnsupportedOperationException(); + } + public Object getArray() { + throw new UnsupportedOperationException(); + } + public Object getArray(long index, int count) { + throw new UnsupportedOperationException(); + } + public Object getArray(long index, int count, + Map<String, Class<?>> map) { + throw new UnsupportedOperationException(); + } + public Object getArray(Map<String, Class<?>> map) { + throw new UnsupportedOperationException(); + } + public int getBaseType() { + throw new UnsupportedOperationException(); + } + public ResultSet getResultSet() { + throw new UnsupportedOperationException(); + } + public ResultSet getResultSet(long index, int count) { + throw new UnsupportedOperationException(); + } + public ResultSet getResultSet(long index, int count, + Map<String, Class<?>> map) { + throw new UnsupportedOperationException(); + } + public ResultSet getResultSet(Map<String, Class<?>> map) { + throw new UnsupportedOperationException(); + } + } + + public void addBandwidthHistory(String fingerprint, long published, + List<String> bandwidthHistoryStrings) { + + /* Split history lines by date and rewrite them so that the date + * comes first. */ + SortedSet<String> historyLinesByDate = new TreeSet<String>(); + for (String bandwidthHistoryString : bandwidthHistoryStrings) { + String[] parts = bandwidthHistoryString.split(" "); + if (parts.length != 6) { + this.logger.finer("Bandwidth history line does not have expected " + + "number of elements. Ignoring this line."); + continue; + } + long intervalLength = 0L; + try { + intervalLength = Long.parseLong(parts[3].substring(1)); + } catch (NumberFormatException e) { + this.logger.fine("Bandwidth history line does not have valid " + + "interval length '" + parts[3] + " " + parts[4] + "'. " + + "Ignoring this line."); + continue; + } + if (intervalLength != 900L) { + this.logger.fine("Bandwidth history line does not consist of " + + "15-minute intervals. Ignoring this line."); + continue; + } + String type = parts[0]; + String intervalEndTime = parts[1] + " " + parts[2]; + long intervalEnd, dateStart; + try { + intervalEnd = dateTimeFormat.parse(intervalEndTime).getTime(); + dateStart = dateTimeFormat.parse(parts[1] + " 00:00:00"). + getTime(); + } catch (ParseException e) { + this.logger.fine("Parse exception while parsing timestamp in " + + "bandwidth history line. Ignoring this line."); + continue; + } + if (Math.abs(published - intervalEnd) > + 7L * 24L * 60L * 60L * 1000L) { + this.logger.fine("Extra-info descriptor publication time " + + dateTimeFormat.format(published) + " and last interval " + + "time " + intervalEndTime + " in " + type + " line differ " + + "by more than 7 days! Not adding this line!"); + continue; + } + long currentIntervalEnd = intervalEnd; + StringBuilder sb = new StringBuilder(); + String[] values = parts[5].split(","); + SortedSet<String> newHistoryLines = new TreeSet<String>(); + try { + for (int i = values.length - 1; i >= -1; i--) { + if (i == -1 || currentIntervalEnd < dateStart) { + sb.insert(0, intervalEndTime + " " + type + " (" + + intervalLength + " s) "); + sb.setLength(sb.length() - 1); + String historyLine = sb.toString(); + newHistoryLines.add(historyLine); + sb = new StringBuilder(); + dateStart -= 24L * 60L * 60L * 1000L; + intervalEndTime = dateTimeFormat.format(currentIntervalEnd); + } + if (i == -1) { + break; + } + Long.parseLong(values[i]); + sb.insert(0, values[i] + ","); + currentIntervalEnd -= intervalLength * 1000L; + } + } catch (NumberFormatException e) { + this.logger.fine("Number format exception while parsing " + + "bandwidth history line. Ignoring this line."); + continue; + } + historyLinesByDate.addAll(newHistoryLines); + } + + /* Add split history lines to database. */ + String lastDate = null; + historyLinesByDate.add("EOL"); + long[] readArray = null, writtenArray = null, dirreadArray = null, + dirwrittenArray = null; + int readOffset = 0, writtenOffset = 0, dirreadOffset = 0, + dirwrittenOffset = 0; + for (String historyLine : historyLinesByDate) { + String[] parts = historyLine.split(" "); + String currentDate = parts[0]; + if (lastDate != null && (historyLine.equals("EOL") || + !currentDate.equals(lastDate))) { + BigIntArray readIntArray = new BigIntArray(readArray, + readOffset); + BigIntArray writtenIntArray = new BigIntArray(writtenArray, + writtenOffset); + BigIntArray dirreadIntArray = new BigIntArray(dirreadArray, + dirreadOffset); + BigIntArray dirwrittenIntArray = new BigIntArray(dirwrittenArray, + dirwrittenOffset); + if (this.importIntoDatabase) { + try { + long dateMillis = dateTimeFormat.parse(lastDate + + " 00:00:00").getTime(); + this.addDateToScheduledUpdates(dateMillis); + this.csH.setString(1, fingerprint); + this.csH.setDate(2, new java.sql.Date(dateMillis)); + this.csH.setArray(3, readIntArray); + this.csH.setArray(4, writtenIntArray); + this.csH.setArray(5, dirreadIntArray); + this.csH.setArray(6, dirwrittenIntArray); + this.csH.addBatch(); + rhsCount++; + if (rhsCount % autoCommitCount == 0) { + this.csH.executeBatch(); + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not insert bandwidth " + + "history line into database. We won't make any " + + "further SQL requests in this execution.", e); + this.importIntoDatabase = false; + } catch (ParseException e) { + this.logger.log(Level.WARNING, "Could not insert bandwidth " + + "history line into database. We won't make any " + + "further SQL requests in this execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.bwhistOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.bwhistOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/bwhist.sql")); + } + this.bwhistOut.write("SELECT insert_bwhist('" + fingerprint + + "','" + lastDate + "','" + readIntArray.toString() + + "','" + writtenIntArray.toString() + "','" + + dirreadIntArray.toString() + "','" + + dirwrittenIntArray.toString() + "');\n"); + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write bandwidth " + + "history to raw database import file. We won't make " + + "any further attempts to write raw import files in " + + "this execution.", e); + this.writeRawImportFiles = false; + } + } + readArray = writtenArray = dirreadArray = dirwrittenArray = null; + } + if (historyLine.equals("EOL")) { + break; + } + long lastIntervalTime; + try { + lastIntervalTime = dateTimeFormat.parse(parts[0] + " " + + parts[1]).getTime() - dateTimeFormat.parse(parts[0] + + " 00:00:00").getTime(); + } catch (ParseException e) { + continue; + } + String[] stringValues = parts[5].split(","); + long[] longValues = new long[stringValues.length]; + for (int i = 0; i < longValues.length; i++) { + longValues[i] = Long.parseLong(stringValues[i]); + } + + int offset = (int) (lastIntervalTime / (15L * 60L * 1000L)) + - longValues.length + 1; + String type = parts[2]; + if (type.equals("read-history")) { + readArray = longValues; + readOffset = offset; + } else if (type.equals("write-history")) { + writtenArray = longValues; + writtenOffset = offset; + } else if (type.equals("dirreq-read-history")) { + dirreadArray = longValues; + dirreadOffset = offset; + } else if (type.equals("dirreq-write-history")) { + dirwrittenArray = longValues; + dirwrittenOffset = offset; + } + lastDate = currentDate; + } + } + + /** + * Insert network status consensus into database. + */ + public void addConsensus(long validAfter, byte[] rawDescriptor) { + if (this.importIntoDatabase) { + try { + this.addDateToScheduledUpdates(validAfter); + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + Timestamp validAfterTimestamp = new Timestamp(validAfter); + this.psCs.setTimestamp(1, validAfterTimestamp, cal); + ResultSet rs = psCs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + this.psC.clearParameters(); + this.psC.setTimestamp(1, validAfterTimestamp, cal); + this.psC.setBytes(2, rawDescriptor); + this.psC.executeUpdate(); + rcsCount++; + if (rcsCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add network status " + + "consensus. We won't make any further SQL requests in " + + "this execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.consensusOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.consensusOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/consensus.sql")); + this.consensusOut.write(" COPY consensus (validafter, rawdesc) " + + "FROM stdin;\n"); + } + String validAfterString = this.dateTimeFormat.format(validAfter); + this.consensusOut.write(validAfterString + "\t"); + this.consensusOut.write(PGbytea.toPGString(rawDescriptor). + replaceAll("\\\\", "\\\\\\\\") + "\n"); + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not write network status " + + "consensus to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write network status " + + "consensus to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } + } + } + + /** + * Insert network status vote into database. + */ + public void addVote(long validAfter, String dirSource, + byte[] rawDescriptor) { + if (this.importIntoDatabase) { + try { + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + Timestamp validAfterTimestamp = new Timestamp(validAfter); + this.psVs.setTimestamp(1, validAfterTimestamp, cal); + this.psVs.setString(2, dirSource); + ResultSet rs = psVs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + this.psV.clearParameters(); + this.psV.setTimestamp(1, validAfterTimestamp, cal); + this.psV.setString(2, dirSource); + this.psV.setBytes(3, rawDescriptor); + this.psV.executeUpdate(); + rvsCount++; + if (rvsCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add network status " + + "vote. We won't make any further SQL requests in this " + + "execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.voteOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.voteOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/vote.sql")); + this.voteOut.write(" COPY vote (validafter, dirsource, " + + "rawdesc) FROM stdin;\n"); + } + String validAfterString = this.dateTimeFormat.format(validAfter); + this.voteOut.write(validAfterString + "\t" + dirSource + "\t"); + this.voteOut.write(PGbytea.toPGString(rawDescriptor). + replaceAll("\\\\", "\\\\\\\\") + "\n"); + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not write network status " + + "vote to raw database import file. We won't make any " + + "further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write network status " + + "vote to raw database import file. We won't make any " + + "further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } + } + } + + /** + * Insert a conn-bi-direct stats string into the database. + */ + public void addConnBiDirect(String source, String statsEnd, + long seconds, long below, long read, long write, long both) { + long statsEndTime = 0L; + try { + statsEndTime = this.dateTimeFormat.parse(statsEnd).getTime(); + } catch (ParseException e) { + this.logger.log(Level.WARNING, "Could not add conn-bi-direct " + + "stats string with interval ending '" + statsEnd + "'.", e); + return; + } + if (this.importIntoDatabase) { + try { + this.addDateToScheduledUpdates(statsEndTime); + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + Timestamp statsEndTimestamp = new Timestamp(statsEndTime); + this.psBs.setString(1, source); + this.psBs.setTimestamp(2, statsEndTimestamp, cal); + ResultSet rs = psBs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + this.psB.clearParameters(); + this.psB.setString(1, source); + this.psB.setTimestamp(2, statsEndTimestamp, cal); + this.psB.setLong(3, seconds); + this.psB.setLong(4, below); + this.psB.setLong(5, read); + this.psB.setLong(6, write); + this.psB.setLong(7, both); + this.psB.executeUpdate(); + rbsCount++; + if (rbsCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add conn-bi-direct " + + "stats string. We won't make any further SQL requests in " + + "this execution.", e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.connBiDirectOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.connBiDirectOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/connbidirect.sql")); + this.connBiDirectOut.write(" COPY connbidirect (source, " + + "statsend, seconds, belownum, readnum, writenum, " + + "bothnum) FROM stdin;\n"); + } + this.connBiDirectOut.write(source + "\t" + statsEnd + "\t" + + seconds + "\t" + below + "\t" + read + "\t" + write + "\t" + + both + "\n"); + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write conn-bi-direct " + + "stats string to raw database import file. We won't make " + + "any further attempts to write raw import files in this " + + "execution.", e); + this.writeRawImportFiles = false; + } + } + } + + /** + * Adds observations on the number of directory requests by country as + * seen on a directory at a given date to the database. + */ + public void addDirReqStats(String source, String statsEnd, long seconds, + Map<String, String> dirReqsPerCountry) { + long statsEndTime = 0L; + try { + statsEndTime = this.dateTimeFormat.parse(statsEnd).getTime(); + } catch (ParseException e) { + this.logger.log(Level.WARNING, "Could not add dirreq stats with " + + "interval ending '" + statsEnd + "'.", e); + return; + } + if (this.importIntoDatabase) { + try { + this.addDateToScheduledUpdates(statsEndTime); + Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); + Timestamp statsEndTimestamp = new Timestamp(statsEndTime); + this.psQs.setString(1, source); + this.psQs.setTimestamp(2, statsEndTimestamp, cal); + ResultSet rs = psQs.executeQuery(); + rs.next(); + if (rs.getInt(1) == 0) { + for (Map.Entry<String, String> e : + dirReqsPerCountry.entrySet()) { + this.psQ.clearParameters(); + this.psQ.setString(1, source); + this.psQ.setTimestamp(2, statsEndTimestamp, cal); + this.psQ.setLong(3, seconds); + this.psQ.setString(4, e.getKey()); + this.psQ.setLong(5, Long.parseLong(e.getValue())); + this.psQ.executeUpdate(); + rqsCount++; + if (rqsCount % autoCommitCount == 0) { + this.conn.commit(); + } + } + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add dirreq stats. We " + + "won't make any further SQL requests in this execution.", + e); + this.importIntoDatabase = false; + } + } + if (this.writeRawImportFiles) { + try { + if (this.dirReqOut == null) { + new File(rawFilesDirectory).mkdirs(); + this.dirReqOut = new BufferedWriter(new FileWriter( + rawFilesDirectory + "/dirreq_stats.sql")); + this.dirReqOut.write(" COPY dirreq_stats (source, statsend, " + + "seconds, country, requests) FROM stdin;\n"); + } + for (Map.Entry<String, String> e : + dirReqsPerCountry.entrySet()) { + this.dirReqOut.write(source + "\t" + statsEnd + "\t" + seconds + + "\t" + e.getKey() + "\t" + e.getValue() + "\n"); + } + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not write dirreq stats to " + + "raw database import file. We won't make any further " + + "attempts to write raw import files in this execution.", e); + this.writeRawImportFiles = false; + } + } + } + + /** + * Close the relay descriptor database connection. + */ + public void closeConnection() { + + /* Log stats about imported descriptors. */ + this.logger.info(String.format("Finished importing relay " + + "descriptors: %d consensuses, %d network status entries, %d " + + "votes, %d server descriptors, %d extra-info descriptors, %d " + + "bandwidth history elements, %d dirreq stats elements, and %d " + + "conn-bi-direct stats lines", rcsCount, rrsCount, rvsCount, + rdsCount, resCount, rhsCount, rqsCount, rbsCount)); + + /* Insert scheduled updates a second time, just in case the refresh + * run has started since inserting them the first time in which case + * it will miss the data inserted afterwards. We cannot, however, + * insert them only now, because if a Java execution fails at a random + * point, we might have added data, but not the corresponding dates to + * update statistics. */ + if (this.importIntoDatabase) { + try { + for (long dateMillis : this.scheduledUpdates) { + this.psU.setDate(1, new java.sql.Date(dateMillis)); + this.psU.execute(); + } + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not add scheduled dates " + + "for the next refresh run.", e); + } + } + + /* Commit any stragglers before closing. */ + if (this.conn != null) { + try { + this.csH.executeBatch(); + + this.conn.commit(); + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not commit final records to " + + "database", e); + } + try { + this.conn.close(); + } catch (SQLException e) { + this.logger.log(Level.WARNING, "Could not close database " + + "connection.", e); + } + } + + /* Close raw import files. */ + try { + if (this.statusentryOut != null) { + this.statusentryOut.write("\\.\n"); + this.statusentryOut.close(); + } + if (this.descriptorOut != null) { + this.descriptorOut.write("\\.\n"); + this.descriptorOut.close(); + } + if (this.extrainfoOut != null) { + this.extrainfoOut.write("\\.\n"); + this.extrainfoOut.close(); + } + if (this.bwhistOut != null) { + this.bwhistOut.write("\\.\n"); + this.bwhistOut.close(); + } + if (this.consensusOut != null) { + this.consensusOut.write("\\.\n"); + this.consensusOut.close(); + } + if (this.voteOut != null) { + this.voteOut.write("\\.\n"); + this.voteOut.close(); + } + if (this.connBiDirectOut != null) { + this.connBiDirectOut.write("\\.\n"); + this.connBiDirectOut.close(); + } + } catch (IOException e) { + this.logger.log(Level.WARNING, "Could not close one or more raw " + + "database import files.", e); + } + } +} + diff --git a/src/org/torproject/ernie/cron/RelayDescriptorParser.java b/src/org/torproject/ernie/cron/RelayDescriptorParser.java index 24b512b..d02022a 100644 --- a/src/org/torproject/ernie/cron/RelayDescriptorParser.java +++ b/src/org/torproject/ernie/cron/RelayDescriptorParser.java @@ -17,6 +17,12 @@ import org.apache.commons.codec.binary.*; */ public class RelayDescriptorParser { + /** + * Relay descriptor database importer that stores relay descriptor + * contents for later evaluation. + */ + private RelayDescriptorDatabaseImporter rddi; + private ConsensusHealthChecker chc; /** @@ -29,8 +35,10 @@ public class RelayDescriptorParser { /** * Initializes this class. */ - public RelayDescriptorParser(ConsensusHealthChecker chc) { + public RelayDescriptorParser(ConsensusHealthChecker chc, + RelayDescriptorDatabaseImporter rddi) { this.chc = chc; + this.rddi = rddi; /* Initialize logger. */ this.logger = Logger.getLogger(RelayDescriptorParser.class.getName()); @@ -60,31 +68,256 @@ public class RelayDescriptorParser { // time to see when we switch from hourly to half-hourly // consensuses boolean isConsensus = true; - String validAfterTime = null; - String dirSource = null; + String validAfterTime = null, nickname = null, + relayIdentity = null, serverDesc = null, version = null, + ports = null; + String fingerprint = null, dirSource = null, address = null; + long validAfter = -1L, published = -1L, bandwidth = -1L, + orPort = 0L, dirPort = 0L; + SortedSet<String> relayFlags = null; + StringBuilder rawStatusEntry = null; while ((line = br.readLine()) != null) { if (line.equals("vote-status vote")) { isConsensus = false; } else if (line.startsWith("valid-after ")) { validAfterTime = line.substring("valid-after ".length()); + validAfter = parseFormat.parse(validAfterTime).getTime(); } else if (line.startsWith("dir-source ")) { dirSource = line.split(" ")[2]; - break; + } else if (line.startsWith("fingerprint ")) { + fingerprint = line.split(" ")[1]; + } else if (line.startsWith("r ")) { + if (isConsensus && relayIdentity != null && + this.rddi != null) { + byte[] rawDescriptor = rawStatusEntry.toString().getBytes(); + this.rddi.addStatusEntry(validAfter, nickname, + relayIdentity, serverDesc, published, address, orPort, + dirPort, relayFlags, version, bandwidth, ports, + rawDescriptor); + relayFlags = null; + version = null; + bandwidth = -1L; + ports = null; + } + rawStatusEntry = new StringBuilder(line + "\n"); + String[] parts = line.split(" "); + if (parts.length < 9) { + this.logger.log(Level.WARNING, "Could not parse r line '" + + line + "' in descriptor. Skipping."); + break; + } + String publishedTime = parts[4] + " " + parts[5]; + nickname = parts[1]; + relayIdentity = Hex.encodeHexString( + Base64.decodeBase64(parts[2] + "=")). + toLowerCase(); + serverDesc = Hex.encodeHexString(Base64.decodeBase64( + parts[3] + "=")).toLowerCase(); + published = parseFormat.parse(parts[4] + " " + parts[5]). + getTime(); + address = parts[6]; + orPort = Long.parseLong(parts[7]); + dirPort = Long.parseLong(parts[8]); + } else if (line.startsWith("s ") || line.equals("s")) { + rawStatusEntry.append(line + "\n"); + relayFlags = new TreeSet<String>(); + if (line.length() > 2) { + for (String flag : line.substring(2).split(" ")) { + relayFlags.add(flag); + } + } + } else if (line.startsWith("v ")) { + rawStatusEntry.append(line + "\n"); + version = line.substring(2); + } else if (line.startsWith("w ")) { + rawStatusEntry.append(line + "\n"); + String[] parts = line.split(" "); + for (String part : parts) { + if (part.startsWith("Bandwidth=")) { + bandwidth = Long.parseLong(part.substring( + "Bandwidth=".length())); + } + } + } else if (line.startsWith("p ")) { + rawStatusEntry.append(line + "\n"); + ports = line.substring(2); } } if (isConsensus) { if (this.chc != null) { this.chc.processConsensus(validAfterTime, data); } + if (this.rddi != null) { + this.rddi.addConsensus(validAfter, data); + if (relayIdentity != null) { + byte[] rawDescriptor = rawStatusEntry.toString().getBytes(); + this.rddi.addStatusEntry(validAfter, nickname, + relayIdentity, serverDesc, published, address, orPort, + dirPort, relayFlags, version, bandwidth, ports, + rawDescriptor); + } + } } else { if (this.chc != null) { this.chc.processVote(validAfterTime, dirSource, data); } + if (this.rddi != null) { + this.rddi.addVote(validAfter, dirSource, data); + } + } + } else if (line.startsWith("router ")) { + String platformLine = null, publishedLine = null, + bandwidthLine = null, extraInfoDigest = null, + relayIdentifier = null; + String[] parts = line.split(" "); + String nickname = parts[1]; + String address = parts[2]; + int orPort = Integer.parseInt(parts[3]); + int dirPort = Integer.parseInt(parts[4]); + long published = -1L, uptime = -1L; + while ((line = br.readLine()) != null) { + if (line.startsWith("platform ")) { + platformLine = line; + } else if (line.startsWith("published ")) { + String publishedTime = line.substring("published ".length()); + published = parseFormat.parse(publishedTime).getTime(); + } else if (line.startsWith("opt fingerprint") || + line.startsWith("fingerprint")) { + relayIdentifier = line.substring(line.startsWith("opt ") ? + "opt fingerprint".length() : "fingerprint".length()). + replaceAll(" ", "").toLowerCase(); + } else if (line.startsWith("bandwidth ")) { + bandwidthLine = line; + } else if (line.startsWith("opt extra-info-digest ") || + line.startsWith("extra-info-digest ")) { + extraInfoDigest = line.startsWith("opt ") ? + line.split(" ")[2].toLowerCase() : + line.split(" ")[1].toLowerCase(); + } else if (line.startsWith("uptime ")) { + uptime = Long.parseLong(line.substring("uptime ".length())); + } + } + String ascii = new String(data, "US-ASCII"); + String startToken = "router "; + String sigToken = "\nrouter-signature\n"; + int start = ascii.indexOf(startToken); + int sig = ascii.indexOf(sigToken) + sigToken.length(); + String digest = null; + if (start >= 0 || sig >= 0 || sig > start) { + byte[] forDigest = new byte[sig - start]; + System.arraycopy(data, start, forDigest, 0, sig - start); + digest = DigestUtils.shaHex(forDigest); + } + if (this.rddi != null && digest != null) { + String[] bwParts = bandwidthLine.split(" "); + long bandwidthAvg = Long.parseLong(bwParts[1]); + long bandwidthBurst = Long.parseLong(bwParts[2]); + long bandwidthObserved = Long.parseLong(bwParts[3]); + String platform = platformLine.substring("platform ".length()); + this.rddi.addServerDescriptor(digest, nickname, address, orPort, + dirPort, relayIdentifier, bandwidthAvg, bandwidthBurst, + bandwidthObserved, platform, published, uptime, + extraInfoDigest, data); + } + } else if (line.startsWith("extra-info ")) { + String nickname = line.split(" ")[1]; + long published = -1L; + String dir = line.split(" ")[2]; + String statsEnd = null; + long seconds = -1L; + List<String> bandwidthHistory = new ArrayList<String>(); + boolean skip = false; + while ((line = br.readLine()) != null) { + if (line.startsWith("published ")) { + String publishedTime = line.substring("published ".length()); + published = parseFormat.parse(publishedTime).getTime(); + } else if (line.startsWith("read-history ") || + line.startsWith("write-history ") || + line.startsWith("dirreq-read-history ") || + line.startsWith("dirreq-write-history ")) { + bandwidthHistory.add(line); + } else if (line.startsWith("dirreq-stats-end ")) { + String[] parts = line.split(" "); + if (parts.length < 5) { + this.logger.warning("Could not parse dirreq-stats-end " + + "line '" + line + "' in descriptor. Skipping."); + break; + } + statsEnd = parts[1] + " " + parts[2]; + seconds = Long.parseLong(parts[3].substring(1)); + } else if (line.startsWith("dirreq-v3-reqs ") + && line.length() > "dirreq-v3-reqs ".length()) { + if (this.rddi != null) { + try { + int allUsers = 0; + Map<String, String> obs = new HashMap<String, String>(); + String[] parts = line.substring("dirreq-v3-reqs ". + length()).split(","); + for (String p : parts) { + String country = p.substring(0, 2); + int users = Integer.parseInt(p.substring(3)) - 4; + allUsers += users; + obs.put(country, "" + users); + } + obs.put("zy", "" + allUsers); + this.rddi.addDirReqStats(dir, statsEnd, seconds, obs); + } catch (NumberFormatException e) { + this.logger.log(Level.WARNING, "Could not parse " + + "dirreq-v3-reqs line '" + line + "' in descriptor. " + + "Skipping.", e); + break; + } + } + } else if (line.startsWith("conn-bi-direct ")) { + if (this.rddi != null) { + String[] parts = line.split(" "); + if (parts.length == 6 && + parts[5].split(",").length == 4) { + try { + String connBiDirectStatsEnd = parts[1] + " " + parts[2]; + long connBiDirectSeconds = Long.parseLong(parts[3]. + substring(1)); + String[] parts2 = parts[5].split(","); + long below = Long.parseLong(parts2[0]); + long read = Long.parseLong(parts2[1]); + long write = Long.parseLong(parts2[2]); + long both = Long.parseLong(parts2[3]); + this.rddi.addConnBiDirect(dir, connBiDirectStatsEnd, + connBiDirectSeconds, below, read, write, both); + } catch (NumberFormatException e) { + this.logger.log(Level.WARNING, "Number format " + + "exception while parsing conn-bi-direct stats " + + "string '" + line + "'. Skipping.", e); + } + } else { + this.logger.warning("Skipping invalid conn-bi-direct " + + "stats string '" + line + "'."); + } + } + } + } + String ascii = new String(data, "US-ASCII"); + String startToken = "extra-info "; + String sigToken = "\nrouter-signature\n"; + String digest = null; + int start = ascii.indexOf(startToken); + int sig = ascii.indexOf(sigToken) + sigToken.length(); + if (start >= 0 || sig >= 0 || sig > start) { + byte[] forDigest = new byte[sig - start]; + System.arraycopy(data, start, forDigest, 0, sig - start); + digest = DigestUtils.shaHex(forDigest); + } + if (this.rddi != null && digest != null) { + this.rddi.addExtraInfoDescriptor(digest, nickname, + dir.toLowerCase(), published, data, bandwidthHistory); } } } catch (IOException e) { this.logger.log(Level.WARNING, "Could not parse descriptor. " + "Skipping.", e); + } catch (ParseException e) { + this.logger.log(Level.WARNING, "Could not parse descriptor. " + + "Skipping.", e); } } }

1 0

[metrics-db/master] Remove database import from metrics-db.
by karsten＠torproject.org 02 Mar '11

02 Mar '11

commit 0cb4eefde5bc6e8ae98cb823bda25229ff755e47 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Mar 1 16:41:26 2011 +0100 Remove database import from metrics-db. --- config.template | 11 - src/org/torproject/ernie/db/Configuration.java | 28 +- src/org/torproject/ernie/db/Main.java | 29 +- .../ernie/db/RelayDescriptorDatabaseImporter.java | 1213 -------------------- .../torproject/ernie/db/RelayDescriptorParser.java | 184 +--- 5 files changed, 17 insertions(+), 1448 deletions(-) diff --git a/config.template b/config.template index 1ef915c..7385f7b 100644 --- a/config.template +++ b/config.template @@ -67,9 +67,6 @@ ## Relative path to directory to write directory archives to #DirectoryArchivesOutputDirectory directory-archive/ # -## Write relay descriptors to a database for later evaluation -#WriteRelayDescriptorDatabase 0 -# ## Write aggregate statistics (bridges and bridge users per day, directory ## clients per day, torperf results, packages requested from GetTor, etc.) ## to database for later evaluation @@ -78,14 +75,6 @@ ## JDBC string for relay descriptor database #RelayDescriptorDatabaseJDBC jdbc:postgresql://localhost/tordir?user=ernie&password=password # -## Write relay descriptors to raw text files for importing them into a -## database using PostgreSQL's \copy command -#WriteRelayDescriptorsRawFiles 0 -# -## Relative path to directory to write raw text files; note that existing -## files will be overwritten! -#RelayDescriptorRawFilesDirectory pg-import/ -# ## Write statistics about the current consensus and votes to the ## website #WriteConsensusHealth 0 diff --git a/src/org/torproject/ernie/db/Configuration.java b/src/org/torproject/ernie/db/Configuration.java index 262ec78..b8f8f39 100644 --- a/src/org/torproject/ernie/db/Configuration.java +++ b/src/org/torproject/ernie/db/Configuration.java @@ -23,12 +23,9 @@ public class Configuration { private boolean importDirectoryArchives = false; private String directoryArchivesDirectory = "archives/"; private boolean keepDirectoryArchiveImportHistory = false; - private boolean writeRelayDescriptorDatabase = false; private boolean writeAggregateStatsDatabase = false; private String relayDescriptorDatabaseJdbc = "jdbc:postgresql://localhost/tordir?user=ernie&password=password"; - private boolean writeRelayDescriptorsRawFiles = false; - private String relayDescriptorRawFilesDirectory = "pg-import/"; private boolean writeSanitizedBridges = false; private boolean replaceIPAddressesWithHashes = false; private long limitBridgeDescriptorMappings = -1L; @@ -98,19 +95,11 @@ public class Configuration { } else if (line.startsWith("KeepDirectoryArchiveImportHistory")) { this.keepDirectoryArchiveImportHistory = Integer.parseInt( line.split(" ")[1]) != 0; - } else if (line.startsWith("WriteRelayDescriptorDatabase")) { - this.writeRelayDescriptorDatabase = Integer.parseInt( - line.split(" ")[1]) != 0; } else if (line.startsWith("WriteAggregateStatsDatabase")) { this.writeAggregateStatsDatabase = Integer.parseInt( line.split(" ")[1]) != 0; } else if (line.startsWith("RelayDescriptorDatabaseJDBC")) { this.relayDescriptorDatabaseJdbc = line.split(" ")[1]; - } else if (line.startsWith("WriteRelayDescriptorsRawFiles")) { - this.writeRelayDescriptorsRawFiles = Integer.parseInt( - line.split(" ")[1]) != 0; - } else if (line.startsWith("RelayDescriptorRawFilesDirectory")) { - this.relayDescriptorRawFilesDirectory = line.split(" ")[1]; } else if (line.startsWith("WriteSanitizedBridges")) { this.writeSanitizedBridges = Integer.parseInt( line.split(" ")[1]) != 0; @@ -199,7 +188,6 @@ public class Configuration { !this.importWriteTorperfStats && !this.downloadProcessGetTorStats && !this.downloadExitList && !this.writeDirectoryArchives && - !this.writeRelayDescriptorDatabase && !this.writeAggregateStatsDatabase && !this.writeSanitizedBridges && !this.writeConsensusStats && !this.writeBridgeStats) { @@ -211,9 +199,7 @@ public class Configuration { } if ((this.importCachedRelayDescriptors || this.importDirectoryArchives || this.downloadRelayDescriptors) && - !(this.writeDirectoryArchives || - this.writeRelayDescriptorDatabase || - this.writeRelayDescriptorsRawFiles || this.writeConsensusStats || + !(this.writeDirectoryArchives || this.writeConsensusStats || this.writeBridgeStats)) { logger.warning("We are configured to import/download relay " + "descriptors, but we don't have a single data sink to write " @@ -221,8 +207,7 @@ public class Configuration { } if (!(this.importCachedRelayDescriptors || this.importDirectoryArchives || this.downloadRelayDescriptors) && - (this.writeDirectoryArchives || - this.writeRelayDescriptorDatabase)) { + this.writeDirectoryArchives) { logger.warning("We are configured to write relay descriptor to at " + "least one data sink, but we don't have a single data source " + "containing relay descriptors."); @@ -281,21 +266,12 @@ public class Configuration { public boolean getKeepDirectoryArchiveImportHistory() { return this.keepDirectoryArchiveImportHistory; } - public boolean getWriteRelayDescriptorDatabase() { - return this.writeRelayDescriptorDatabase; - } public boolean getWriteAggregateStatsDatabase() { return this.writeAggregateStatsDatabase; } public String getRelayDescriptorDatabaseJDBC() { return this.relayDescriptorDatabaseJdbc; } - public boolean getWriteRelayDescriptorsRawFiles() { - return this.writeRelayDescriptorsRawFiles; - } - public String getRelayDescriptorRawFilesDirectory() { - return this.relayDescriptorRawFilesDirectory; - } public boolean getWriteSanitizedBridges() { return this.writeSanitizedBridges; } diff --git a/src/org/torproject/ernie/db/Main.java b/src/org/torproject/ernie/db/Main.java index a495a03..4dced2c 100644 --- a/src/org/torproject/ernie/db/Main.java +++ b/src/org/torproject/ernie/db/Main.java @@ -44,23 +44,11 @@ public class Main { new ArchiveWriter( new File(config.getDirectoryArchivesOutputDirectory())) : null; - // Prepare writing relay descriptors to database - RelayDescriptorDatabaseImporter rddi = - config.getWriteRelayDescriptorDatabase() || - config.getWriteRelayDescriptorsRawFiles() ? - new RelayDescriptorDatabaseImporter( - config.getWriteRelayDescriptorDatabase() ? - config.getRelayDescriptorDatabaseJDBC() : null, - config.getWriteRelayDescriptorsRawFiles() ? - config.getRelayDescriptorRawFilesDirectory() : null) : null; - // Prepare relay descriptor parser (only if we are writing stats or // directory archives to disk) RelayDescriptorParser rdp = config.getWriteBridgeStats() || - config.getWriteDirectoryArchives() || - config.getWriteRelayDescriptorDatabase() || - config.getWriteRelayDescriptorsRawFiles() ? - new RelayDescriptorParser(bsfh, aw, rddi) : null; + config.getWriteDirectoryArchives() ? + new RelayDescriptorParser(bsfh, aw) : null; // Import/download relay descriptors from the various sources if (rdp != null) { @@ -68,12 +56,10 @@ public class Main { if (config.getDownloadRelayDescriptors()) { List<String> dirSources = config.getDownloadFromDirectoryAuthorities(); - boolean downloadCurrentConsensus = aw != null || bsfh != null || - rddi != null; + boolean downloadCurrentConsensus = aw != null || bsfh != null; boolean downloadCurrentVotes = aw != null; - boolean downloadAllServerDescriptors = aw != null || - rddi != null; - boolean downloadAllExtraInfos = aw != null || rddi != null; + boolean downloadAllServerDescriptors = aw != null; + boolean downloadAllExtraInfos = aw != null; rdd = new RelayDescriptorDownloader(rdp, dirSources, downloadCurrentConsensus, downloadCurrentVotes, downloadAllServerDescriptors, downloadAllExtraInfos); @@ -108,11 +94,6 @@ public class Main { } } - // Close database connection (if active) - if (rddi != null) { - rddi.closeConnection(); - } - // Write output to disk that only depends on relay descriptors if (aw != null) { aw.dumpStats(); diff --git a/src/org/torproject/ernie/db/RelayDescriptorDatabaseImporter.java b/src/org/torproject/ernie/db/RelayDescriptorDatabaseImporter.java deleted file mode 100644 index 3a06a58..0000000 --- a/src/org/torproject/ernie/db/RelayDescriptorDatabaseImporter.java +++ /dev/null @@ -1,1213 +0,0 @@ -/* Copyright 2010 The Tor Project - * See LICENSE for licensing information */ -package org.torproject.ernie.db; - -import java.io.*; -import java.sql.*; -import java.text.*; -import java.util.*; -import java.util.logging.*; -import org.postgresql.util.*; - -/** - * Parse directory data. - */ - -public final class RelayDescriptorDatabaseImporter { - - /** - * How many records to commit with each database transaction. - */ - private final long autoCommitCount = 500; - - /** - * Keep track of the number of records committed before each transaction - */ - private int rdsCount = 0; - private int resCount = 0; - private int rhsCount = 0; - private int rrsCount = 0; - private int rcsCount = 0; - private int rvsCount = 0; - private int rbsCount = 0; - private int rqsCount = 0; - - /** - * Relay descriptor database connection. - */ - private Connection conn; - - /** - * Prepared statement to check whether any network status consensus - * entries matching a given valid-after time have been imported into the - * database before. - */ - private PreparedStatement psSs; - - /** - * Prepared statement to check whether a given network status consensus - * entry has been imported into the database before. - */ - private PreparedStatement psRs; - - /** - * Prepared statement to check whether a given extra-info descriptor has - * been imported into the database before. - */ - private PreparedStatement psEs; - - /** - * Prepared statement to check whether a given server descriptor has - * been imported into the database before. - */ - private PreparedStatement psDs; - - /** - * Prepared statement to check whether a given network status consensus - * has been imported into the database before. - */ - private PreparedStatement psCs; - - /** - * Prepared statement to check whether a given network status vote has - * been imported into the database before. - */ - private PreparedStatement psVs; - - /** - * Prepared statement to check whether a given conn-bi-direct stats - * string has been imported into the database before. - */ - private PreparedStatement psBs; - - /** - * Prepared statement to check whether a given dirreq stats string has - * been imported into the database before. - */ - private PreparedStatement psQs; - - /** - * Set of dates that have been inserted into the database for being - * included in the next refresh run. - */ - private Set<Long> scheduledUpdates; - - /** - * Prepared statement to insert a date into the database that shall be - * included in the next refresh run. - */ - private PreparedStatement psU; - - /** - * Prepared statement to insert a network status consensus entry into - * the database. - */ - private PreparedStatement psR; - - /** - * Prepared statement to insert a server descriptor into the database. - */ - private PreparedStatement psD; - - /** - * Prepared statement to insert an extra-info descriptor into the - * database. - */ - private PreparedStatement psE; - - /** - * Callable statement to insert the bandwidth history of an extra-info - * descriptor into the database. - */ - private CallableStatement csH; - - /** - * Prepared statement to insert a network status consensus into the - * database. - */ - private PreparedStatement psC; - - /** - * Prepared statement to insert a network status vote into the - * database. - */ - private PreparedStatement psV; - - /** - * Prepared statement to insert a conn-bi-direct stats string into the - * database. - */ - private PreparedStatement psB; - - /** - * Prepared statement to insert a given dirreq stats string into the - * database. - */ - private PreparedStatement psQ; - - /** - * Logger for this class. - */ - private Logger logger; - - /** - * Directory for writing raw import files. - */ - private String rawFilesDirectory; - - /** - * Raw import file containing status entries. - */ - private BufferedWriter statusentryOut; - - /** - * Raw import file containing server descriptors. - */ - private BufferedWriter descriptorOut; - - /** - * Raw import file containing extra-info descriptors. - */ - private BufferedWriter extrainfoOut; - - /** - * Raw import file containing bandwidth histories. - */ - private BufferedWriter bwhistOut; - - /** - * Raw import file containing consensuses. - */ - private BufferedWriter consensusOut; - - /** - * Raw import file containing votes. - */ - private BufferedWriter voteOut; - - /** - * Raw import file containing conn-bi-direct stats strings. - */ - private BufferedWriter connBiDirectOut; - - /** - * Raw import file containing dirreq stats. - */ - private BufferedWriter dirReqOut; - - /** - * Date format to parse timestamps. - */ - private SimpleDateFormat dateTimeFormat; - - /** - * The last valid-after time for which we checked whether they have been - * any network status entries in the database. - */ - private long lastCheckedStatusEntries; - - /** - * Set of fingerprints that we imported for the valid-after time in - * <code>lastCheckedStatusEntries</code>. - */ - private Set<String> insertedStatusEntries; - - /** - * Flag that tells us whether we need to check whether a network status - * entry is already contained in the database or not. - */ - private boolean separateStatusEntryCheckNecessary; - - private boolean importIntoDatabase; - private boolean writeRawImportFiles; - - /** - * Initialize database importer by connecting to the database and - * preparing statements. - */ - public RelayDescriptorDatabaseImporter(String connectionURL, - String rawFilesDirectory) { - - /* Initialize logger. */ - this.logger = Logger.getLogger( - RelayDescriptorDatabaseImporter.class.getName()); - - if (connectionURL != null) { - try { - /* Connect to database. */ - this.conn = DriverManager.getConnection(connectionURL); - - /* Turn autocommit off */ - this.conn.setAutoCommit(false); - - /* Prepare statements. */ - this.psSs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM statusentry WHERE validafter = ?"); - this.psRs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM statusentry WHERE validafter = ? AND descriptor = ?"); - this.psDs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM descriptor WHERE descriptor = ?"); - this.psEs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM extrainfo WHERE extrainfo = ?"); - this.psCs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM consensus WHERE validafter = ?"); - this.psVs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM vote WHERE validafter = ? AND dirsource = ?"); - this.psBs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM connbidirect WHERE source = ? AND statsend = ?"); - this.psQs = conn.prepareStatement("SELECT COUNT(*) " - + "FROM dirreq_stats WHERE source = ? AND statsend = ?"); - this.psR = conn.prepareStatement("INSERT INTO statusentry " - + "(validafter, nickname, fingerprint, descriptor, " - + "published, address, orport, dirport, isauthority, " - + "isbadexit, isbaddirectory, isexit, isfast, isguard, " - + "ishsdir, isnamed, isstable, isrunning, isunnamed, " - + "isvalid, isv2dir, isv3dir, version, bandwidth, ports, " - + "rawdesc) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, " - + "?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"); - this.psD = conn.prepareStatement("INSERT INTO descriptor " - + "(descriptor, nickname, address, orport, dirport, " - + "fingerprint, bandwidthavg, bandwidthburst, " - + "bandwidthobserved, platform, published, uptime, " - + "extrainfo, rawdesc) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, " - + "?, ?, ?, ?)"); - this.psE = conn.prepareStatement("INSERT INTO extrainfo " - + "(extrainfo, nickname, fingerprint, published, rawdesc) " - + "VALUES (?, ?, ?, ?, ?)"); - this.csH = conn.prepareCall("{call insert_bwhist(?, ?, ?, ?, ?, " - + "?)}"); - this.psC = conn.prepareStatement("INSERT INTO consensus " - + "(validafter, rawdesc) VALUES (?, ?)"); - this.psV = conn.prepareStatement("INSERT INTO vote " - + "(validafter, dirsource, rawdesc) VALUES (?, ?, ?)"); - this.psB = conn.prepareStatement("INSERT INTO connbidirect " - + "(source, statsend, seconds, belownum, readnum, writenum, " - + "bothnum) VALUES (?, ?, ?, ?, ?, ?, ?)"); - this.psQ = conn.prepareStatement("INSERT INTO dirreq_stats " - + "(source, statsend, seconds, country, requests) VALUES " - + "(?, ?, ?, ?, ?)"); - this.psU = conn.prepareStatement("INSERT INTO scheduled_updates " - + "(date) VALUES (?)"); - this.scheduledUpdates = new HashSet<Long>(); - this.importIntoDatabase = true; - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not connect to database or " - + "prepare statements.", e); - } - - /* Initialize set of fingerprints to remember which status entries - * we already imported. */ - this.insertedStatusEntries = new HashSet<String>(); - } - - /* Remember where we want to write raw import files. */ - if (rawFilesDirectory != null) { - this.rawFilesDirectory = rawFilesDirectory; - this.writeRawImportFiles = true; - } - - /* Initialize date format, so that we can format timestamps. */ - this.dateTimeFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); - this.dateTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); - } - - private void addDateToScheduledUpdates(long timestamp) - throws SQLException { - if (!this.importIntoDatabase) { - return; - } - long dateMillis = 0L; - try { - dateMillis = this.dateTimeFormat.parse( - this.dateTimeFormat.format(timestamp).substring(0, 10) - + " 00:00:00").getTime(); - } catch (ParseException e) { - this.logger.log(Level.WARNING, "Internal parsing error.", e); - return; - } - if (!this.scheduledUpdates.contains(dateMillis)) { - this.psU.setDate(1, new java.sql.Date(dateMillis)); - this.psU.execute(); - this.scheduledUpdates.add(dateMillis); - } - } - - /** - * Insert network status consensus entry into database. - */ - public void addStatusEntry(long validAfter, String nickname, - String fingerprint, String descriptor, long published, - String address, long orPort, long dirPort, - SortedSet<String> flags, String version, long bandwidth, - String ports, byte[] rawDescriptor) { - if (this.importIntoDatabase) { - try { - this.addDateToScheduledUpdates(validAfter); - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - Timestamp validAfterTimestamp = new Timestamp(validAfter); - if (lastCheckedStatusEntries != validAfter) { - this.psSs.setTimestamp(1, validAfterTimestamp, cal); - ResultSet rs = psSs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - separateStatusEntryCheckNecessary = false; - insertedStatusEntries.clear(); - } else { - separateStatusEntryCheckNecessary = true; - } - rs.close(); - lastCheckedStatusEntries = validAfter; - } - boolean alreadyContained = false; - if (separateStatusEntryCheckNecessary || - insertedStatusEntries.contains(fingerprint)) { - this.psRs.setTimestamp(1, validAfterTimestamp, cal); - this.psRs.setString(2, descriptor); - ResultSet rs = psRs.executeQuery(); - rs.next(); - if (rs.getInt(1) > 0) { - alreadyContained = true; - } - rs.close(); - } else { - insertedStatusEntries.add(fingerprint); - } - if (!alreadyContained) { - this.psR.clearParameters(); - this.psR.setTimestamp(1, validAfterTimestamp, cal); - this.psR.setString(2, nickname); - this.psR.setString(3, fingerprint); - this.psR.setString(4, descriptor); - this.psR.setTimestamp(5, new Timestamp(published), cal); - this.psR.setString(6, address); - this.psR.setLong(7, orPort); - this.psR.setLong(8, dirPort); - this.psR.setBoolean(9, flags.contains("Authority")); - this.psR.setBoolean(10, flags.contains("BadExit")); - this.psR.setBoolean(11, flags.contains("BadDirectory")); - this.psR.setBoolean(12, flags.contains("Exit")); - this.psR.setBoolean(13, flags.contains("Fast")); - this.psR.setBoolean(14, flags.contains("Guard")); - this.psR.setBoolean(15, flags.contains("HSDir")); - this.psR.setBoolean(16, flags.contains("Named")); - this.psR.setBoolean(17, flags.contains("Stable")); - this.psR.setBoolean(18, flags.contains("Running")); - this.psR.setBoolean(19, flags.contains("Unnamed")); - this.psR.setBoolean(20, flags.contains("Valid")); - this.psR.setBoolean(21, flags.contains("V2Dir")); - this.psR.setBoolean(22, flags.contains("V3Dir")); - this.psR.setString(23, version); - this.psR.setLong(24, bandwidth); - this.psR.setString(25, ports); - this.psR.setBytes(26, rawDescriptor); - this.psR.executeUpdate(); - rrsCount++; - if (rrsCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add network status " - + "consensus entry. We won't make any further SQL requests " - + "in this execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.statusentryOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.statusentryOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/statusentry.sql")); - this.statusentryOut.write(" COPY statusentry (validafter, " - + "nickname, fingerprint, descriptor, published, address, " - + "orport, dirport, isauthority, isbadExit, " - + "isbaddirectory, isexit, isfast, isguard, ishsdir, " - + "isnamed, isstable, isrunning, isunnamed, isvalid, " - + "isv2dir, isv3dir, version, bandwidth, ports, rawdesc) " - + "FROM stdin;\n"); - } - this.statusentryOut.write( - this.dateTimeFormat.format(validAfter) + "\t" + nickname - + "\t" + fingerprint.toLowerCase() + "\t" - + descriptor.toLowerCase() + "\t" - + this.dateTimeFormat.format(published) + "\t" + address - + "\t" + orPort + "\t" + dirPort + "\t" - + (flags.contains("Authority") ? "t" : "f") + "\t" - + (flags.contains("BadExit") ? "t" : "f") + "\t" - + (flags.contains("BadDirectory") ? "t" : "f") + "\t" - + (flags.contains("Exit") ? "t" : "f") + "\t" - + (flags.contains("Fast") ? "t" : "f") + "\t" - + (flags.contains("Guard") ? "t" : "f") + "\t" - + (flags.contains("HSDir") ? "t" : "f") + "\t" - + (flags.contains("Named") ? "t" : "f") + "\t" - + (flags.contains("Stable") ? "t" : "f") + "\t" - + (flags.contains("Running") ? "t" : "f") + "\t" - + (flags.contains("Unnamed") ? "t" : "f") + "\t" - + (flags.contains("Valid") ? "t" : "f") + "\t" - + (flags.contains("V2Dir") ? "t" : "f") + "\t" - + (flags.contains("V3Dir") ? "t" : "f") + "\t" - + (version != null ? version : "\\N") + "\t" - + (bandwidth >= 0 ? bandwidth : "\\N") + "\t" - + (ports != null ? ports : "\\N") + "\t"); - this.statusentryOut.write(PGbytea.toPGString(rawDescriptor). - replaceAll("\\\\", "\\\\\\\\") + "\n"); - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not write network status " - + "consensus entry to raw database import file. We won't " - + "make any further attempts to write raw import files in " - + "this execution.", e); - this.writeRawImportFiles = false; - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write network status " - + "consensus entry to raw database import file. We won't " - + "make any further attempts to write raw import files in " - + "this execution.", e); - this.writeRawImportFiles = false; - } - } - } - - /** - * Insert server descriptor into database. - */ - public void addServerDescriptor(String descriptor, String nickname, - String address, int orPort, int dirPort, String relayIdentifier, - long bandwidthAvg, long bandwidthBurst, long bandwidthObserved, - String platform, long published, long uptime, - String extraInfoDigest, byte[] rawDescriptor) { - if (this.importIntoDatabase) { - try { - this.addDateToScheduledUpdates(published); - this.addDateToScheduledUpdates( - published + 24L * 60L * 60L * 1000L); - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - this.psDs.setString(1, descriptor); - ResultSet rs = psDs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - this.psD.clearParameters(); - this.psD.setString(1, descriptor); - this.psD.setString(2, nickname); - this.psD.setString(3, address); - this.psD.setInt(4, orPort); - this.psD.setInt(5, dirPort); - this.psD.setString(6, relayIdentifier); - this.psD.setLong(7, bandwidthAvg); - this.psD.setLong(8, bandwidthBurst); - this.psD.setLong(9, bandwidthObserved); - this.psD.setString(10, new String(platform.getBytes(), - "US-ASCII")); - this.psD.setTimestamp(11, new Timestamp(published), cal); - this.psD.setLong(12, uptime); - this.psD.setString(13, extraInfoDigest); - this.psD.setBytes(14, rawDescriptor); - this.psD.executeUpdate(); - rdsCount++; - if (rdsCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } catch (UnsupportedEncodingException e) { - // US-ASCII is supported for sure - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add server " - + "descriptor. We won't make any further SQL requests in " - + "this execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.descriptorOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.descriptorOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/descriptor.sql")); - this.descriptorOut.write(" COPY descriptor (descriptor, " - + "nickname, address, orport, dirport, fingerprint, " - + "bandwidthavg, bandwidthburst, bandwidthobserved, " - + "platform, published, uptime, extrainfo, rawdesc) FROM " - + "stdin;\n"); - } - this.descriptorOut.write(descriptor.toLowerCase() + "\t" - + nickname + "\t" + address + "\t" + orPort + "\t" + dirPort - + "\t" + relayIdentifier + "\t" + bandwidthAvg + "\t" - + bandwidthBurst + "\t" + bandwidthObserved + "\t" - + (platform != null && platform.length() > 0 - ? new String(platform.getBytes(), "US-ASCII") : "\\N") - + "\t" + this.dateTimeFormat.format(published) + "\t" - + (uptime >= 0 ? uptime : "\\N") + "\t" - + (extraInfoDigest != null ? extraInfoDigest : "\\N") - + "\t"); - this.descriptorOut.write(PGbytea.toPGString(rawDescriptor). - replaceAll("\\\\", "\\\\\\\\") + "\n"); - } catch (UnsupportedEncodingException e) { - // US-ASCII is supported for sure - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not write server " - + "descriptor to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write server " - + "descriptor to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } - } - } - - /** - * Insert extra-info descriptor into database. - */ - public void addExtraInfoDescriptor(String extraInfoDigest, - String nickname, String fingerprint, long published, - byte[] rawDescriptor, List<String> bandwidthHistoryLines) { - if (this.importIntoDatabase) { - try { - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - this.psEs.setString(1, extraInfoDigest); - ResultSet rs = psEs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - this.psE.clearParameters(); - this.psE.setString(1, extraInfoDigest); - this.psE.setString(2, nickname); - this.psE.setString(3, fingerprint); - this.psE.setTimestamp(4, new Timestamp(published), cal); - this.psE.setBytes(5, rawDescriptor); - this.psE.executeUpdate(); - resCount++; - if (resCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add extra-info " - + "descriptor. We won't make any further SQL requests in " - + "this execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.extrainfoOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.extrainfoOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/extrainfo.sql")); - this.extrainfoOut.write(" COPY extrainfo (extrainfo, nickname, " - + "fingerprint, published, rawdesc) FROM stdin;\n"); - } - this.extrainfoOut.write(extraInfoDigest.toLowerCase() + "\t" - + nickname + "\t" + fingerprint.toLowerCase() + "\t" - + this.dateTimeFormat.format(published) + "\t"); - this.extrainfoOut.write(PGbytea.toPGString(rawDescriptor). - replaceAll("\\\\", "\\\\\\\\") + "\n"); - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write extra-info " - + "descriptor to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not write extra-info " - + "descriptor to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } - } - if (!bandwidthHistoryLines.isEmpty()) { - this.addBandwidthHistory(fingerprint.toLowerCase(), published, - bandwidthHistoryLines); - } - } - - private static class BigIntArray implements java.sql.Array { - - private final String stringValue; - - public BigIntArray(long[] array, int offset) { - if (array == null) { - this.stringValue = "[-1:-1]={0}"; - } else { - StringBuilder sb = new StringBuilder("[" + offset + ":" - + (offset + array.length - 1) + "]={"); - for (int i = 0; i < array.length; i++) { - sb.append((i > 0 ? "," : "") + array[i]); - } - sb.append('}'); - this.stringValue = sb.toString(); - } - } - - public String toString() { - return stringValue; - } - - public String getBaseTypeName() { - return "int8"; - } - - /* The other methods are never called; no need to implement them. */ - public void free() { - throw new UnsupportedOperationException(); - } - public Object getArray() { - throw new UnsupportedOperationException(); - } - public Object getArray(long index, int count) { - throw new UnsupportedOperationException(); - } - public Object getArray(long index, int count, - Map<String, Class<?>> map) { - throw new UnsupportedOperationException(); - } - public Object getArray(Map<String, Class<?>> map) { - throw new UnsupportedOperationException(); - } - public int getBaseType() { - throw new UnsupportedOperationException(); - } - public ResultSet getResultSet() { - throw new UnsupportedOperationException(); - } - public ResultSet getResultSet(long index, int count) { - throw new UnsupportedOperationException(); - } - public ResultSet getResultSet(long index, int count, - Map<String, Class<?>> map) { - throw new UnsupportedOperationException(); - } - public ResultSet getResultSet(Map<String, Class<?>> map) { - throw new UnsupportedOperationException(); - } - } - - public void addBandwidthHistory(String fingerprint, long published, - List<String> bandwidthHistoryStrings) { - - /* Split history lines by date and rewrite them so that the date - * comes first. */ - SortedSet<String> historyLinesByDate = new TreeSet<String>(); - for (String bandwidthHistoryString : bandwidthHistoryStrings) { - String[] parts = bandwidthHistoryString.split(" "); - if (parts.length != 6) { - this.logger.finer("Bandwidth history line does not have expected " - + "number of elements. Ignoring this line."); - continue; - } - long intervalLength = 0L; - try { - intervalLength = Long.parseLong(parts[3].substring(1)); - } catch (NumberFormatException e) { - this.logger.fine("Bandwidth history line does not have valid " - + "interval length '" + parts[3] + " " + parts[4] + "'. " - + "Ignoring this line."); - continue; - } - if (intervalLength != 900L) { - this.logger.fine("Bandwidth history line does not consist of " - + "15-minute intervals. Ignoring this line."); - continue; - } - String type = parts[0]; - String intervalEndTime = parts[1] + " " + parts[2]; - long intervalEnd, dateStart; - try { - intervalEnd = dateTimeFormat.parse(intervalEndTime).getTime(); - dateStart = dateTimeFormat.parse(parts[1] + " 00:00:00"). - getTime(); - } catch (ParseException e) { - this.logger.fine("Parse exception while parsing timestamp in " - + "bandwidth history line. Ignoring this line."); - continue; - } - if (Math.abs(published - intervalEnd) > - 7L * 24L * 60L * 60L * 1000L) { - this.logger.fine("Extra-info descriptor publication time " - + dateTimeFormat.format(published) + " and last interval " - + "time " + intervalEndTime + " in " + type + " line differ " - + "by more than 7 days! Not adding this line!"); - continue; - } - long currentIntervalEnd = intervalEnd; - StringBuilder sb = new StringBuilder(); - String[] values = parts[5].split(","); - SortedSet<String> newHistoryLines = new TreeSet<String>(); - try { - for (int i = values.length - 1; i >= -1; i--) { - if (i == -1 || currentIntervalEnd < dateStart) { - sb.insert(0, intervalEndTime + " " + type + " (" - + intervalLength + " s) "); - sb.setLength(sb.length() - 1); - String historyLine = sb.toString(); - newHistoryLines.add(historyLine); - sb = new StringBuilder(); - dateStart -= 24L * 60L * 60L * 1000L; - intervalEndTime = dateTimeFormat.format(currentIntervalEnd); - } - if (i == -1) { - break; - } - Long.parseLong(values[i]); - sb.insert(0, values[i] + ","); - currentIntervalEnd -= intervalLength * 1000L; - } - } catch (NumberFormatException e) { - this.logger.fine("Number format exception while parsing " - + "bandwidth history line. Ignoring this line."); - continue; - } - historyLinesByDate.addAll(newHistoryLines); - } - - /* Add split history lines to database. */ - String lastDate = null; - historyLinesByDate.add("EOL"); - long[] readArray = null, writtenArray = null, dirreadArray = null, - dirwrittenArray = null; - int readOffset = 0, writtenOffset = 0, dirreadOffset = 0, - dirwrittenOffset = 0; - for (String historyLine : historyLinesByDate) { - String[] parts = historyLine.split(" "); - String currentDate = parts[0]; - if (lastDate != null && (historyLine.equals("EOL") || - !currentDate.equals(lastDate))) { - BigIntArray readIntArray = new BigIntArray(readArray, - readOffset); - BigIntArray writtenIntArray = new BigIntArray(writtenArray, - writtenOffset); - BigIntArray dirreadIntArray = new BigIntArray(dirreadArray, - dirreadOffset); - BigIntArray dirwrittenIntArray = new BigIntArray(dirwrittenArray, - dirwrittenOffset); - if (this.importIntoDatabase) { - try { - long dateMillis = dateTimeFormat.parse(lastDate - + " 00:00:00").getTime(); - this.addDateToScheduledUpdates(dateMillis); - this.csH.setString(1, fingerprint); - this.csH.setDate(2, new java.sql.Date(dateMillis)); - this.csH.setArray(3, readIntArray); - this.csH.setArray(4, writtenIntArray); - this.csH.setArray(5, dirreadIntArray); - this.csH.setArray(6, dirwrittenIntArray); - this.csH.addBatch(); - rhsCount++; - if (rhsCount % autoCommitCount == 0) { - this.csH.executeBatch(); - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not insert bandwidth " - + "history line into database. We won't make any " - + "further SQL requests in this execution.", e); - this.importIntoDatabase = false; - } catch (ParseException e) { - this.logger.log(Level.WARNING, "Could not insert bandwidth " - + "history line into database. We won't make any " - + "further SQL requests in this execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.bwhistOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.bwhistOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/bwhist.sql")); - } - this.bwhistOut.write("SELECT insert_bwhist('" + fingerprint - + "','" + lastDate + "','" + readIntArray.toString() - + "','" + writtenIntArray.toString() + "','" - + dirreadIntArray.toString() + "','" - + dirwrittenIntArray.toString() + "');\n"); - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write bandwidth " - + "history to raw database import file. We won't make " - + "any further attempts to write raw import files in " - + "this execution.", e); - this.writeRawImportFiles = false; - } - } - readArray = writtenArray = dirreadArray = dirwrittenArray = null; - } - if (historyLine.equals("EOL")) { - break; - } - long lastIntervalTime; - try { - lastIntervalTime = dateTimeFormat.parse(parts[0] + " " - + parts[1]).getTime() - dateTimeFormat.parse(parts[0] - + " 00:00:00").getTime(); - } catch (ParseException e) { - continue; - } - String[] stringValues = parts[5].split(","); - long[] longValues = new long[stringValues.length]; - for (int i = 0; i < longValues.length; i++) { - longValues[i] = Long.parseLong(stringValues[i]); - } - - int offset = (int) (lastIntervalTime / (15L * 60L * 1000L)) - - longValues.length + 1; - String type = parts[2]; - if (type.equals("read-history")) { - readArray = longValues; - readOffset = offset; - } else if (type.equals("write-history")) { - writtenArray = longValues; - writtenOffset = offset; - } else if (type.equals("dirreq-read-history")) { - dirreadArray = longValues; - dirreadOffset = offset; - } else if (type.equals("dirreq-write-history")) { - dirwrittenArray = longValues; - dirwrittenOffset = offset; - } - lastDate = currentDate; - } - } - - /** - * Insert network status consensus into database. - */ - public void addConsensus(long validAfter, byte[] rawDescriptor) { - if (this.importIntoDatabase) { - try { - this.addDateToScheduledUpdates(validAfter); - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - Timestamp validAfterTimestamp = new Timestamp(validAfter); - this.psCs.setTimestamp(1, validAfterTimestamp, cal); - ResultSet rs = psCs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - this.psC.clearParameters(); - this.psC.setTimestamp(1, validAfterTimestamp, cal); - this.psC.setBytes(2, rawDescriptor); - this.psC.executeUpdate(); - rcsCount++; - if (rcsCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add network status " - + "consensus. We won't make any further SQL requests in " - + "this execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.consensusOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.consensusOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/consensus.sql")); - this.consensusOut.write(" COPY consensus (validafter, rawdesc) " - + "FROM stdin;\n"); - } - String validAfterString = this.dateTimeFormat.format(validAfter); - this.consensusOut.write(validAfterString + "\t"); - this.consensusOut.write(PGbytea.toPGString(rawDescriptor). - replaceAll("\\\\", "\\\\\\\\") + "\n"); - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not write network status " - + "consensus to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write network status " - + "consensus to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } - } - } - - /** - * Insert network status vote into database. - */ - public void addVote(long validAfter, String dirSource, - byte[] rawDescriptor) { - if (this.importIntoDatabase) { - try { - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - Timestamp validAfterTimestamp = new Timestamp(validAfter); - this.psVs.setTimestamp(1, validAfterTimestamp, cal); - this.psVs.setString(2, dirSource); - ResultSet rs = psVs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - this.psV.clearParameters(); - this.psV.setTimestamp(1, validAfterTimestamp, cal); - this.psV.setString(2, dirSource); - this.psV.setBytes(3, rawDescriptor); - this.psV.executeUpdate(); - rvsCount++; - if (rvsCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add network status " - + "vote. We won't make any further SQL requests in this " - + "execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.voteOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.voteOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/vote.sql")); - this.voteOut.write(" COPY vote (validafter, dirsource, " - + "rawdesc) FROM stdin;\n"); - } - String validAfterString = this.dateTimeFormat.format(validAfter); - this.voteOut.write(validAfterString + "\t" + dirSource + "\t"); - this.voteOut.write(PGbytea.toPGString(rawDescriptor). - replaceAll("\\\\", "\\\\\\\\") + "\n"); - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not write network status " - + "vote to raw database import file. We won't make any " - + "further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write network status " - + "vote to raw database import file. We won't make any " - + "further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } - } - } - - /** - * Insert a conn-bi-direct stats string into the database. - */ - public void addConnBiDirect(String source, String statsEnd, - long seconds, long below, long read, long write, long both) { - long statsEndTime = 0L; - try { - statsEndTime = this.dateTimeFormat.parse(statsEnd).getTime(); - } catch (ParseException e) { - this.logger.log(Level.WARNING, "Could not add conn-bi-direct " - + "stats string with interval ending '" + statsEnd + "'.", e); - return; - } - if (this.importIntoDatabase) { - try { - this.addDateToScheduledUpdates(statsEndTime); - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - Timestamp statsEndTimestamp = new Timestamp(statsEndTime); - this.psBs.setString(1, source); - this.psBs.setTimestamp(2, statsEndTimestamp, cal); - ResultSet rs = psBs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - this.psB.clearParameters(); - this.psB.setString(1, source); - this.psB.setTimestamp(2, statsEndTimestamp, cal); - this.psB.setLong(3, seconds); - this.psB.setLong(4, below); - this.psB.setLong(5, read); - this.psB.setLong(6, write); - this.psB.setLong(7, both); - this.psB.executeUpdate(); - rbsCount++; - if (rbsCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add conn-bi-direct " - + "stats string. We won't make any further SQL requests in " - + "this execution.", e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.connBiDirectOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.connBiDirectOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/connbidirect.sql")); - this.connBiDirectOut.write(" COPY connbidirect (source, " - + "statsend, seconds, belownum, readnum, writenum, " - + "bothnum) FROM stdin;\n"); - } - this.connBiDirectOut.write(source + "\t" + statsEnd + "\t" - + seconds + "\t" + below + "\t" + read + "\t" + write + "\t" - + both + "\n"); - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write conn-bi-direct " - + "stats string to raw database import file. We won't make " - + "any further attempts to write raw import files in this " - + "execution.", e); - this.writeRawImportFiles = false; - } - } - } - - /** - * Adds observations on the number of directory requests by country as - * seen on a directory at a given date to the database. - */ - public void addDirReqStats(String source, String statsEnd, long seconds, - Map<String, String> dirReqsPerCountry) { - long statsEndTime = 0L; - try { - statsEndTime = this.dateTimeFormat.parse(statsEnd).getTime(); - } catch (ParseException e) { - this.logger.log(Level.WARNING, "Could not add dirreq stats with " - + "interval ending '" + statsEnd + "'.", e); - return; - } - if (this.importIntoDatabase) { - try { - this.addDateToScheduledUpdates(statsEndTime); - Calendar cal = Calendar.getInstance(TimeZone.getTimeZone("UTC")); - Timestamp statsEndTimestamp = new Timestamp(statsEndTime); - this.psQs.setString(1, source); - this.psQs.setTimestamp(2, statsEndTimestamp, cal); - ResultSet rs = psQs.executeQuery(); - rs.next(); - if (rs.getInt(1) == 0) { - for (Map.Entry<String, String> e : - dirReqsPerCountry.entrySet()) { - this.psQ.clearParameters(); - this.psQ.setString(1, source); - this.psQ.setTimestamp(2, statsEndTimestamp, cal); - this.psQ.setLong(3, seconds); - this.psQ.setString(4, e.getKey()); - this.psQ.setLong(5, Long.parseLong(e.getValue())); - this.psQ.executeUpdate(); - rqsCount++; - if (rqsCount % autoCommitCount == 0) { - this.conn.commit(); - } - } - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add dirreq stats. We " - + "won't make any further SQL requests in this execution.", - e); - this.importIntoDatabase = false; - } - } - if (this.writeRawImportFiles) { - try { - if (this.dirReqOut == null) { - new File(rawFilesDirectory).mkdirs(); - this.dirReqOut = new BufferedWriter(new FileWriter( - rawFilesDirectory + "/dirreq_stats.sql")); - this.dirReqOut.write(" COPY dirreq_stats (source, statsend, " - + "seconds, country, requests) FROM stdin;\n"); - } - for (Map.Entry<String, String> e : - dirReqsPerCountry.entrySet()) { - this.dirReqOut.write(source + "\t" + statsEnd + "\t" + seconds - + "\t" + e.getKey() + "\t" + e.getValue() + "\n"); - } - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not write dirreq stats to " - + "raw database import file. We won't make any further " - + "attempts to write raw import files in this execution.", e); - this.writeRawImportFiles = false; - } - } - } - - /** - * Close the relay descriptor database connection. - */ - public void closeConnection() { - - /* Log stats about imported descriptors. */ - this.logger.info(String.format("Finished importing relay " - + "descriptors: %d consensuses, %d network status entries, %d " - + "votes, %d server descriptors, %d extra-info descriptors, %d " - + "bandwidth history elements, %d dirreq stats elements, and %d " - + "conn-bi-direct stats lines", rcsCount, rrsCount, rvsCount, - rdsCount, resCount, rhsCount, rqsCount, rbsCount)); - - /* Insert scheduled updates a second time, just in case the refresh - * run has started since inserting them the first time in which case - * it will miss the data inserted afterwards. We cannot, however, - * insert them only now, because if a Java execution fails at a random - * point, we might have added data, but not the corresponding dates to - * update statistics. */ - if (this.importIntoDatabase) { - try { - for (long dateMillis : this.scheduledUpdates) { - this.psU.setDate(1, new java.sql.Date(dateMillis)); - this.psU.execute(); - } - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not add scheduled dates " - + "for the next refresh run.", e); - } - } - - /* Commit any stragglers before closing. */ - if (this.conn != null) { - try { - this.csH.executeBatch(); - - this.conn.commit(); - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not commit final records to " - + "database", e); - } - try { - this.conn.close(); - } catch (SQLException e) { - this.logger.log(Level.WARNING, "Could not close database " - + "connection.", e); - } - } - - /* Close raw import files. */ - try { - if (this.statusentryOut != null) { - this.statusentryOut.write("\\.\n"); - this.statusentryOut.close(); - } - if (this.descriptorOut != null) { - this.descriptorOut.write("\\.\n"); - this.descriptorOut.close(); - } - if (this.extrainfoOut != null) { - this.extrainfoOut.write("\\.\n"); - this.extrainfoOut.close(); - } - if (this.bwhistOut != null) { - this.bwhistOut.write("\\.\n"); - this.bwhistOut.close(); - } - if (this.consensusOut != null) { - this.consensusOut.write("\\.\n"); - this.consensusOut.close(); - } - if (this.voteOut != null) { - this.voteOut.write("\\.\n"); - this.voteOut.close(); - } - if (this.connBiDirectOut != null) { - this.connBiDirectOut.write("\\.\n"); - this.connBiDirectOut.close(); - } - } catch (IOException e) { - this.logger.log(Level.WARNING, "Could not close one or more raw " - + "database import files.", e); - } - } -} - diff --git a/src/org/torproject/ernie/db/RelayDescriptorParser.java b/src/org/torproject/ernie/db/RelayDescriptorParser.java index e94b55b..fcfc0eb 100644 --- a/src/org/torproject/ernie/db/RelayDescriptorParser.java +++ b/src/org/torproject/ernie/db/RelayDescriptorParser.java @@ -35,12 +35,6 @@ public class RelayDescriptorParser { private RelayDescriptorDownloader rdd; /** - * Relay descriptor database importer that stores relay descriptor - * contents for later evaluation. - */ - private RelayDescriptorDatabaseImporter rddi; - - /** * Logger for this class. */ private Logger logger; @@ -51,10 +45,9 @@ public class RelayDescriptorParser { * Initializes this class. */ public RelayDescriptorParser(BridgeStatsFileHandler bsfh, - ArchiveWriter aw, RelayDescriptorDatabaseImporter rddi) { + ArchiveWriter aw) { this.bsfh = bsfh; this.aw = aw; - this.rddi = rddi; /* Initialize logger. */ this.logger = Logger.getLogger(RelayDescriptorParser.class.getName()); @@ -89,18 +82,12 @@ public class RelayDescriptorParser { // time to see when we switch from hourly to half-hourly // consensuses boolean isConsensus = true; - int exit = 0, fast = 0, guard = 0, running = 0, stable = 0; - String validAfterTime = null, nickname = null, - relayIdentity = null, serverDesc = null, version = null, - ports = null; - String fingerprint = null, dirSource = null, address = null; - long validAfter = -1L, published = -1L, bandwidth = -1L, - orPort = 0L, dirPort = 0L; + String validAfterTime = null, fingerprint = null, + dirSource = null; + long validAfter = -1L; SortedSet<String> dirSources = new TreeSet<String>(); SortedSet<String> serverDescriptors = new TreeSet<String>(); SortedSet<String> hashedRelayIdentities = new TreeSet<String>(); - SortedSet<String> relayFlags = null; - StringBuilder rawStatusEntry = null; while ((line = br.readLine()) != null) { if (line.equals("vote-status vote")) { isConsensus = false; @@ -114,19 +101,6 @@ public class RelayDescriptorParser { } else if (line.startsWith("fingerprint ")) { fingerprint = line.split(" ")[1]; } else if (line.startsWith("r ")) { - if (isConsensus && relayIdentity != null && - this.rddi != null) { - byte[] rawDescriptor = rawStatusEntry.toString().getBytes(); - this.rddi.addStatusEntry(validAfter, nickname, - relayIdentity, serverDesc, published, address, orPort, - dirPort, relayFlags, version, bandwidth, ports, - rawDescriptor); - relayFlags = null; - version = null; - bandwidth = -1L; - ports = null; - } - rawStatusEntry = new StringBuilder(line + "\n"); String[] parts = line.split(" "); if (parts.length < 9) { this.logger.log(Level.WARNING, "Could not parse r line '" @@ -134,65 +108,19 @@ public class RelayDescriptorParser { break; } String publishedTime = parts[4] + " " + parts[5]; - nickname = parts[1]; - relayIdentity = Hex.encodeHexString( + String relayIdentity = Hex.encodeHexString( Base64.decodeBase64(parts[2] + "=")). toLowerCase(); - serverDesc = Hex.encodeHexString(Base64.decodeBase64( + String serverDesc = Hex.encodeHexString(Base64.decodeBase64( parts[3] + "=")).toLowerCase(); serverDescriptors.add(publishedTime + "," + relayIdentity + "," + serverDesc); hashedRelayIdentities.add(DigestUtils.shaHex( Base64.decodeBase64(parts[2] + "=")). toUpperCase()); - published = parseFormat.parse(parts[4] + " " + parts[5]). - getTime(); - address = parts[6]; - orPort = Long.parseLong(parts[7]); - dirPort = Long.parseLong(parts[8]); - } else if (line.startsWith("s ") || line.equals("s")) { - rawStatusEntry.append(line + "\n"); - if (line.contains(" Running")) { - exit += line.contains(" Exit") ? 1 : 0; - fast += line.contains(" Fast") ? 1 : 0; - guard += line.contains(" Guard") ? 1 : 0; - stable += line.contains(" Stable") ? 1 : 0; - running++; - } - relayFlags = new TreeSet<String>(); - if (line.length() > 2) { - for (String flag : line.substring(2).split(" ")) { - relayFlags.add(flag); - } - } - } else if (line.startsWith("v ")) { - rawStatusEntry.append(line + "\n"); - version = line.substring(2); - } else if (line.startsWith("w ")) { - rawStatusEntry.append(line + "\n"); - String[] parts = line.split(" "); - for (String part : parts) { - if (part.startsWith("Bandwidth=")) { - bandwidth = Long.parseLong(part.substring( - "Bandwidth=".length())); - } - } - } else if (line.startsWith("p ")) { - rawStatusEntry.append(line + "\n"); - ports = line.substring(2); } } if (isConsensus) { - if (this.rddi != null) { - this.rddi.addConsensus(validAfter, data); - if (relayIdentity != null) { - byte[] rawDescriptor = rawStatusEntry.toString().getBytes(); - this.rddi.addStatusEntry(validAfter, nickname, - relayIdentity, serverDesc, published, address, orPort, - dirPort, relayFlags, version, bandwidth, ports, - rawDescriptor); - } - } if (this.bsfh != null) { for (String hashedRelayIdentity : hashedRelayIdentities) { this.bsfh.addHashedRelay(hashedRelayIdentity); @@ -206,9 +134,6 @@ public class RelayDescriptorParser { this.aw.storeConsensus(data, validAfter); } } else { - if (this.rddi != null) { - this.rddi.addVote(validAfter, dirSource, data); - } if (this.rdd != null) { this.rdd.haveParsedVote(validAfterTime, fingerprint, serverDescriptors); @@ -231,19 +156,16 @@ public class RelayDescriptorParser { } } } else if (line.startsWith("router ")) { - String platformLine = null, publishedLine = null, - publishedTime = null, bandwidthLine = null, + String publishedLine = null, publishedTime = null, extraInfoDigest = null, relayIdentifier = null; String[] parts = line.split(" "); String nickname = parts[1]; String address = parts[2]; int orPort = Integer.parseInt(parts[3]); int dirPort = Integer.parseInt(parts[4]); - long published = -1L, uptime = -1L; + long published = -1L; while ((line = br.readLine()) != null) { - if (line.startsWith("platform ")) { - platformLine = line; - } else if (line.startsWith("published ")) { + if (line.startsWith("published ")) { publishedTime = line.substring("published ".length()); published = parseFormat.parse(publishedTime).getTime(); } else if (line.startsWith("opt fingerprint") || @@ -251,15 +173,11 @@ public class RelayDescriptorParser { relayIdentifier = line.substring(line.startsWith("opt ") ? "opt fingerprint".length() : "fingerprint".length()). replaceAll(" ", "").toLowerCase(); - } else if (line.startsWith("bandwidth ")) { - bandwidthLine = line; } else if (line.startsWith("opt extra-info-digest ") || line.startsWith("extra-info-digest ")) { extraInfoDigest = line.startsWith("opt ") ? line.split(" ")[2].toLowerCase() : line.split(" ")[1].toLowerCase(); - } else if (line.startsWith("uptime ")) { - uptime = Long.parseLong(line.substring("uptime ".length())); } } String ascii = new String(data, "US-ASCII"); @@ -280,93 +198,14 @@ public class RelayDescriptorParser { this.rdd.haveParsedServerDescriptor(publishedTime, relayIdentifier, digest, extraInfoDigest); } - if (this.rddi != null && digest != null) { - String[] bwParts = bandwidthLine.split(" "); - long bandwidthAvg = Long.parseLong(bwParts[1]); - long bandwidthBurst = Long.parseLong(bwParts[2]); - long bandwidthObserved = Long.parseLong(bwParts[3]); - String platform = platformLine.substring("platform ".length()); - this.rddi.addServerDescriptor(digest, nickname, address, orPort, - dirPort, relayIdentifier, bandwidthAvg, bandwidthBurst, - bandwidthObserved, platform, published, uptime, - extraInfoDigest, data); - } } else if (line.startsWith("extra-info ")) { - String nickname = line.split(" ")[1]; String publishedTime = null, relayIdentifier = line.split(" ")[2]; long published = -1L; - String dir = line.split(" ")[2]; - String statsEnd = null; - long seconds = -1L; - List<String> bandwidthHistory = new ArrayList<String>(); boolean skip = false; while ((line = br.readLine()) != null) { if (line.startsWith("published ")) { publishedTime = line.substring("published ".length()); published = parseFormat.parse(publishedTime).getTime(); - } else if (line.startsWith("read-history ") || - line.startsWith("write-history ") || - line.startsWith("dirreq-read-history ") || - line.startsWith("dirreq-write-history ")) { - bandwidthHistory.add(line); - } else if (line.startsWith("dirreq-stats-end ")) { - String[] parts = line.split(" "); - if (parts.length < 5) { - this.logger.warning("Could not parse dirreq-stats-end " - + "line '" + line + "' in descriptor. Skipping."); - break; - } - statsEnd = parts[1] + " " + parts[2]; - seconds = Long.parseLong(parts[3].substring(1)); - } else if (line.startsWith("dirreq-v3-reqs ") - && line.length() > "dirreq-v3-reqs ".length()) { - if (this.rddi != null) { - try { - int allUsers = 0; - Map<String, String> obs = new HashMap<String, String>(); - String[] parts = line.substring("dirreq-v3-reqs ". - length()).split(","); - for (String p : parts) { - String country = p.substring(0, 2); - int users = Integer.parseInt(p.substring(3)) - 4; - allUsers += users; - obs.put(country, "" + users); - } - obs.put("zy", "" + allUsers); - this.rddi.addDirReqStats(dir, statsEnd, seconds, obs); - } catch (NumberFormatException e) { - this.logger.log(Level.WARNING, "Could not parse " - + "dirreq-v3-reqs line '" + line + "' in descriptor. " - + "Skipping.", e); - break; - } - } - } else if (line.startsWith("conn-bi-direct ")) { - if (this.rddi != null) { - String[] parts = line.split(" "); - if (parts.length == 6 && - parts[5].split(",").length == 4) { - try { - String connBiDirectStatsEnd = parts[1] + " " + parts[2]; - long connBiDirectSeconds = Long.parseLong(parts[3]. - substring(1)); - String[] parts2 = parts[5].split(","); - long below = Long.parseLong(parts2[0]); - long read = Long.parseLong(parts2[1]); - long write = Long.parseLong(parts2[2]); - long both = Long.parseLong(parts2[3]); - this.rddi.addConnBiDirect(dir, connBiDirectStatsEnd, - connBiDirectSeconds, below, read, write, both); - } catch (NumberFormatException e) { - this.logger.log(Level.WARNING, "Number format " - + "exception while parsing conn-bi-direct stats " - + "string '" + line + "'. Skipping.", e); - } - } else { - this.logger.warning("Skipping invalid conn-bi-direct " - + "stats string '" + line + "'."); - } - } } } String ascii = new String(data, "US-ASCII"); @@ -387,10 +226,6 @@ public class RelayDescriptorParser { this.rdd.haveParsedExtraInfoDescriptor(publishedTime, relayIdentifier.toLowerCase(), digest); } - if (this.rddi != null && digest != null) { - this.rddi.addExtraInfoDescriptor(digest, nickname, - dir.toLowerCase(), published, data, bandwidthHistory); - } } } catch (IOException e) { this.logger.log(Level.WARNING, "Could not parse descriptor. " @@ -401,3 +236,4 @@ public class RelayDescriptorParser { } } } +

1 0

r24286: {projects} translation xml files for orbot (in projects/android/trunk/Orbot: . res res/values-ar res/values-ca res/values-de res/values-es res/values-fa res/values-nl res/values-pl res/values-ru res/values-zh)
by Runa Sandvik 02 Mar '11

02 Mar '11

Author: runa Date: 2011-03-02 15:26:23 +0000 (Wed, 02 Mar 2011) New Revision: 24286 Added: projects/android/trunk/Orbot/res/values-ca/ projects/android/trunk/Orbot/res/values-ca/strings.xml Modified: projects/android/trunk/Orbot/po2xml.sh projects/android/trunk/Orbot/res/values-ar/strings.xml projects/android/trunk/Orbot/res/values-de/strings.xml projects/android/trunk/Orbot/res/values-es/strings.xml projects/android/trunk/Orbot/res/values-fa/strings.xml projects/android/trunk/Orbot/res/values-nl/strings.xml projects/android/trunk/Orbot/res/values-pl/strings.xml projects/android/trunk/Orbot/res/values-ru/strings.xml projects/android/trunk/Orbot/res/values-zh/strings.xml Log: translation xml files for orbot Modified: projects/android/trunk/Orbot/po2xml.sh =================================================================== --- projects/android/trunk/Orbot/po2xml.sh 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/po2xml.sh 2011-03-02 15:26:23 UTC (rev 24286) @@ -15,12 +15,12 @@ # Location of the translated files, i.e. the path to the orbot # directory in the translation module. Do not add the trailing slash. -translated="" +translated="/home/runa/tor/translation/projects/orbot" # Location of the orbot directory, i.e. the original English xml file. # In svn, this should be svn/projects/android/trunk/Orbot/res. Do not add the # trailing slash. -xml="" +xml="/home/runa/tor/orbot/res" ### End config ### @@ -37,7 +37,8 @@ xmlfile="${pofile%.*}.xml" # Figure out which language we are dealing with. - lang=`dirname $file | sed "s#$translated/##"` + dir=`dirname $file | sed "s#$translated/##"` + lang=`basename $dir` # The translated document is written if 80% or more of the po # file has been translated. Also, po4a-translate will only write Modified: projects/android/trunk/Orbot/res/values-ar/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-ar/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-ar/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -50,6 +50,13 @@ <string name="pref_transparent_all_title">تور كل شيء</string> <string name="pref_transparent_all_summary">حركة مرور الوكيل لجميع التطبيقات عبر تور</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">تم تثبيت ثنائيات تور بنجاح!</string> <string name="status_install_fail">غير قادر على تثبيت ملفات ثنائيات تور. يرجى التحقق من السجل وإعلامنا على البريد الإلكتروني tor-assistants(a)torproject.org</string> Added: projects/android/trunk/Orbot/res/values-ca/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-ca/strings.xml (rev 0) +++ projects/android/trunk/Orbot/res/values-ca/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -0,0 +1,137 @@ +<?xml version="1.0" encoding="utf-8"?> +<resources> + <string name="app_name">Orbot</string> + <string name="app_version">1.0.4</string> + + <string name="internal_web_url">http://orbot/</string> + <string name="default_web_url">http://check.torproject.org</string> + <string name="secure_default_web_url">https://check.torproject.org</string> + <string name="tor_check_api_url">https://check.torproject.org/?TorButton=true</string> + <string name="control_permission_label">Engega o atura Tor</string> + <string name="tor_proxy_service_process">torproxyservice</string> + +<string name="status_starting_up">Orbot s\'està iniciant...</string> +<string name="status_activated">Connectat a la xarxa Tor</string> +<string name="status_disabled">Orbot està desactivat</string> +<string name="status_shutting_down">Orbot s\'està aturant</string> + +<string name="tor_process_connecting">Iniciant Tor...</string> +<string name="tor_process_connecting_step2">Preparant el control...</string> +<string name="tor_process_connecting_step3">acomplert.</string> +<string name="tor_process_connecting_step4">esperant.</string> + +<string name="not_anonymous_yet">ATENCIÓ : El trànsit de la teva connexió ja no es anònim! Siusplau, configura les teves aplicacions per a usar HTTP proxy 127.0.0.1:8118, per SOCKS4A o SOCKS5 proxy 127.0.0.1:9050</string> +<string name="menu_home">Lloc d\'inici</string> +<string name="menu_browse">Navega</string> +<string name="menu_settings">Preferències</string> +<string name="menu_log">Registre</string> +<string name="menu_info">Ajut</string> +<string name="menu_apps">Aplicacions</string> +<string name="menu_start">Inici</string> +<string name="menu_stop">Atura</string> + + +<string name="button_help">Ajut</string> +<string name="button_close">Tanca</string> +<string name="button_about">Quant a...</string> + +<string name="button_clear_log">Esborra el registre</string> + + +<string name="menu_verify">Prova</string> +<string name="menu_exit">Surt</string> +<string name="powered_by">Funciona gràcies al Projecte Tor</string> +<string name="press_to_start">- prem per a iniciar -</string> + +<string name="pref_trans_proxy_group">Utilitza un proxy transparent (requereix accés)</string> +<string name="pref_trans_proxy_title">Utilitza un proxy transparent</string> +<string name="pref_trans_proxy_summary">Automatitza Tor per a les aplicacions</string> + +<string name="pref_transparent_all_title">Automatitza Tor per a tot</string> +<string name="pref_transparent_all_summary">Canalitza tot el trànsit de proxy de les aplicacions per Tor</string> + +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + +<string name="status_install_success">Binaris de Tor instal·lats correctament!</string> +<string name="status_install_fail">Els fitxers binaris de Tor no s\'han pogut instalar. Per favor, comprova el registre i notifica-ho a tor-assistants(a)torproject.org</string> + +<string name="title_error">Error de l\'aplicació</string> + +<string name="wizard_title">Benvingut/da a Orbot</string> + +<string name="wizard_btn_tell_me_more">Quant a Orbot...</string> +<string name="btn_next">Següent</string> +<string name="btn_back">Enrere</string> +<string name="btn_finish">Fi</string> + +<string name="btn_okay">D\'acord</string> +<string name="btn_cancel">Cancel·la</string> + + + + + <string name="wizard_welcome_msg">Orbot et porta Tor a l\'Android. Tor és programari lliure i una xarxa oberta que t\'ajuda a defensar-te d\'una vigilància a través de la xarxa, amenaçant la teva privacitat i llibertat personal, informació confidencial, activitats i relacions, i seguretat estatal coneguda com anàlisi de trànsit. + +*ALERTA:* Instal·lar Orbot _no_ convertirà la teva connexió en anònima instantàniament! Aquest assistent t\'ajudarà a començar.</string> + <string name="wizard_details">Alguns detalls d\'Orbot...</string> + <string name="wizard_details_msg">Orbot és una aplicació de codi lliure que conté Tor, LibEvent i Privoxy. Proporciona un Proxy HTTP local (8118) i un proxy SOCKS (9050) a la xarxa Tor. Orbot també té capacitat d\'enviar tot el trànsit de dades d\'internet a través de Tor.</string> + <string name="wizard_permissions_root">Permís concedit</string> + <string name="wizard_permissions_stock">Permisos d\'Orbot</string> + <string name="wizard_premissions_msg_root">Perfecte! Hem detectat que tens els permisos matriu activats per a Orbot. Utilitzarem aquest poder sàviament.</string> + <string name="wizard_permissions_msg_stock">Encara que no és imprescindible, Orbot pot esdevenir una eina molt més potent si el teu terminal té accés complet. Utilitza el botó d\'aquí sota per conferir superpoders a Orbot!</string> + + <string name="wizard_permissions_no_root">Si no tens accés complet, o no tens ni idea del què t\'estem parlant, simplement assegura\'t d\'utilitzar aplicacions fetes per a funcionar amb Orbot.</string> + <string name="wizard_permissions_consent">Ho entenc i vull continuar sense accés complet.</string> + + <string name="wizard_permission_enable_root">Concedeix accés complet a Orbot</string> + <string name="wizard_configure">Configura la connexió a Tor</string> + <string name="wizard_configure_msg">Orbot t\'ofereix l\'opció de conduir el trànsit de totes les aplicacions per Tor o bé escollir quines aplicacions vols utilitzar amb Tor.</string> + <string name="wizard_configure_all">Connecta totes les aplicacions per Proxy per Tor.</string> + <string name="wizard_configure_select_apps">Selecciona aplicacions individuals per a funcionar amb Tor</string> + + + <string name="wizard_tips_tricks">Aplicacions per a usar-se amb Orbot</string> + <string name="wizard_tips_msg">T\'animem a descarregar-te & utilitza aplicacions que sàpiguen com connectar-se directament a Orbot. Fes clic als botons de sota per instal·lar.</string> + <string name="wizard_tips_otrchat">OTRCHAT - Client de missatgeria instantània segura per a Android</string> + <string name="wizard_tips_orweb">ORWEB (Només Android 1.x) - Navegador dissenyat per la privacitat & per a Orbot</string> + <string name="wizard_tips_proxy">Coniguració de Proxy - Aprèn a configurar aplicacions per a què funcionin amb Orbot</string> + + <string name="wizard_proxy_help_info">Configuració de Proxy</string> + <string name="wizard_proxy_help_msg">Si l\'aplicació d\'Android que estàs utilitzant admet l\'ús d\'un proxy HTTP o SOCKS, pots configurar-la per a connectar-se a Orbot i utilitzar Tor. + + +La configuració de l\'amfitrió és 127.0.0.1 o "localhost". Per a HTTP, el port actiu és 8118. Per a SOCKS, el proxy és 9050. Hauries d\'utlitzar SOCKS4A o SOCKS5 si és possible. + + + +Pots aprendre més coses sobre l\'ús dels proxys a Android al FAQ de: http://tinyurl.com/proxyandroid +</string> + + <string name="wizard_final">Orbot és a punt!</string> + <string name="wizard_final_msg">Centenars de milers de persones arreu del món utilitzen Tor per un gran ventall de raons: periodistes i blocaires, treballadors pels drets humans, agents de l\'autoritat, soldats, corporacions, ciutadans de règims opressors, i ciutadans qualssevol... I ara tu també ho pots fer!</string> + + <string name="otrchat_apk_url">https://guardianproject.info/getgibber</string> + <string name="orweb_apk_url">https://guardianproject.info/getorweb</string> + + + + + <string name="connect_first_time">T\'has connectat correctament a la xarxa Tor, però això NO vol dir que el teu dispositiu sigui segur. Pots utilitzar la opció \'Comprova\' del menú per provar el teu navegador. + +Visita\'ns a https://guardianproject.info/apps/orbot o envia\'ns un correu electrònic a help(a)guardianproject.info per a saber-ne més.</string> + + <string name="tor_check">Això obrirà el teu navegador web per defecte i es connectarà a https://check.torproject.org per tal de comprovar si Orbot està configurat correctament i estàs connectat/da a Tor.</string> + + + <string name="pref_hs_group">Serveis ocults</string> + + <string name="pref_general_group">General</string> + <string name="pref_start_boot_title">Start Orbot on Boot</string> + <string name="pref_start_boot_summary">Automatically start Orbot and connect Tor when your Android device boots</string> + +</resources> Modified: projects/android/trunk/Orbot/res/values-de/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-de/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-de/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -1,6 +1,5 @@ <?xml version="1.0" encoding="utf-8"?> <resources> - <string name="pref_hs_group">Versteckte Dienste</string> <string name="app_name">Orbot</string> <string name="app_version">1.0.4</string> @@ -36,7 +35,7 @@ <string name="button_close">Schließen</string> <string name="button_about">Über</string> -<string name="button_clear_log">Clear Log</string> +<string name="button_clear_log">Log löschen</string> <string name="menu_verify">Überprüfen</string> @@ -44,15 +43,22 @@ <string name="powered_by">ermöglicht durch das Tor Projekt</string> <string name="press_to_start">- zum Starten drücken -</string> -<string name="pref_trans_proxy_group">Transparenter Proxy (nur gerootete Geräte)</string> +<string name="pref_trans_proxy_group">Transparenter Proxy (benötigt Administratorrechte)</string> <string name="pref_trans_proxy_title">Transparenter Proxy</string> <string name="pref_trans_proxy_summary">Anwendungen automatisch durch Tor leiten</string> <string name="pref_transparent_all_title">Alles durch Tor leiten</string> <string name="pref_transparent_all_summary">Verbindungen aller Anwendungen durch Tor leiten</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">Die Tor-Pakete wurden erfolgreich installiert!</string> -<string name="status_install_fail">Die Tor Pakete konnten nicht erfolgreich installiert werden. Bitte prüfe das Logfile und wende Dich an tor-assistants(a)torproject.org.</string> +<string name="status_install_fail">Die Tor-Pakete konnten nicht erfolgreich installiert werden. Bitte prüfen Sie das Logfile und wenden Sie sich an tor-assistants(a)torproject.org.</string> <string name="title_error">Anwendungsfehler</string> @@ -73,51 +79,59 @@ *WARNUNG:* Durch die Installation von Orbot wird _nicht_ sofort Ihr kompletter Datenverkehr anonymisiert! Hilfestellung erhalten Sie von diesem Assitenten.</string> <string name="wizard_details">Nähere Informationen zu Orbot</string> - <string name="wizard_details_msg">Orbot is an open-source application that contains Tor, LibEvent and Privoxy. It provides a local HTTP proxy (8118) and a SOCKS proxy (9050) into the Tor network. Orbot also has the ability, on rooted device, to send all internet traffic through Tor.</string> + <string name="wizard_details_msg">Orbot ist eine quelloffene Anwendung die Tor, LibEvent und Privoxy enthält. Es stellt einen lokalen HTTP Proxy (8118) und einen SOCKS Proxy (9050) zum Tor-Netzwerk zur Verfügung. Orbot hat auf einem System mit Administrationsrechten auch die Fähigkeit, jeglichen Internetverkehr über das Tor-Netzwerk abzuwickeln.</string> <string name="wizard_permissions_root">Erlaubnis erteilt</string> - <string name="wizard_permissions_stock">Orbot Permissions</string> - <string name="wizard_premissions_msg_root">Exzellent! Wir haben festgestellt, dass sie Orbot Root-Rechte eingeräumt haben. Wir werden diese Macht weise nutzen.</string> - <string name="wizard_permissions_msg_stock">Obwohl es nicht nötig ist, kann Orbot ein mächtigeres Tool werden, wenn sie ihn Root-Rechte einräumen. Drücken sie auf den Kopf unten um Orbot diese Superkräfte einzuräumen.</string> + <string name="wizard_permissions_stock">Orbot Rechte</string> + <string name="wizard_premissions_msg_root">Exzellent! Wir haben festgestellt, dass Sie Orbot Administrator-Rechte eingeräumt haben. Wir werden diese Macht weise nutzen.</string> + <string name="wizard_permissions_msg_stock">Obwohl es nicht nötig ist, kann Orbot ein mächtigeres Tool werden, wenn Sie ihm Administrator-Rechte einräumen. Drücken Sie auf den Kopf unten um Orbot diese Superkräfte einzuräumen.</string> - <string name="wizard_permissions_no_root">If you don\'t have root access or have no idea what we\'re talking about, just be sure to use apps made to work with Orbot.</string> - <string name="wizard_permissions_consent">I understand and would like to continue without root</string> + <string name="wizard_permissions_no_root">Falls Sie keinen Administrator-Zugang haben oder nicht wissen was das ist, stellen Sie sicher, dass Sie Anwendungen benutzen die mit Orbot kompatibel können.</string> + <string name="wizard_permissions_consent">Ich habe verstanden und möchte ohne Administrator-Rechte weitermachen.</string> - <string name="wizard_permission_enable_root">Orbot Root-Rechte einräumen</string> + <string name="wizard_permission_enable_root">Orbot Administrator-Rechte einräumen</string> <string name="wizard_configure">Configure Torification</string> - <string name="wizard_configure_msg">Orbot gibt ihnen die Wahl den Datenverkehr aller Programme durch Tor zu leiten ODER die Wahl für alle Programme einzeln zu treffen.</string> + <string name="wizard_configure_msg">Orbot gibt Ihnen die Wahl den Datenverkehr aller Programme durch Tor zu leiten ODER die Wahl für alle Programme einzeln zu treffen.</string> <string name="wizard_configure_all">Den Verkehr aller Programme durch Tor leiten</string> <string name="wizard_configure_select_apps">Programme einzeln wählen</string> <string name="wizard_tips_tricks">Programme, für die Orbot aktiviert ist</string> - <string name="wizard_tips_msg">Wir raten ihnen Programme herunterzuladen & zu nutzen, die wissen, wie sie sich direkt mit Orbot verbinden. Klicken sie zum Installieren auf den Knopf unten.</string> + <string name="wizard_tips_msg">Wir raten Ihnen Programme herunterzuladen & zu nutzen, die wissen, wie sie sich direkt mit Orbot verbinden. Klicken Sie zum Installieren auf den Knopf unten.</string> <string name="wizard_tips_otrchat">OTRCHAT - Ein sicheres Instant-Messaging-Programm für Android</string> <string name="wizard_tips_orweb">ORWEB (Nur Android 1.x) - Ein für Privatsphäre & Orbot entworfener Browser</string> - <string name="wizard_tips_proxy">Proxy Settings - Learn how to configure apps to work with Orbot</string> + <string name="wizard_tips_proxy">Proxy-Einstellungen - Lernen Sie Anwendungen so zu konfigurieren, dass sie mit Orbot zusammenarbeiten.</string> - <string name="wizard_proxy_help_info">Proxy Settings</string> - <string name="wizard_proxy_help_msg">Wenn das Android-Programm, das du benutzt, die Verwendung von HTTP- oder SOCKS-Proxys unterstützt kannst du es konfigurieren sich mit Orbot zu verbinden und Tor zu nutzen. + <string name="wizard_proxy_help_info">Proxy-Einstellungen</string> + <string name="wizard_proxy_help_msg">Wenn das Android-Programm, das Sie benutzten die Verwendung von HTTP- oder SOCKS-Proxys unterstützt, können Sie es konfigurieren sich mit Orbot zu verbinden und Tor zu nutzen. Die Host-Einstellungen sind 127.0.0.1 oder "localhost". Die Port-Einstellungen sind 8118 für HTTP und 9050 für SOCKS. Sie sollten versuchen SOCKS4A oder SOCKS5 zu benutzen. -Erfahren sie mehr über die Verwendung von Proxys unter Android im FAQ unter http://tinyurl.com/proxyandroid -</string> +Erfahren Sie mehr über die Verwendung von Proxys unter Android in den FAQ unter http://tinyurl.com/proxyandroid + </string> - <string name="wizard_final">Orbot is ready!</string> - <string name="wizard_final_msg">Hunderttausende Menschen auf der ganzen Welt nutzen Tor aus einer Vielzahl von Gründen: Journalisten und Blogger, Menschenrechtsaktivisten, Strafverfolgungsbehörden, Soldaten, Unternehmen, Bürger repressiver Regime und ganz normale Menschen... und sind sie ebenfalls bereit!</string> + <string name="wizard_final">Orbot ist bereit!</string> + <string name="wizard_final_msg">Hunderttausende Menschen auf der ganzen Welt nutzen Tor aus einer Vielzahl von Gründen: Journalisten und Blogger, Menschenrechtsaktivisten, Strafverfolgungsbehörden, Soldaten, Unternehmen, Bürger repressiver Regime und ganz normale Menschen... und sind Sie ebenfalls bereit!</string> - <string name="otrchat_apk_url">https://guardianproject/getgibber</string> - <string name="orweb_apk_url">https://guardianproject/getorweb</string> + <string name="otrchat_apk_url">https://guardianproject.info/getgibber</string> + <string name="orweb_apk_url">https://guardianproject.info/getorweb</string>  - <string name="connect_first_time">Sie haben sich erfolgreich mit dem Tor-Netzwerk verbunden - das bedeutet aber NICHT, dass dein Gerät sicher ist. Du kannst die \'Überprüfen\'-Option aus dem Menü benutzen, um deinen Browser zu testen. + <string name="connect_first_time">Sie haben sich erfolgreich mit dem Tor-Netzwerk verbunden - das bedeutet aber NICHT, dass Ihr Gerät sicher ist. Sie können die \'Überprüfen\'-Option aus dem Menü benutzen, um Ihren Browser zu testen. -Besuchen sie https://guardianproject.info/apps/orbot oder senden sie eine E-Mail an help(a)guardianproject.info um mehr zu erfahren.</string> +Besuchen Sie https://guardianproject.info/apps/orbot oder senden Sie eine E-Mail an help(a)guardianproject.info um mehr zu erfahren.</string> - <string name="tor_check">This will open your default web browser to https://check.torproject.org in order to see if Orbot is probably configured and you are connected to Tor.</string> + <string name="tor_check">Das wird die Seite https://check.torproject.org in Ihrem Webbrowser öffnen, so dass Sie überprüfen können, ob Orbot richtig configuriert ist und Sie mit Tor verbunden sind.</string> + + + <string name="pref_hs_group">Versteckte Dienste</string> + + <string name="pref_general_group">General</string> + <string name="pref_start_boot_title">Start Orbot on Boot</string> + <string name="pref_start_boot_summary">Automatically start Orbot and connect Tor when your Android device boots</string> + </resources> Modified: projects/android/trunk/Orbot/res/values-es/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-es/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-es/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -50,6 +50,13 @@ <string name="pref_transparent_all_title">Pasar todo el tráfico por Tor</string> <string name="pref_transparent_all_summary">Tráfico Proxy para todas las aplicaciones mediante Tor</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">¡Binarios de Tor instalados con éxito!</string> <string name="status_install_fail">Los archivos binarios de Tor no se han podido instalar. Por favor, verifique el Historial y notifique a: tor-assistants(a)torproject.org</string> Modified: projects/android/trunk/Orbot/res/values-fa/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-fa/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-fa/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -50,6 +50,13 @@ <string name="pref_transparent_all_title">شامل همه</string> <string name="pref_transparent_all_summary">فعالیت همه اپلیکیشن ها را از طریق Tor پروکسی کنید.</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">دوتایی های Tor با موفقیت نصب شدند!</string> <string name="status_install_fail">نصب فایل های دوتایی Tor ممکن نبود. لطفن قطعه را وارسی کنید و به tor-assistance(a)torproject.org اطلاع دهید.</string> Modified: projects/android/trunk/Orbot/res/values-nl/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-nl/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-nl/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -50,6 +50,13 @@ <string name="pref_transparent_all_title">Tor Alles</string> <string name="pref_transparent_all_summary">Proxy verkeer voor alle toepasingen door Tor</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">Tor\'s binaire bestanden succesvol geïnstalleerd</string> <string name="status_install_fail">The binaire bestanden konden niet worden geïnstalleerd. Gelieve het log te raadplegen en tor-assistants(a)torproject.org op de hoogte te stellen</string> Modified: projects/android/trunk/Orbot/res/values-pl/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-pl/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-pl/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -50,6 +50,13 @@ <string name="pref_transparent_all_title">Przepuść wszystko przez Tora</string> <string name="pref_transparent_all_summary">Przekierowanie ruchu wszystkich aplikacji przez Tora</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">Binarki Tora pomyslnie zainstalowane!</string> <string name="status_install_fail">Nie można zainstalować plików binarnych Tora. Zajrzyj do dziennika i powiadom tor-assistants(a)torproject.org</string> Modified: projects/android/trunk/Orbot/res/values-ru/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-ru/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-ru/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -50,6 +50,13 @@ <string name="pref_transparent_all_title">Направлять все через Tor</string> <string name="pref_transparent_all_summary">Трафик всех приложений будет проходить через Tor</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> + +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + <string name="status_install_success">Программа Tor успешно установлена!</string> <string name="status_install_fail">Не удалось установить программу Tor. Пожалуйста, проверьте системный журнал и сообщите нам: tor-assistants(a)torproject.org</string> Modified: projects/android/trunk/Orbot/res/values-zh/strings.xml =================================================================== --- projects/android/trunk/Orbot/res/values-zh/strings.xml 2011-03-01 17:06:10 UTC (rev 24285) +++ projects/android/trunk/Orbot/res/values-zh/strings.xml 2011-03-02 15:26:23 UTC (rev 24286) @@ -7,20 +7,20 @@ <string name="default_web_url">http://check.torproject.org</string> <string name="secure_default_web_url">https://check.torproject.org</string> <string name="tor_check_api_url">https://check.torproject.org/?TorButton=true</string> - <string name="control_permission_label">启动和停止Tor</string> - <string name="tor_proxy_service_process">Tor代理服务</string> + <string name="control_permission_label">启动和停止 Tor</string> + <string name="tor_proxy_service_process">Tor 代理服务</string> -<string name="status_starting_up">Orbot正在启动...</string> -<string name="status_activated">已经连接上了Tor网络</string> -<string name="status_disabled">Orbot已停用</string> -<string name="status_shutting_down">Orbot正在关闭</string> +<string name="status_starting_up">Orbot 正在启动...</string> +<string name="status_activated">已连接到 Tor 网络</string> +<string name="status_disabled">Orbot 已停用</string> +<string name="status_shutting_down">Orbot 正在关闭</string> <string name="tor_process_connecting">正在启动Tor....</string> -<string name="tor_process_connecting_step2">启动控制连接...</string> -<string name="tor_process_connecting_step3">完毕.</string> -<string name="tor_process_connecting_step4">请等待...</string> +<string name="tor_process_connecting_step2">设置控制...</string> +<string name="tor_process_connecting_step3">完毕。</string> +<string name="tor_process_connecting_step4">等待中。</string> -<string name="not_anonymous_yet">警告: 您的通讯还不是匿名的! 请设置您的应用程序使用代理服务器127.0.0.1:8118或者SOCKS4A或者SOCKS5代理服务器127.0.0.1:9050</string> +<string name="not_anonymous_yet">警告: 您的通讯还不是匿名的! 请设置您的应用程序使用代理服务器 127.0.0.1:8118 或者 SOCKS4A 或者 SOCKS5 代理服务器 127.0.0.1:9050</string> <string name="menu_home">首页</string> <string name="menu_browse">浏览</string> <string name="menu_settings">设置</string> @@ -35,31 +35,38 @@ <string name="button_close">关闭</string> <string name="button_about">关于</string> -<string name="button_clear_log">清除日志</string> +<string name="button_clear_log">Clear Log</string> -<string name="menu_verify">验证</string> +<string name="menu_verify">检查</string> <string name="menu_exit">退出</string> -<string name="powered_by">由 Tor 项目提供动力</string> -<string name="press_to_start">- 按下后启动 -</string> +<string name="powered_by">由 Tor 项目强力驱动</string> +<string name="press_to_start">- 按下以开始 -</string> -<string name="pref_trans_proxy_group">透明代理(需要 Root 权限)</string> +<string name="pref_trans_proxy_group">透明代理(需要 root)</string> <string name="pref_trans_proxy_title">透明代理</string> -<string name="pref_trans_proxy_summary">应用程序自动通过 Tor 访问网络</string> +<string name="pref_trans_proxy_summary">自动对应用程序使用 Tor</string> -<string name="pref_transparent_all_title">全局使用 Tor</string> -<string name="pref_transparent_all_summary">所有应用程序均通过 Tor 访问网络</string> +<string name="pref_transparent_all_title">对所有应用使用 Tor</string> +<string name="pref_transparent_all_summary">使用 Tor 代理所有程序的流量</string> -<string name="status_install_success">Tor 程序安装成功！</string> -<string name="status_install_fail">Tor 程序安装失败，请检查日志并报告给 tor-assistants(a)torproject.org</string> +<string name="pref_transparent_port_fallback_title">Port Proxy Fallback</string> +<string name="pref_transparent_port_fallback_summary">WARNING: Circumvents common ports (80, 443, etc). *USE ONLY* if \'All\' or \'App\' mode doesn\'t work.</string> +<string name="pref_transparent_port_title">Port List</string> +<string name="pref_transparent_port_summary">List of ports to proxy. *USE ONLY* if \'All\' or \'App\' mode doesn\'t work</string> +<string name="pref_transparent_port_dialog">Enter ports to proxy</string> + +<string name="status_install_success">Tor 二进制程序安装成功！</string> +<string name="status_install_fail">Tor 二进制程序无法安装。请查看日志 tor-assistants(a)torproject.org</string> + <string name="title_error">应用程序错误</string> <string name="wizard_title">欢迎使用 Orbot</string> <string name="wizard_btn_tell_me_more">关于 Orbot</string> -<string name="btn_next">前进</string> -<string name="btn_back">后退</string> +<string name="btn_next">下一步</string> +<string name="btn_back">上一步</string> <string name="btn_finish">完成</string> <string name="btn_okay">确定</string> @@ -68,42 +75,38 @@  - <string name="wizard_welcome_msg">Orbot 把 Tor 带到 Android 中。Tor 是一个自由软件，也是一个开放的网络。它可以帮助你防御可能威胁到人身自由和隐私，秘密的商业活动和商业关系，国家安全网络监控 - 流量分析。 - -警告：* 仅仅是安装上 Orbot 并不能神奇地使你的移动网络流量变得无迹可寻！下面的向导将会引领你的匿名之路。</string> - <string name="wizard_details">关于 Orbot 的详情</string> - <string name="wizard_details_msg">Orbot 是一个开放源代码的应用程序，它包含了 Tor, LibEvent 和 Privoxy. 它在本地提供一个连接到 Tor 网络的 HTTP 代理（8118）和一个 SOCKS 代理（9050）。在有 Root 权限的设备上，Orbot 能够让所有的互联网访问通过 Tor 网络发出。</string> - <string name="wizard_permissions_root">已经获得授权</string> + <string name="wizard_welcome_msg">Orbot 将 Tor 带到 Android 之上。Tor 是自由软件，致力于构建开放网络，帮助保护网络访问的自由和隐私，可以抵御秘密的商业活动、关系以及已知的流量分析技术。\n\n*警告：*仅仅安装 Orbot 不会自动使您的移动流量匿名化！这个向导将带您入门。</string> + <string name="wizard_details">一些 Orbot 详情</string> + <string name="wizard_details_msg">Orbot 是包含 Tor、LibEvent 和 Privoxy 的开源软件。它包含了一个本地 HTTP 代理(8118)和 SOCKS 代理(9050)，使您接入 Tor 网络。Orbot 在可以获得 root 权限的设备上接管所有流量经由 Tor 网络。</string> + <string name="wizard_permissions_root">已赋予权限</string> <string name="wizard_permissions_stock">Orbot 权限</string> - <string name="wizard_premissions_msg_root">很好！我们检测到 Orbot 已经获得了 Root 权限。我们会谨慎地使用它。</string> - <string name="wizard_permissions_msg_stock">虽然 Root 权限并不是必须的，但是一旦拥有，Orbot 就会变成更加强大的工具。点击下面的按钮获得 Root 权限！</string> + <string name="wizard_premissions_msg_root">非常好！我们检测到您已经为 Orbot 配置了 root 权限。我们会聪明地使用这个特权。</string> + <string name="wizard_permissions_msg_stock">虽然并非必须，如果你的设备有 root 权限，Orbot 可以表现得更为强大。使用以下按钮来赋予 Orbot 超级能力！</string> - <string name="wizard_permissions_no_root">如果你没有 Root 权限，或者不明白我们在说什么，请一定只使用那些支持 Orbot的应用程序。</string> - <string name="wizard_permissions_consent">我知道我没有 Root 权限，继续</string> + <string name="wizard_permissions_no_root">如果你没有 root 权限，或不明白上面说的是什么，那么只需使用可与 Orbot 工作的应用程序。</string> + <string name="wizard_permissions_consent">我明白并会以非 root 继续</string> - <string name="wizard_permission_enable_root">授予 Orbot Root 权限</string> - <string name="wizard_configure">配置 Tor 功能</string> - <string name="wizard_configure_msg">Orbot 允许你选择代理所有的应用程序或者只代理某几个选中的应用程序</string> - <string name="wizard_configure_all">使用 Tor 代理所有的应用程序</string> - <string name="wizard_configure_select_apps">选择要使用 Tor 的应用程序</string> + <string name="wizard_permission_enable_root">为 Orbot 赋予 root 权限</string> + <string name="wizard_configure">配置 Tor 使用</string> + <string name="wizard_configure_msg">Orbot 给予您路由所有应用程序流量通过 Tor 或者为单个程序设置选择的选项。</string> + <string name="wizard_configure_all">通过 Tor 代理全部应用程序</string> + <string name="wizard_configure_select_apps">选择使用 Tor 的单个应用程序</string> - <string name="wizard_tips_tricks">支持 Orbot 的应用程序</string> - <string name="wizard_tips_msg">我们鼓励你下载并使用能直接连接到 Orbot 的应用程序。点击下面的按钮来安装。</string> - <string name="wizard_tips_otrchat">OTRCHAT - Android 上的安全的即时聊天客户端</string> - <string name="wizard_tips_orweb">ORWEB (仅限 Android 1.x) - 通过 Orbot 保护隐私而设计的浏览器</string> + <string name="wizard_tips_tricks">可用 Orbot 的应用程序</string> + <string name="wizard_tips_msg">我们建议您下载和使用能直接连接到 Orbot 的应用程序。点击以下按钮安装。</string> + <string name="wizard_tips_otrchat">OTRCHAT - Android 上的安全即时消息客户端</string> + <string name="wizard_tips_orweb">ORWEB (仅 Android 1.x) - 着重隐私设计的浏览器</string> <string name="wizard_tips_proxy">代理设置 - 学习如何配置应用程序使用 Orbot</string> <string name="wizard_proxy_help_info">代理设置</string> - <string name="wizard_proxy_help_msg">如果你的 Android 程序能支持 HTTP 或者 SOCKS 代码，那么你可配置它们连接到 Orbot 以便使用 Tor。 - -主机地址是 127.0.0.1 或者 localhost, HTTP 代理端口是 8118, SOCKS 代理端口是 9050. 你可以使用 SOCKS4A 或者 SOCKS5. - -获取更多关于 Android 上使用代理设置的信息请访问： http://tinyurl.com/proxyandroid - </string> + <string name="wizard_proxy_help_msg">如果您正在使用的 Android 应用程序支持使用 HTTP 或 SOCKS 代理，您可以配置它连接到 Orbot 并使用 Tor。\n\n + 主机设置为 127.0.0.1 或 localhost。对于 HTTP，端口为 8118，对于 SOCKS 端口为 9050。如果可能，应当使用 SOCKS4A 或 SOCKS5。 + \n\n + 您可以通过 http://tinyurl.com/proxyandroid 了解更多 Android 上使用代理的知识。</string> - <string name="wizard_final">Orbot 配置完成</string> - <string name="wizard_final_msg">全世界有100多万的人由于各种原因正在使用 Tor：新闻记者，博客，人权工作者，法律援助者，士兵，公司，强权政制下的公民，和普通的公民...现在你也一样在使用 Tor 了！</string> + <string name="wizard_final">Orbot 已就绪！</string> + <string name="wizard_final_msg">数以万计的各国人们因为各种原因使用 Tor：记者、人权工作者、法律工作者、士兵、公司、迫于实际情况使用的市民，当然也有普通的市民。现在你也即将成为其中的一员！</string> <string name="otrchat_apk_url">https://guardianproject.info/getgibber</string> <string name="orweb_apk_url">https://guardianproject.info/getorweb</string> @@ -111,14 +114,12 @@  - <string name="connect_first_time">你已经成功连接到Tor网络-但是，这并不意味着你的设备是安全的。你可以使用菜单中的\'检查\'选项来测试你的浏览器。 + <string name="connect_first_time">您已成功连接至 Tor 网络 - 但这并不意味着您的设备安全。您可以使用“检查”选项测试您的浏览器。\n\n访问 https://guardianproject.info/apps/orbot 或发送邮件到 help(a)guardianproject.info 以了解更多信息。</string> -访问我们的网站https://guardianproject.info/apps/orbot或发送电子邮件至help@guardianproject.info了解更多信息。</string> + <string name="tor_check">这将启动默认浏览器并访问 https://check.torproject.org 以测试是否正确配置了 Orbot 并确认您是否已经连接到 Tor。</string> - <string name="tor_check">这将打开你的默认浏览器至 https://check.torproject.org 为了确认 Orbot是否安装完成和你是否连接到了Tor。</string> - - <string name="pref_hs_group">隐匿服务</string> + <string name="pref_hs_group">隐藏服务</string> <string name="pref_general_group">General</string> <string name="pref_start_boot_title">Start Orbot on Boot</string>

1 0