July 2016 - tor-commits - lists.torproject.org

[webwml/master] Used to run -> Ran
by sebastian＠torproject.org 07 Jul '16

07 Jul '16

commit 452d1290e5aeff12c6e19876f8f6fa6cd2d4f724 Author: Sebastian Hahn <sebastian(a)torproject.org> Date: Thu Jul 7 16:49:04 2016 +0200 Used to run -> Ran --- about/en/contributors.wml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/about/en/contributors.wml b/about/en/contributors.wml index e90052d..c912596 100644 --- a/about/en/contributors.wml +++ b/about/en/contributors.wml @@ -27,8 +27,7 @@ href="http://cryptocracy.net/">personal website</a>.</dd> <dt>Jacob Appelbaum, Advocate, Developer</dt> <dd>Represented Tor at conferences and trainings, and developed tools -for the Tor ecosystem. Used to run one of the Tor directory -authorities.</dd> +for the Tor ecosystem. Ran one of the Tor directory authorities.</dd> <dt>Katherine Bergeron</dt> <dd>Helped streamlining our operations, including fundraising, merchandise

1 0

[torspec/master] Add the twstrike guard selection draft as a separate proposal
by nickm＠torproject.org 07 Jul '16

07 Jul '16

commit f6f897c365044bcb293726b4579358826ceb038c Author: Nick Mathewson <nickm(a)torproject.org> Date: Thu Jul 7 10:08:45 2016 -0400 Add the twstrike guard selection draft as a separate proposal Taken from https://raw.githubusercontent.com/twstrike/torspec/review/proposals/259-gua… See editorial note for comment on why I'm not just dropping this in over prop259. --- proposals/000-index.txt | 2 + proposals/268-guard-selection.txt | 507 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 509 insertions(+) diff --git a/proposals/000-index.txt b/proposals/000-index.txt index 23fc3dc..180ba4e 100644 --- a/proposals/000-index.txt +++ b/proposals/000-index.txt @@ -188,6 +188,7 @@ Proposals by number: 265 Load Balancing with Overhead Parameters [ACCEPTED] 266 Removing current obsolete clients from the Tor network [DRAFT] 267 Tor Consensus Transparency [DRAFT] +268 New Guard Selection Behaviour [DRAFT] Proposals by status: @@ -213,6 +214,7 @@ Proposals by status: 260 Rendezvous Single Onion Services 266 Removing current obsolete clients from the Tor network 267 Tor Consensus Transparency + 268 New Guard Selection Behaviour NEEDS-REVISION: 190 Bridge Client Authorization Based on a Shared Secret NEEDS-RESEARCH: diff --git a/proposals/268-guard-selection.txt b/proposals/268-guard-selection.txt new file mode 100644 index 0000000..9277dd8 --- /dev/null +++ b/proposals/268-guard-selection.txt @@ -0,0 +1,507 @@ +Filename: 268-guard-selection.txt +Title: New Guard Selection Behaviour +Author: Isis Lovecruft, George Kadianakis, [Ola Bini] +Created: 2015-10-28 +Status: Draft + + (Editorial note: this was origianlly written as a revision of + proposal 259, but it diverges so substantially that it seemed + better to assign it a new number for reference, so that we + aren't always talking about "The old 259" and "the new 259". -NM) + +§1. Overview + + Tor uses entry guards to prevent an attacker who controls some + fraction of the network from observing a fraction of every user's + traffic. If users chose their entries and exits uniformly at + random from the list of servers every time they build a circuit, + then an adversary who had (k/N) of the network would deanonymize + F=(k/N)^2 of all circuits... and after a given user had built C + circuits, the attacker would see them at least once with + probability 1-(1-F)^C. With large C, the attacker would get a + sample of every user's traffic with probability 1. + + To prevent this from happening, Tor clients choose a small number of + guard nodes (currently 3). These guard nodes are the only nodes + that the client will connect to directly. If they are not + compromised, the user's paths are not compromised. + + But attacks remain. Consider an attacker who can run a firewall + between a target user and the Tor network, and make + many of the guards they don't control appear to be unreachable. + Or consider an attacker who can identify a user's guards, and mount + denial-of-service attacks on them until the user picks a guard + that the attacker controls. + + In the presence of these attacks, we can't continue to connect to + the Tor network unconditionally. Doing so would eventually result + in the user choosing a hostile node as their guard, and losing + anonymity. + + This proposal outlines a new entry guard selection algorithm, which + addresses the following concerns: + + - Heuristics and algorithms for determining how and which guard(s) + is(/are) chosen should be kept as simple and easy to understand + as possible. + + - Clients in censored regions or who are behind a fascist firewall + who connect to the Tor network should not experience any + significant disadvantage in terms of reachability or usability. + + - Tor should make a best attempt at discovering the most + appropriate behaviour, with as little user input and + configuration as possible. + + +§2. Design + + Alice, an OP attempting to connect to the Tor network, should + undertake the following steps to determine information about the + local network and to select (some) appropriate entry guards. In the + following scenario, it is assumed that Alice has already obtained a + recent, valid, and verifiable consensus document. + + The algorithm is divided into four components such that the full + algorithm is implemented by first invoking START, then repeatedly + calling NEXT while adviced it SHOULD_CONTINUE and finally calling + END. For an example usage see §A. Appendix. + + Several components of NEXT can be invoked asynchronously. SHOULD_CONTINUE + is used for the algorithm to be able to tell the caller whether we + consider the work done or not - this can be used to retry primary + guards when we finally are able to connect to a guard after a long + network outage, for example. + + This algorithm keeps track of the unreachability status for guards + in state global to the system, so that repeated runs will not have + to rediscover unreachability over and over again. However, this + state does not need to be persisted permanently - it is purely an + optimization. + + The algorithm expects several arguments to guide its behavior. These + will be defined in §2.1. + + The goal of this algorithm is to strongly prefer connecting to the + same guards we have connected to before, while also trying to detect + conditions such as a network outage. The way it does this is by keeping + track of how many guards we have exposed ourselves to, and if we have + connected to too many we will fall back to only retrying the ones we have + already tried. The algorithm also decides on sample set that should + be persisted - in order to minimize the risk of an attacker forcing + enumeration of the whole network by triggering rebuilding of + circuits. + + +§2.1. Definitions + + Bad guard: a guard is considered bad if it conforms with the function IS_BAD + (see §G. Appendix for details). + + Dead guard: a guard is considered dead if it conforms with the function + IS_DEAD (see §H. Appendix for details). + + Obsolete guard: a guard is considered obsolete if it conforms with the + function IS_OBSOLETE (see §I. Appendix for details). + + Live entry guard: a guard is considered live if it conforms with the function + IS_LIVE (see §D. Appendix for details). + +§2.1. The START algorithm + + In order to start choosing an entry guard, use the START + algorithm. This takes four arguments that can be used to fine tune + the workings: + + USED_GUARDS + This is a list that contains all the guards that have been used + before by this client. We will prioritize using guards from this + list in order to minimize our exposure. The list is expected to + be sorted based on priority, where the first entry will have the + highest priority. + + SAMPLED_GUARDS + This is a set that contains all guards that should be considered + for connection. This set should be persisted between runs. It + should be filled by using NEXT_BY_BANDWIDTH with GUARDS as an + argument if it's empty, or if it contains less than SAMPLE_SET_THRESHOLD + guards after winnowing out older guards. + + N_PRIMARY_GUARDS + The number of guards we should consider our primary + guards. These guards will be retried more frequently and will + take precedence in most situations. By default the primary + guards will be the first N_PRIMARY_GUARDS guards from USED_GUARDS. + When the algorith is used in constrained mode (have bridges or entry + nodes in the configuration file), this value should be 1 otherwise the + proposed value is 3. + + DIR + If this argument is set, we should only consider guards that can + be directory guards. If not set, we will consider all guards. + + The primary work of START is to initialize the state machine depicted + in §2.2. The initial state of the machine is defined by: + + GUARDS + This is a set of all guards from the consensus. It will primarily be used + to fill in SAMPLED_GUARDS + + FILTERED_SAMPLED + This is a set that contains all guards that we are willing to connect to. + It will be obtained from calling FILTER_SET with SAMPLED_GUARDS as + argument. + + REMAINING_GUARDS + This is a running set of the guards we have not yet tried to connect to. + It should be initialized to be FILTERED_SAMPLED without USED_GUARDS. + + STATE + A variable that keeps track of which state in the state + machine we are currently in. It should be initialized to + STATE_PRIMARY_GUARDS. + + PRIMARY_GUARDS + This list keeps track of our primary guards. These are guards + that we will prioritize when trying to connect, and will also + retry more often in case of failure with other guards. + It should be initialized by calling algorithm + NEXT_PRIMARY_GUARD repeatedly until PRIMARY_GUARDS contains + N_PRIMARY_GUARDS elements. + + +§2.2. The NEXT algorithm + + The NEXT algorithm is composed of several different possibly flows. The + first one is a simple state machine that can transfer between two + different states. Every time NEXT is invoked, it will resume at the + state where it left off previously. In the course of selecting an + entry guard, a new consensus can arrive. When that happens we need + to update the data structures used, but nothing else should change. + + Before jumping in to the state machine, we should first check if it + was at least PRIMARY_GUARDS_RETRY_INTERVAL minutes since we tried + any of the PRIMARY_GUARDS. If this is the case, and we are not in + STATE_PRIMARY_GUARDS, we should save the previous state and set the + state to STATE_PRIMARY_GUARDS. + + +§2.2.1. The STATE_PRIMARY_GUARDS state + + Return each entry in PRIMARY_GUARDS in turn. For each entry, if the + guard should be retried and considered suitable use it. A guard is + considered to eligible to retry if is marked for retry or is live + and id not bad. Also, a guard is considered to be suitable if is + live and, if is a directory it should not be a cache. + + If all entries have been tried transition to STATE_TRY_REMAINING. + +§2.2.2. The STATE_TRY_REMAINING state + + Return each entry in USED_GUARDS that is not in PRIMARY_GUARDS in + turn.For each entry, if a guard is found return it. + + Return each entry from REMAINING_GUARDS in turn. + For each entry, if the guard should be retried and considered + suitable use it and mark it as unreachable. A guard is + considered to eligible to retry if is marked for retry or is live + and id not bad. Also, a guard is considered to be suitable if is + live and, if is a directory it should not be a cache. + + If no entries remain in REMAINING_GUARDS, transition to + STATE_PRIMARY_GUARDS. + + +§2.2.3. ON_NEW_CONSENSUS + + First, ensure that all guard profiles are updated with information + about whether they were in the newest consensus or not. + + Update the bad status for all guards in USED_GUARDS and SAMPLED_GUARDS. + Remove all dead guards from USED_GUARDS and SAMPLED_GUARDS. + Remove all obsolete guards from USED_GUARDS and SAMPLED_GUARDS. + +§2.3. The SHOULD_CONTINUE algorithm + + This algorithm takes as an argument a boolean indicating whether the + circuit was successfully built or not. + + After the caller have tried to build a circuit with a returned + guard, they should invoke SHOULD_CONTINUE to understand if the + algorithm is finished or not. SHOULD_CONTINUE will always return + true if the circuit failed. If the circuit succeeded, + SHOULD_CONTINUE will always return false, unless the guard that + succeeded was the first guard to succeed after + INTERNET_LIKELY_DOWN_INTERVAL minutes - in that case it will set the + state to STATE_PRIMARY_GUARDS and return true. + + +§2.4. The END algorithm + + The goal of this algorithm is simply to make sure that we keep track + of successful connections made. This algorithm should be invoked + with the guard that was used to correctly set up a circuit. + + Once invoked, this algorithm will mark the guard as used, and make + sure it is in USED_GUARDS, by adding it at the end if it was not there. + + +§2.5. Helper algorithms + + These algorithms are used in the above algorithms, but have been + separated out here in order to make the flow clearer. + + NEXT_PRIMARY_GUARD + - Return the first entry from USED_GUARDS that is not in + PRIMARY_GUARDS and that is in the most recent consensus. + - If USED_GUARDS is empty, use NEXT_BY_BANDWIDTH with + REMAINING_GUARDS as the argument. + + NEXT_BY_BANDWIDTH + - Takes G as an argument, which should be a set of guards to + choose from. + - Return a randomly select element from G, weighted by bandwidth. + + FILTER_SET + - Takes G as an argument, which should be a set of guards to filter. + - Filter out guards in G that don't comply with IS_LIVE (see + §D. Appendix for details). + - If the filtered set is smaller than MINIMUM_FILTERED_SAMPLE_SIZE and G + is smaller than MAXIMUM_SAMPLE_SIZE_THRESHOLD, expand G and try to + filter out again. G is expanded by adding one new guard at a time using + NEXT_BY_BANDWIDTH with GUARDS as an argument. + - If G is not smaller than MAXIMUM_SAMPLE_SIZE_THRESHOLD, G should not be + expanded. Abort execution of this function by returning null and report + an error to the user. + + +§3. Consensus Parameters, & Configurable Variables + + This proposal introduces several new parameters that ideally should + be set in the consensus but that should also be possible to + set or override in the client configuration file. Some of these have + proposed values, but for others more simulation and trial needs to + happen. + + PRIMARY_GUARDS_RETRY_INTERVAL + In order to make it more likely we connect to a primary guard, + we would like to retry the primary guards more often than other + types of guards. This parameter controls how many minutes should + pass before we consider retrying primary guards again. The + proposed value is 3. + + SAMPLE_SET_THRESHOLD + In order to allow us to recognize completely unreachable network, + we would like to avoid connecting to too many guards before switching + modes. We also want to avoid exposing ourselves to too many nodes in a + potentially hostile situation. This parameter, expressed as a + fraction, determines the number of guards we should keep as the + sampled set of the only guards we will consider connecting + to. It will be used as a fraction for the sampled set. + If we assume there are 1900 guards, a setting of 0.02 + means we will have a sample set of 38 guards. + This limits our total exposure. Proposed value is 0.02. + + MINIMUM_FILTERED_SAMPLE_SIZE + The minimum size of the sampled set after filtering out nodes based on + client configuration (FILTERED_SAMPLED). Proposed value is ???. + + MAXIMUM_SAMPLE_SIZE_THRESHOLD + In order to guarantee a minimum size of guards after filtering, + we expand SAMPLED_GUARDS until a limit. This fraction of GUARDS will be + used as an upper bound when expanding SAMPLED_GUARDS. + Proposed value is 0.03. + + INTERNET_LIKELY_DOWN_INTERVAL + The number of minutes since we started trying to find an entry + guard before we should consider the network down and consider + retrying primary guards before using a functioning guard + found. Proposed value 5. + +§4. Security properties and behavior under various conditions + + Under normal conditions, this algorithm will allow us to quickly + connect and use guards we have used before with high likelihood of + working. Assuming the first primary guard is reachable and in the + consensus, this algorithm will deterministically always return that + guard. + + Under dystopic conditions (when a firewall is in place that blocks + all ports except for potentially port 80 and 443), this algorithm + will try to connect to 2% of all guards before switching modes to try + dystopic guards. Currently, that means trying to connect to circa 40 + guards before getting a successful connection. If we assume a + connection try will take maximum 10 seconds, that means it will take + up to 6 minutes to get a working connection. + + When the network is completely down, we will try to connect to 2% of + all guards plus 2% of all dystopic guards before realizing we are + down. This means circa 50 guards tried assuming there are 1900 guards + in the network. + + In terms of exposure, we will connect to a maximum of 2% of all + guards plus 2% of all dystopic guards, or 3% of all guards, + whichever is lower. If N is the number of guards, and k is the + number of guards an attacker controls, that means an attacker would + have a probability of 1-(1-(k/N)^2)^(N * 0.03) to have one of their + guards selected before we fall back. In real terms, this means an + attacker would need to control over 10% of all guards in order to + have a larger than 50% chance of controlling a guard for any given client. + + In addition, since the sampled set changes slowly (the suggestion + here is that guards in it expire every month) it is not possible for + an attacker to force a connection to an entry guard that isn't + already in the users sampled set. + + +§A. Appendix: An example usage + + In order to clarify how this algorithm is supposed to be used, this + pseudo code illustrates the building of a circuit: + + ESTABLISH_CIRCUIT: + + if chosen_entry_node = NULL + if context = NULL + context = ALGO_CHOOSE_ENTRY_GUARD_START(used_guards, + sampled_guards=[], + options, + n_primary_guards=3, + dir=false, + guards_in_consensus) + + chosen_entry_node = ALGO_CHOOSE_ENTRY_GUARD_NEXT(context) + if not IS_SUITABLE(chosen_entry_node) + try another entry guard + + circuit = composeCircuit(chosen_entry_node) + return circuit + + ON_FIRST_HOP_CALLBACK(channel): + + if !SHOULD_CONTINUE: + ALGO_CHOOSE_ENTRY_GUARD_END(entryGuard) + else + chosen_entry_node = NULL + + +§B. Appendix: Entry Points in Tor + + In order to clarify how this algorithm is supposed to be integrated with + Tor, here are some entry points to trigger actions mentioned in spec: + + When establish_circuit: + + If *chosen_entry_node* doesn't exist + If *context* exist, populate the first one as *context* + Otherwise, use ALGO_CHOOSE_ENTRY_GUARD_START to initalize a new *context*. + + After this when we want to choose_good_entry_server, we will use + ALGO_CHOOSE_ENTRY_GUARD_NEXT to get a candidate. + + Use chosen_entry_node to build_circuit and handle_first_hop, + return this circuit + + When entry_guard_register_connect_status(should_continue): + + if !should_continue: + Call ALGO_CHOOSE_ENTRY_GUARD_END(chosen_entry_node) + else: + Set chosen_entry_node to NULL + + When new directory_info_has_arrived: + + Do ON_NEW_CONSENSUS + + +§C. Appendix: IS_SUITABLE helper function + + A guard is suitable if it satisfies all of the folowing conditions: + - It's considered to be live, according to IS_LIVE. + - It's a directory cache if a directory guard is requested. + - It's not the chosen exit node. + - It's not in the family of the chosen exit node. + + This conforms to the existing conditions in "populate_live_entry_guards()". + + +§D. Appendix: IS_LIVE helper function + + A guard is considered live if it satisfies all of the folowing conditions: + - It's not disabled because of path bias issues (path_bias_disabled). + - It was not observed to become unusable according to the directory or + the user configuration (bad_since). + - It's marked for retry (can_retry) or it's been unreachable for some + time (unreachable_since) but enough time has passed since we last tried + to connect to it (entry_is_time_to_retry). + - It's in our node list, meaninig it's present in the latest consensus. + - It has a usable descriptor (either a routerdescriptor or a + microdescriptor) unless a directory guard is requested. + - It's a general-purpose router unless UseBridges is configured. + - It's reachable by the configuration (fascist_firewall_allows_node). + + This conforms to the existing conditions in "entry_is_live()". + + A guard is observed to become unusable according to the directory or the + user configuration if it satisfies any of the following conditions: + - It's not in our node list, meaninig it's present in the latest consensus. + - It's not currently running (is_running). + - It's not a bridge and not a configured bridge + (node_is_a_configured_bridge) and UseBridges is True. + - It's not a possible guard and is not in EntryNodes and UseBridges is + False. + - It's in ExcludeNodes. Nevertheless this is ignored when + loading from config. + - It's not reachable by the configuration (fascist_firewall_allows_node). + - It's disabled because of path bias issues (path_bias_disabled). + + This conforms to the existing conditions in "entry_guards_compute_status()". + +§E. Appendix: UseBridges and Bridges configurations + + This is mutually exclusive with EntryNodes. + + If options->UseBridges OR options->EntryNodes: + - guards = populate_live_entry_guards() - this is the "bridge flavour" of + IS_SUITABLE as mentioned before. + - return node_sl_choose_by_bandwidth(guards, WEIGHT_FOR_GUARD) + This is "choose a guard from S by bandwidth weight". + + UseBridges and Bridges must be set together. Bridges go to bridge_list (via + bridge_add_from_config()), but how is it used? + learned_bridge_descriptor() adds the bridge to the global entry_guards if + UseBridges = True. + + We either keep the existing global entry_guards OR incorporate bridges in the + proposal (remove non bridges from USED_GUARDS, and REMAINING_GUARDS = bridges?) + + If UseBridges is set as true, we need to fill the SAMPLED_GUARDS + with bridges specified and learned from consensus. + +§F. Appendix: EntryNodes configuration + + This is mutually exclusive with Bridges. + + The global entry_guards will be updated with entries in EntryNodes + (see entry_guards_set_from_config()). + + If EntryNodes is set, we need to fill the SAMPLED_GUARDS with + EntryNodes specified in options. + +§G. Appendix: IS_BAD helper function + + A guard is considered bad if is not included in the newest + consensus. + +§H. Appendix: IS_DEAD helper function + + A guard is considered dead if it's marked as bad for + ENTRY_GUARD_REMOVE_AFTER period (30 days) unless they have been disabled + because of path bias issues (path_bias_disabled). + +§I. Appendix: IS_OBSOLETE helper function + + A guard is considered obsolete if it was chosen by an Tor + version we can't recognize or it was chosen more than GUARD_LIFETIME ago. + +-*- coding: utf-8 -*-

1 0

[onionoo/master] Add checkstyle task.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit ce4be425c23185d4dcd7c0de11154baff85a9f3b Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 11:45:25 2016 +0200 Add checkstyle task. Copied over from CollecTor. --- build.xml | 25 ++++ src/test/resources/metrics_checks.xml | 217 ++++++++++++++++++++++++++++++++++ 2 files changed, 242 insertions(+) diff --git a/build.xml b/build.xml index 49376be..1688207 100644 --- a/build.xml +++ b/build.xml @@ -6,7 +6,9 @@ <property name="descriptorversion" value="1.2.0"/> <property name="javasources" value="src/main/java"/> <property name="tests" value="src/test/java"/> + <property name="generated" value="generated/"/> <property name="classes" value="classes"/> + <property name="testresources" value="src/test/resources/"/> <property name="dist" value="dist"/> <property name="libs" value="lib"/> <property name="config" value="etc"/> @@ -73,11 +75,18 @@ </fileset> </path> + <path id="checkstyle.classpath" > + <fileset dir="${libs}"> + <include name="checkstyle-6.17-all.jar" /> + </fileset> + </path> + <target name="init"> <copy file="${contextxmltemplate}" tofile="${contextxml}"/> <copy file="${webxmltemplate}" tofile="${webxml}"/> <mkdir dir="${classes}"/> <mkdir dir="${dist}"/> + <mkdir dir="${generated}"/> </target> <target name="clean" > @@ -124,6 +133,22 @@ </junit> </target> + <taskdef resource="com/puppycrawl/tools/checkstyle/ant/checkstyle-ant-task.properties"> + <classpath refid="checkstyle.classpath" /> + </taskdef> + <target name="checks" depends="compile"> + <checkstyle config="${testresources}/metrics_checks.xml"> + <fileset dir="${javasources}" + includes="**/*.java" /> + <classpath> + <path refid="classpath" /> + <path refid="checkstyle.classpath" /> + </classpath> + <formatter type="plain" + toFile="${generated}/checkstyle_report.txt"/> + </checkstyle> + </target> + <target name="war" depends="compile"> <war destfile="${warfile}" diff --git a/src/test/resources/metrics_checks.xml b/src/test/resources/metrics_checks.xml new file mode 100644 index 0000000..0777894 --- /dev/null +++ b/src/test/resources/metrics_checks.xml @@ -0,0 +1,217 @@ +<?xml version="1.0"?> +<!DOCTYPE module PUBLIC + "-//Puppy Crawl//DTD Check Configuration 1.3//EN" + "http://www.puppycrawl.com/dtds/configuration_1_3.dtd"> + + + +<module name = "Checker"> + <property name="charset" value="UTF-8"/> + + <property name="severity" value="warning"/> + + <property name="fileExtensions" value="java, properties, xml"/> +  +  + <module name="FileTabCharacter"> + <property name="eachLine" value="true"/> + </module> + + <module name="TreeWalker"> + <module name="OuterTypeFilename"/> + <module name="IllegalTokenText"> + <property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/> + <property name="format" value="\\u00(08|09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/> + <property name="message" value="Avoid using corresponding octal or Unicode escape."/> + </module> + <module name="AvoidEscapedUnicodeCharacters"> + <property name="allowEscapesForControlCharacters" value="true"/> + <property name="allowByTailComment" value="true"/> + <property name="allowNonPrintableEscapes" value="true"/> + </module> + <module name="LineLength"> + <property name="max" value="100"/> + <property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/> + </module> + <module name="AvoidStarImport"/> + <module name="OneTopLevelClass"/> + <module name="NoLineWrap"/> + <module name="EmptyBlock"> + <property name="option" value="TEXT"/> + <property name="tokens" value="LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH"/> + </module> + <module name="NeedBraces"/> + <module name="LeftCurly"> + <property name="maxLineLength" value="100"/> + </module> + <module name="RightCurly"/> + <module name="RightCurly"> + <property name="option" value="alone"/> + <property name="tokens" value="CLASS_DEF, METHOD_DEF, CTOR_DEF, LITERAL_FOR, LITERAL_WHILE, LITERAL_DO, STATIC_INIT, INSTANCE_INIT"/> + </module> + <module name="WhitespaceAround"> + <property name="allowEmptyConstructors" value="true"/> + <property name="allowEmptyMethods" value="true"/> + <property name="allowEmptyTypes" value="true"/> + <property name="allowEmptyLoops" value="true"/> + <message key="ws.notFollowed" + value="WhitespaceAround: ''{0}'' is not followed by whitespace. Empty blocks may only be represented as '{}' when not part of a multi-block statement (4.1.3)"/> + <message key="ws.notPreceded" + value="WhitespaceAround: ''{0}'' is not preceded with whitespace."/> + </module> + <module name="OneStatementPerLine"/> + <module name="MultipleVariableDeclarations"/> + <module name="ArrayTypeStyle"/> + <module name="MissingSwitchDefault"/> + <module name="FallThrough"/> + <module name="UpperEll"/> + <module name="ModifierOrder"/> + <module name="EmptyLineSeparator"> + <property name="allowNoEmptyLineBetweenFields" value="true"/> + </module> + <module name="SeparatorWrap"> + <property name="tokens" value="DOT"/> + <property name="option" value="nl"/> + </module> + <module name="SeparatorWrap"> + <property name="tokens" value="COMMA"/> + <property name="option" value="EOL"/> + </module> + <module name="PackageName"> + <property name="format" value="^[a-z]+(\.[a-z][a-z0-9]*)*$"/> + <message key="name.invalidPattern" + value="Package name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="TypeName"> + <message key="name.invalidPattern" + value="Type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="MemberName"> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9]*$"/> + <message key="name.invalidPattern" + value="Member name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="ParameterName"> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9]*$"/> + <message key="name.invalidPattern" + value="Parameter name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="CatchParameterName"> + <property name="format" value="^[a-z][a-zA-Z0-9]*$"/> + <message key="name.invalidPattern" + value="Catch parameter name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="LocalVariableName"> + <property name="tokens" value="VARIABLE_DEF"/> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9]*$"/> + <property name="allowOneCharVarInForLoop" value="true"/> + <message key="name.invalidPattern" + value="Local variable name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="ClassTypeParameterName"> + <property name="format" value="(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)"/> + <message key="name.invalidPattern" + value="Class type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="MethodTypeParameterName"> + <property name="format" value="(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)"/> + <message key="name.invalidPattern" + value="Method type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="InterfaceTypeParameterName"> + <property name="format" value="(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)"/> + <message key="name.invalidPattern" + value="Interface type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="NoFinalizer"/> + <module name="GenericWhitespace"> + <message key="ws.followed" + value="GenericWhitespace ''{0}'' is followed by whitespace."/> + <message key="ws.preceded" + value="GenericWhitespace ''{0}'' is preceded with whitespace."/> + <message key="ws.illegalFollow" + value="GenericWhitespace ''{0}'' should followed by whitespace."/> + <message key="ws.notPreceded" + value="GenericWhitespace ''{0}'' is not preceded with whitespace."/> + </module> + <module name="Indentation"> + <property name="basicOffset" value="2"/> + <property name="braceAdjustment" value="0"/> + <property name="caseIndent" value="2"/> + <property name="throwsIndent" value="4"/> + <property name="lineWrappingIndentation" value="4"/> + <property name="arrayInitIndent" value="2"/> + </module> + <module name="AbbreviationAsWordInName"> + <property name="ignoreFinal" value="false"/> + <property name="allowedAbbreviationLength" value="1"/> + </module> + <module name="OverloadMethodsDeclarationOrder"/> + <module name="VariableDeclarationUsageDistance"/> + <module name="CustomImportOrder"> + <property name="specialImportsRegExp" value="org.torproject"/> + <property name="sortImportsInGroupAlphabetically" value="true"/> + <property name="customImportOrderRules" value="STATIC###SPECIAL_IMPORTS###THIRD_PARTY_PACKAGE###STANDARD_JAVA_PACKAGE"/> + </module> + <module name="MethodParamPad"/> + <module name="OperatorWrap"> + <property name="option" value="NL"/> + <property name="tokens" value="BAND, BOR, BSR, BXOR, DIV, EQUAL, GE, GT, LAND, LE, LITERAL_INSTANCEOF, LOR, LT, MINUS, MOD, NOT_EQUAL, PLUS, QUESTION, SL, SR, STAR "/> + </module> + <module name="AnnotationLocation"> + <property name="tokens" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF"/> + </module> + <module name="AnnotationLocation"> + <property name="tokens" value="VARIABLE_DEF"/> + <property name="allowSamelineMultipleAnnotations" value="true"/> + </module> + <module name="NonEmptyAtclauseDescription"/> + <module name="JavadocTagContinuationIndentation"/> + <module name="SummaryJavadoc"> + <property name="forbiddenSummaryFragments" value="^@return the *|^This method returns |^A [{]@code [a-zA-Z0-9]+[}]( is a )"/> + </module> + <module name="JavadocParagraph"/> + <module name="AtclauseOrder"> + <property name="tagOrder" value="@param, @return, @throws, @deprecated"/> + <property name="target" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF, VARIABLE_DEF"/> + </module> + <module name="JavadocMethod"> + <property name="scope" value="public"/> + <property name="allowMissingParamTags" value="true"/> + <property name="allowMissingThrowsTags" value="true"/> + <property name="allowMissingReturnTag" value="true"/> + <property name="minLineCount" value="2"/> + <property name="allowedAnnotations" value="Override, Test"/> + <property name="allowThrowsTagsForSubclasses" value="true"/> + </module> + <module name="MethodName"> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9_]*$"/> + <message key="name.invalidPattern" + value="Method name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="SingleLineJavadoc"> + <property name="ignoreInlineTags" value="false"/> + </module> + <module name="EmptyCatchBlock"> + <property name="exceptionVariableName" value="expected"/> + </module> + <module name="CommentsIndentation"/> + </module> +</module>

1 0

[onionoo/master] Fix most of the style issues found by checkstyle.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit ab05509d3b78693aa72cd3bca7da05b650cfe665 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 16:21:09 2016 +0200 Fix most of the style issues found by checkstyle. Fixed the following number of issues per type: 559 EmptyLineSeparator 196 CustomImportOrder 180 OperatorWrap 120 Indentation 117 JavadocMethod 80 MemberName 56 MultipleVariableDeclarations 23 AbbreviationAsWordInName 16 ModifierOrder 11 ParameterName 9 LocalVariableName 7 VariableDeclarationUsageDistance 6 SeparatorWrap 4 EmptyCatchBlock 3 OneTopLevelClass 2 JavadocTagContinuationIndentation 1 (... types with single occurrence omitted...) --- .../java/org/torproject/onionoo/cron/Main.java | 63 +++--- .../torproject/onionoo/docs/BandwidthDocument.java | 4 + .../torproject/onionoo/docs/BandwidthStatus.java | 50 +++-- .../torproject/onionoo/docs/ClientsDocument.java | 3 + .../onionoo/docs/ClientsGraphHistory.java | 19 ++ .../torproject/onionoo/docs/ClientsHistory.java | 30 ++- .../org/torproject/onionoo/docs/ClientsStatus.java | 44 ++-- .../torproject/onionoo/docs/DateTimeHelper.java | 73 ++++--- .../torproject/onionoo/docs/DetailsDocument.java | 102 ++++++++- .../org/torproject/onionoo/docs/DetailsStatus.java | 104 ++++++++- .../java/org/torproject/onionoo/docs/Document.java | 3 + .../org/torproject/onionoo/docs/DocumentStore.java | 136 +++++++----- .../onionoo/docs/DocumentStoreFactory.java | 3 + .../org/torproject/onionoo/docs/GraphHistory.java | 14 ++ .../org/torproject/onionoo/docs/NodeStatus.java | 77 +++++-- .../torproject/onionoo/docs/SummaryDocument.java | 44 +++- .../org/torproject/onionoo/docs/UpdateStatus.java | 3 + .../torproject/onionoo/docs/UptimeDocument.java | 6 + .../org/torproject/onionoo/docs/UptimeHistory.java | 24 ++- .../org/torproject/onionoo/docs/UptimeStatus.java | 40 ++-- .../torproject/onionoo/docs/WeightsDocument.java | 7 + .../org/torproject/onionoo/docs/WeightsStatus.java | 67 +++--- .../onionoo/server/HttpServletRequestWrapper.java | 10 +- .../onionoo/server/HttpServletResponseWrapper.java | 11 +- .../org/torproject/onionoo/server/NodeIndex.java | 38 +++- .../org/torproject/onionoo/server/NodeIndexer.java | 96 +++++---- .../onionoo/server/NodeIndexerFactory.java | 3 + .../onionoo/server/PerformanceMetrics.java | 76 ++++--- .../torproject/onionoo/server/RequestHandler.java | 91 ++++---- .../torproject/onionoo/server/ResourceServlet.java | 39 ++-- .../torproject/onionoo/server/ResponseBuilder.java | 44 ++-- .../org/torproject/onionoo/server/ServerMain.java | 1 + .../onionoo/updater/BandwidthStatusUpdater.java | 1 + .../onionoo/updater/ClientsStatusUpdater.java | 19 +- .../onionoo/updater/DescriptorDownloader.java | 65 +++--- .../onionoo/updater/DescriptorHistory.java | 3 +- .../onionoo/updater/DescriptorListener.java | 3 +- .../onionoo/updater/DescriptorQueue.java | 126 +++++------ .../onionoo/updater/DescriptorSource.java | 78 ++++--- .../onionoo/updater/DescriptorSourceFactory.java | 3 + .../torproject/onionoo/updater/DescriptorType.java | 1 + .../torproject/onionoo/updater/LookupResult.java | 20 +- .../torproject/onionoo/updater/LookupService.java | 49 +++-- .../onionoo/updater/NodeDetailsStatusUpdater.java | 233 +++++++++++---------- .../onionoo/updater/RdnsLookupRequest.java | 16 +- .../onionoo/updater/RdnsLookupWorker.java | 5 +- .../onionoo/updater/ReverseDomainNameResolver.java | 12 +- .../onionoo/updater/StatusUpdateRunner.java | 10 +- .../torproject/onionoo/updater/StatusUpdater.java | 1 + .../onionoo/updater/UptimeStatusUpdater.java | 28 +-- .../onionoo/updater/WeightsStatusUpdater.java | 74 ++++--- .../torproject/onionoo/util/FormattingUtils.java | 7 +- .../java/org/torproject/onionoo/util/LockFile.java | 13 +- .../java/org/torproject/onionoo/util/Time.java | 2 + .../org/torproject/onionoo/util/TimeFactory.java | 3 + .../onionoo/writer/BandwidthDocumentWriter.java | 47 +++-- .../onionoo/writer/ClientsDocumentWriter.java | 61 +++--- .../onionoo/writer/DetailsDocumentWriter.java | 40 ++-- .../torproject/onionoo/writer/DocumentWriter.java | 1 + .../onionoo/writer/DocumentWriterRunner.java | 8 +- .../onionoo/writer/SummaryDocumentWriter.java | 33 +-- .../onionoo/writer/UptimeDocumentWriter.java | 65 +++--- .../onionoo/writer/WeightsDocumentWriter.java | 49 +++-- .../onionoo/docs/BandwidthStatusTest.java | 11 +- .../onionoo/docs/DummyDocumentStore.java | 3 - .../torproject/onionoo/docs/NodeStatusTest.java | 4 +- .../onionoo/docs/SummaryDocumentTest.java | 4 +- .../torproject/onionoo/docs/UptimeStatusTest.java | 4 +- .../onionoo/server/ResourceServletTest.java | 30 ++- .../onionoo/updater/DummyBridgeStatus.java | 6 +- .../torproject/onionoo/updater/DummyConsensus.java | 10 +- .../onionoo/updater/DummyDescriptorSource.java | 7 +- .../onionoo/updater/DummyStatusEntry.java | 4 +- .../onionoo/updater/LookupServiceTest.java | 12 +- .../onionoo/updater/UptimeStatusUpdaterTest.java | 5 +- .../org/torproject/onionoo/util/DummyTime.java | 2 - .../onionoo/writer/UptimeDocumentWriterTest.java | 11 +- 77 files changed, 1606 insertions(+), 908 deletions(-) diff --git a/src/main/java/org/torproject/onionoo/cron/Main.java b/src/main/java/org/torproject/onionoo/cron/Main.java index 6f39cb6..6392045 100644 --- a/src/main/java/org/torproject/onionoo/cron/Main.java +++ b/src/main/java/org/torproject/onionoo/cron/Main.java @@ -1,15 +1,8 @@ /* Copyright 2011--2015 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.cron; -import java.io.File; -import java.util.Calendar; -import java.util.concurrent.Executors; -import java.util.concurrent.ScheduledExecutorService; -import java.util.concurrent.TimeUnit; +package org.torproject.onionoo.cron; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.onionoo.docs.DocumentStore; import org.torproject.onionoo.docs.DocumentStoreFactory; import org.torproject.onionoo.updater.DescriptorSource; @@ -18,6 +11,15 @@ import org.torproject.onionoo.updater.StatusUpdateRunner; import org.torproject.onionoo.util.LockFile; import org.torproject.onionoo.writer.DocumentWriterRunner; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.File; +import java.util.Calendar; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; + /* Update search data and status data files. */ public class Main implements Runnable { @@ -32,8 +34,15 @@ public class Main implements Runnable { main.runOrScheduleExecutions(); } - boolean defaultMode = false, singleRun = false, downloadOnly = false, - updateOnly = false, writeOnly = false; + boolean defaultMode = false; + + boolean singleRun = false; + + boolean downloadOnly = false; + + boolean updateOnly = false; + + boolean writeOnly = false; /* TODO Parsing command-line arguments is only a workaround until we're * more certain what kind of options we want to support. We should then @@ -44,23 +53,23 @@ public class Main implements Runnable { this.defaultMode = true; } else if (args.length == 1) { switch (args[0]) { - case "--help": - this.printUsageAndExit(0); - break; - case "--single-run": - this.singleRun = true; - break; - case "--download-only": - this.downloadOnly = true; - break; - case "--update-only": - this.updateOnly = true; - break; - case "--write-only": - this.writeOnly = true; - break; - default: - validArgs = false; + case "--help": + this.printUsageAndExit(0); + break; + case "--single-run": + this.singleRun = true; + break; + case "--download-only": + this.downloadOnly = true; + break; + case "--update-only": + this.updateOnly = true; + break; + case "--write-only": + this.writeOnly = true; + break; + default: + validArgs = false; } } else if (args.length > 1) { validArgs = false; diff --git a/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java b/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java index ea20a5e..9196fe9 100644 --- a/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java @@ -1,5 +1,6 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import java.util.Map; @@ -8,18 +9,21 @@ public class BandwidthDocument extends Document { @SuppressWarnings("unused") private String fingerprint; + public void setFingerprint(String fingerprint) { this.fingerprint = fingerprint; } @SuppressWarnings("unused") private Map<String, GraphHistory> write_history; + public void setWriteHistory(Map<String, GraphHistory> writeHistory) { this.write_history = writeHistory; } @SuppressWarnings("unused") private Map<String, GraphHistory> read_history; + public void setReadHistory(Map<String, GraphHistory> readHistory) { this.read_history = readHistory; } diff --git a/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java b/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java index 06f295b..828fbab 100644 --- a/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java @@ -1,44 +1,52 @@ /* Copyright 2013--2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.torproject.descriptor.BandwidthHistory; +import org.torproject.onionoo.util.TimeFactory; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.Map; import java.util.Scanner; import java.util.SortedMap; import java.util.TreeMap; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.descriptor.BandwidthHistory; -import org.torproject.onionoo.util.TimeFactory; - public class BandwidthStatus extends Document { private static Logger log = LoggerFactory.getLogger( BandwidthStatus.class); private transient boolean isDirty = false; + public boolean isDirty() { return this.isDirty; } + public void clearDirty() { this.isDirty = false; } private SortedMap<Long, long[]> writeHistory = new TreeMap<Long, long[]>(); + public void setWriteHistory(SortedMap<Long, long[]> writeHistory) { this.writeHistory = writeHistory; } + public SortedMap<Long, long[]> getWriteHistory() { return this.writeHistory; } private SortedMap<Long, long[]> readHistory = new TreeMap<Long, long[]>(); + public void setReadHistory(SortedMap<Long, long[]> readHistory) { this.readHistory = readHistory; } + public SortedMap<Long, long[]> getReadHistory() { return this.readHistory; } @@ -68,8 +76,8 @@ public class BandwidthStatus extends Document { : history.get(history.headMap(startMillis).lastKey())[1]; long nextStartMillis = history.tailMap(startMillis).isEmpty() ? endMillis : history.tailMap(startMillis).firstKey(); - if (previousEndMillis <= startMillis && - nextStartMillis >= endMillis) { + if (previousEndMillis <= startMillis + && nextStartMillis >= endMillis) { history.put(startMillis, new long[] { startMillis, endMillis, bandwidth }); } @@ -111,34 +119,38 @@ public class BandwidthStatus extends Document { SortedMap<Long, long[]> uncompressedHistory = new TreeMap<Long, long[]>(history); history.clear(); - long lastStartMillis = 0L, lastEndMillis = 0L, lastBandwidth = 0L; + long lastStartMillis = 0L; + long lastEndMillis = 0L; + long lastBandwidth = 0L; String lastMonthString = "1970-01"; long now = TimeFactory.getTime().currentTimeMillis(); for (long[] v : uncompressedHistory.values()) { - long startMillis = v[0], endMillis = v[1], bandwidth = v[2]; + long startMillis = v[0]; + long endMillis = v[1]; + long bandwidth = v[2]; long intervalLengthMillis; if (now - endMillis <= DateTimeHelper.THREE_DAYS) { intervalLengthMillis = DateTimeHelper.FIFTEEN_MINUTES; } else if (now - endMillis <= DateTimeHelper.ONE_WEEK) { intervalLengthMillis = DateTimeHelper.ONE_HOUR; - } else if (now - endMillis <= - DateTimeHelper.ROUGHLY_ONE_MONTH) { + } else if (now - endMillis + <= DateTimeHelper.ROUGHLY_ONE_MONTH) { intervalLengthMillis = DateTimeHelper.FOUR_HOURS; - } else if (now - endMillis <= - DateTimeHelper.ROUGHLY_THREE_MONTHS) { + } else if (now - endMillis + <= DateTimeHelper.ROUGHLY_THREE_MONTHS) { intervalLengthMillis = DateTimeHelper.TWELVE_HOURS; - } else if (now - endMillis <= - DateTimeHelper.ROUGHLY_ONE_YEAR) { + } else if (now - endMillis + <= DateTimeHelper.ROUGHLY_ONE_YEAR) { intervalLengthMillis = DateTimeHelper.TWO_DAYS; } else { intervalLengthMillis = DateTimeHelper.TEN_DAYS; } String monthString = DateTimeHelper.format(startMillis, DateTimeHelper.ISO_YEARMONTH_FORMAT); - if (lastEndMillis == startMillis && - ((lastEndMillis - 1L) / intervalLengthMillis) == - ((endMillis - 1L) / intervalLengthMillis) && - lastMonthString.equals(monthString)) { + if (lastEndMillis == startMillis + && ((lastEndMillis - 1L) / intervalLengthMillis) + == ((endMillis - 1L) / intervalLengthMillis) + && lastMonthString.equals(monthString)) { lastEndMillis = endMillis; lastBandwidth += bandwidth; } else { diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java b/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java index 27b1588..0b4be77 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import java.util.Map; @@ -8,12 +9,14 @@ public class ClientsDocument extends Document { @SuppressWarnings("unused") private String fingerprint; + public void setFingerprint(String fingerprint) { this.fingerprint = fingerprint; } @SuppressWarnings("unused") private Map<String, ClientsGraphHistory> average_clients; + public void setAverageClients( Map<String, ClientsGraphHistory> averageClients) { this.average_clients = averageClients; diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java b/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java index e679078..a3c0ef9 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import java.util.ArrayList; @@ -9,73 +10,91 @@ import java.util.SortedMap; public class ClientsGraphHistory { private String first; + public void setFirst(long first) { this.first = DateTimeHelper.format(first); } + public long getFirst() { return DateTimeHelper.parse(this.first); } private String last; + public void setLast(long last) { this.last = DateTimeHelper.format(last); } + public long getLast() { return DateTimeHelper.parse(this.last); } private Integer interval; + public void setInterval(Integer interval) { this.interval = interval; } + public Integer getInterval() { return this.interval; } private Double factor; + public void setFactor(Double factor) { this.factor = factor; } + public Double getFactor() { return this.factor; } private Integer count; + public void setCount(Integer count) { this.count = count; } + public Integer getCount() { return this.count; } private List<Integer> values = new ArrayList<Integer>(); + public void setValues(List<Integer> values) { this.values = values; } + public List<Integer> getValues() { return this.values; } private SortedMap<String, Float> countries; + public void setCountries(SortedMap<String, Float> countries) { this.countries = countries; } + public SortedMap<String, Float> getCountries() { return this.countries; } private SortedMap<String, Float> transports; + public void setTransports(SortedMap<String, Float> transports) { this.transports = transports; } + public SortedMap<String, Float> getTransports() { return this.transports; } private SortedMap<String, Float> versions; + public void setVersions(SortedMap<String, Float> versions) { this.versions = versions; } + public SortedMap<String, Float> getVersions() { return this.versions; } diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java b/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java index effddf5..b100344 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java @@ -1,45 +1,52 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.Map; import java.util.SortedMap; import java.util.TreeMap; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - public class ClientsHistory implements Comparable<ClientsHistory> { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( ClientsHistory.class); private long startMillis; + public long getStartMillis() { return this.startMillis; } private long endMillis; + public long getEndMillis() { return this.endMillis; } private double totalResponses; + public double getTotalResponses() { return this.totalResponses; } private SortedMap<String, Double> responsesByCountry; + public SortedMap<String, Double> getResponsesByCountry() { return this.responsesByCountry; } private SortedMap<String, Double> responsesByTransport; + public SortedMap<String, Double> getResponsesByTransport() { return this.responsesByTransport; } private SortedMap<String, Double> responsesByVersion; + public SortedMap<String, Double> getResponsesByVersion() { return this.responsesByVersion; } @@ -92,8 +99,8 @@ public class ClientsHistory implements Comparable<ClientsHistory> { parseResponses(parts[6]); SortedMap<String, Double> responsesByVersion = parseResponses(parts[7]); - if (responsesByCountry == null || responsesByTransport == null || - responsesByVersion == null) { + if (responsesByCountry == null || responsesByTransport == null + || responsesByVersion == null) { log.warn("Invalid format of responses by country, transport, or " + "version in clients history: '" + responseHistoryString + "'. Skipping."); @@ -177,16 +184,17 @@ public class ClientsHistory implements Comparable<ClientsHistory> { } public int compareTo(ClientsHistory other) { - return this.startMillis < other.startMillis ? -1 : - this.startMillis > other.startMillis ? 1 : 0; + return this.startMillis < other.startMillis ? -1 + : this.startMillis > other.startMillis ? 1 : 0; } public boolean equals(Object other) { - return other instanceof ClientsHistory && - this.startMillis == ((ClientsHistory) other).startMillis; + return other instanceof ClientsHistory + && this.startMillis == ((ClientsHistory) other).startMillis; } public int hashCode() { return (int) this.startMillis; } -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java b/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java index 3cce0dd..bdd5666 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java @@ -1,14 +1,16 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; -import java.util.Scanner; -import java.util.SortedSet; -import java.util.TreeSet; +import org.torproject.onionoo.util.TimeFactory; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import org.torproject.onionoo.util.TimeFactory; + +import java.util.Scanner; +import java.util.SortedSet; +import java.util.TreeSet; public class ClientsStatus extends Document { @@ -16,18 +18,22 @@ public class ClientsStatus extends Document { ClientsStatus.class); private transient boolean isDirty = false; + public boolean isDirty() { return this.isDirty; } + public void clearDirty() { this.isDirty = false; } private SortedSet<ClientsHistory> history = new TreeSet<ClientsHistory>(); + public void setHistory(SortedSet<ClientsHistory> history) { this.history = history; } + public SortedSet<ClientsHistory> getHistory() { return this.history; } @@ -49,12 +55,12 @@ public class ClientsStatus extends Document { public void addToHistory(SortedSet<ClientsHistory> newIntervals) { for (ClientsHistory interval : newIntervals) { - if ((this.history.headSet(interval).isEmpty() || - this.history.headSet(interval).last().getEndMillis() <= - interval.getStartMillis()) && - (this.history.tailSet(interval).isEmpty() || - this.history.tailSet(interval).first().getStartMillis() >= - interval.getEndMillis())) { + if ((this.history.headSet(interval).isEmpty() + || this.history.headSet(interval).last().getEndMillis() + <= interval.getStartMillis()) + && (this.history.tailSet(interval).isEmpty() + || this.history.tailSet(interval).first().getStartMillis() + >= interval.getEndMillis())) { this.history.add(interval); this.isDirty = true; } @@ -70,11 +76,11 @@ public class ClientsStatus extends Document { long now = TimeFactory.getTime().currentTimeMillis(); for (ClientsHistory responses : uncompressedHistory) { long intervalLengthMillis; - if (now - responses.getEndMillis() <= - DateTimeHelper.ROUGHLY_THREE_MONTHS) { + if (now - responses.getEndMillis() + <= DateTimeHelper.ROUGHLY_THREE_MONTHS) { intervalLengthMillis = DateTimeHelper.ONE_DAY; - } else if (now - responses.getEndMillis() <= - DateTimeHelper.ROUGHLY_ONE_YEAR) { + } else if (now - responses.getEndMillis() + <= DateTimeHelper.ROUGHLY_ONE_YEAR) { intervalLengthMillis = DateTimeHelper.TWO_DAYS; } else { intervalLengthMillis = DateTimeHelper.TEN_DAYS; @@ -82,11 +88,11 @@ public class ClientsStatus extends Document { String monthString = DateTimeHelper.format( responses.getStartMillis(), DateTimeHelper.ISO_YEARMONTH_FORMAT); - if (lastResponses != null && - lastResponses.getEndMillis() == responses.getStartMillis() && - ((lastResponses.getEndMillis() - 1L) / intervalLengthMillis) == - ((responses.getEndMillis() - 1L) / intervalLengthMillis) && - lastMonthString.equals(monthString)) { + if (lastResponses != null + && lastResponses.getEndMillis() == responses.getStartMillis() + && ((lastResponses.getEndMillis() - 1L) / intervalLengthMillis) + == ((responses.getEndMillis() - 1L) / intervalLengthMillis) + && lastMonthString.equals(monthString)) { lastResponses.addResponses(responses); } else { if (lastResponses != null) { diff --git a/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java b/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java index 27b54a0..67a38c7 100644 --- a/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java +++ b/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java @@ -1,7 +1,11 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.text.DateFormat; import java.text.ParseException; import java.text.SimpleDateFormat; @@ -9,38 +13,53 @@ import java.util.HashMap; import java.util.Map; import java.util.TimeZone; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - public class DateTimeHelper { - public final static long NO_TIME_AVAILABLE = -1L; + public static final long NO_TIME_AVAILABLE = -1L; - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( DateTimeHelper.class); private DateTimeHelper() { } - public static final long ONE_SECOND = 1000L, - TEN_SECONDS = 10L * ONE_SECOND, - ONE_MINUTE = 60L * ONE_SECOND, - FIVE_MINUTES = 5L * ONE_MINUTE, - FIFTEEN_MINUTES = 15L * ONE_MINUTE, - FOURTY_FIVE_MINUTES = 45L * ONE_MINUTE, - ONE_HOUR = 60L * ONE_MINUTE, - FOUR_HOURS = 4L * ONE_HOUR, - SIX_HOURS = 6L * ONE_HOUR, - TWELVE_HOURS = 12L * ONE_HOUR, - ONE_DAY = 24L * ONE_HOUR, - TWO_DAYS = 2L * ONE_DAY, - THREE_DAYS = 3L * ONE_DAY, - ONE_WEEK = 7L * ONE_DAY, - TEN_DAYS = 10L * ONE_DAY, - ROUGHLY_ONE_MONTH = 31L * ONE_DAY, - ROUGHLY_THREE_MONTHS = 92L * ONE_DAY, - ROUGHLY_ONE_YEAR = 366L * ONE_DAY, - ROUGHLY_FIVE_YEARS = 5L * ROUGHLY_ONE_YEAR; + public static final long ONE_SECOND = 1000L; + + public static final long TEN_SECONDS = 10L * ONE_SECOND; + + public static final long ONE_MINUTE = 60L * ONE_SECOND; + + public static final long FIVE_MINUTES = 5L * ONE_MINUTE; + + public static final long FIFTEEN_MINUTES = 15L * ONE_MINUTE; + + public static final long FOURTY_FIVE_MINUTES = 45L * ONE_MINUTE; + + public static final long ONE_HOUR = 60L * ONE_MINUTE; + + public static final long FOUR_HOURS = 4L * ONE_HOUR; + + public static final long SIX_HOURS = 6L * ONE_HOUR; + + public static final long TWELVE_HOURS = 12L * ONE_HOUR; + + public static final long ONE_DAY = 24L * ONE_HOUR; + + public static final long TWO_DAYS = 2L * ONE_DAY; + + public static final long THREE_DAYS = 3L * ONE_DAY; + + public static final long ONE_WEEK = 7L * ONE_DAY; + + public static final long TEN_DAYS = 10L * ONE_DAY; + + public static final long ROUGHLY_ONE_MONTH = 31L * ONE_DAY; + + public static final long ROUGHLY_THREE_MONTHS = 92L * ONE_DAY; + + public static final long ROUGHLY_ONE_YEAR = 366L * ONE_DAY; + + public static final long ROUGHLY_FIVE_YEARS = 5L * ROUGHLY_ONE_YEAR; public static final String ISO_DATETIME_FORMAT = "yyyy-MM-dd HH:mm:ss"; @@ -52,16 +71,20 @@ public class DateTimeHelper { public static final String DATEHOUR_NOSPACE_FORMAT = "yyyy-MM-dd-HH"; private static ThreadLocal<Map<String, DateFormat>> dateFormats = - new ThreadLocal<Map<String, DateFormat>> () { + new ThreadLocal<Map<String, DateFormat>>() { + public Map<String, DateFormat> get() { return super.get(); } + protected Map<String, DateFormat> initialValue() { return new HashMap<String, DateFormat>(); } + public void remove() { super.remove(); } + public void set(Map<String, DateFormat> value) { super.set(value); } diff --git a/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java b/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java index 38bbc3b..a80a504 100644 --- a/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java @@ -1,14 +1,15 @@ /* Copyright 2013--2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.apache.commons.lang3.StringEscapeUtils; + import java.util.ArrayList; import java.util.List; import java.util.Map; import java.util.SortedSet; -import org.apache.commons.lang3.StringEscapeUtils; - public class DetailsDocument extends Document { /* We must ensure that details files only contain ASCII characters @@ -22,359 +23,444 @@ public class DetailsDocument extends Document { private static String escapeJSON(String s) { return StringEscapeUtils.escapeJava(s); } + private static String unescapeJSON(String s) { return StringEscapeUtils.unescapeJava(s); } private String nickname; + public void setNickname(String nickname) { this.nickname = nickname; } + public String getNickname() { return this.nickname; } private String fingerprint; + public void setFingerprint(String fingerprint) { this.fingerprint = fingerprint; } + public String getFingerprint() { return this.fingerprint; } private String hashed_fingerprint; + public void setHashedFingerprint(String hashedFingerprint) { this.hashed_fingerprint = hashedFingerprint; } + public String getHashedFingerprint() { return this.hashed_fingerprint; } private List<String> or_addresses; + public void setOrAddresses(List<String> orAddresses) { this.or_addresses = orAddresses; } + public List<String> getOrAddresses() { return this.or_addresses; } private List<String> exit_addresses; + public void setExitAddresses(List<String> exitAddresses) { this.exit_addresses = !exitAddresses.isEmpty() ? exitAddresses : null; } + public List<String> getExitAddresses() { return this.exit_addresses == null ? new ArrayList<String>() : this.exit_addresses; } private String dir_address; + public void setDirAddress(String dirAddress) { this.dir_address = dirAddress; } + public String getDirAddress() { return this.dir_address; } private String last_seen; + public void setLastSeen(long lastSeen) { this.last_seen = DateTimeHelper.format(lastSeen); } + public long getLastSeen() { return DateTimeHelper.parse(this.last_seen); } private String last_changed_address_or_port; + public void setLastChangedAddressOrPort( long lastChangedAddressOrPort) { this.last_changed_address_or_port = DateTimeHelper.format( lastChangedAddressOrPort); } + public long getLastChangedAddressOrPort() { return DateTimeHelper.parse(this.last_changed_address_or_port); } private String first_seen; + public void setFirstSeen(long firstSeen) { this.first_seen = DateTimeHelper.format(firstSeen); } + public long getFirstSeen() { return DateTimeHelper.parse(this.first_seen); } private Boolean running; + public void setRunning(Boolean running) { this.running = running; } + public Boolean getRunning() { return this.running; } private SortedSet<String> flags; + public void setFlags(SortedSet<String> flags) { this.flags = flags; } + public SortedSet<String> getFlags() { return this.flags; } private String country; + public void setCountry(String country) { this.country = country; } + public String getCountry() { return this.country; } private String country_name; + public void setCountryName(String countryName) { this.country_name = escapeJSON(countryName); } + public String getCountryName() { return unescapeJSON(this.country_name); } private String region_name; + public void setRegionName(String regionName) { this.region_name = escapeJSON(regionName); } + public String getRegionName() { return unescapeJSON(this.region_name); } private String city_name; + public void setCityName(String cityName) { this.city_name = escapeJSON(cityName); } + public String getCityName() { return unescapeJSON(this.city_name); } private Float latitude; + public void setLatitude(Float latitude) { this.latitude = latitude; } + public Float getLatitude() { return this.latitude; } private Float longitude; + public void setLongitude(Float longitude) { this.longitude = longitude; } + public Float getLongitude() { return this.longitude; } private String as_number; + public void setAsNumber(String asNumber) { this.as_number = escapeJSON(asNumber); } + public String getAsNumber() { return unescapeJSON(this.as_number); } private String as_name; + public void setAsName(String asName) { this.as_name = escapeJSON(asName); } + public String getAsName() { return unescapeJSON(this.as_name); } private Long consensus_weight; + public void setConsensusWeight(Long consensusWeight) { this.consensus_weight = consensusWeight; } + public Long getConsensusWeight() { return this.consensus_weight; } private String host_name; + public void setHostName(String hostName) { this.host_name = escapeJSON(hostName); } + public String getHostName() { return unescapeJSON(this.host_name); } private String last_restarted; + public void setLastRestarted(Long lastRestarted) { - this.last_restarted = (lastRestarted == null ? null : - DateTimeHelper.format(lastRestarted)); + this.last_restarted = (lastRestarted == null ? null + : DateTimeHelper.format(lastRestarted)); } + public Long getLastRestarted() { return this.last_restarted == null ? null : DateTimeHelper.parse(this.last_restarted); } private Integer bandwidth_rate; + public void setBandwidthRate(Integer bandwidthRate) { this.bandwidth_rate = bandwidthRate; } + public Integer getBandwidthRate() { return this.bandwidth_rate; } private Integer bandwidth_burst; + public void setBandwidthBurst(Integer bandwidthBurst) { this.bandwidth_burst = bandwidthBurst; } + public Integer getBandwidthBurst() { return this.bandwidth_burst; } private Integer observed_bandwidth; + public void setObservedBandwidth(Integer observedBandwidth) { this.observed_bandwidth = observedBandwidth; } + public Integer getObservedBandwidth() { return this.observed_bandwidth; } private Integer advertised_bandwidth; + public void setAdvertisedBandwidth(Integer advertisedBandwidth) { this.advertised_bandwidth = advertisedBandwidth; } + public Integer getAdvertisedBandwidth() { return this.advertised_bandwidth; } private List<String> exit_policy; + public void setExitPolicy(List<String> exitPolicy) { this.exit_policy = exitPolicy; } + public List<String> getExitPolicy() { return this.exit_policy; } private Map<String, List<String>> exit_policy_summary; + public void setExitPolicySummary( Map<String, List<String>> exitPolicySummary) { this.exit_policy_summary = exitPolicySummary; } + public Map<String, List<String>> getExitPolicySummary() { return this.exit_policy_summary; } private Map<String, List<String>> exit_policy_v6_summary; + public void setExitPolicyV6Summary( Map<String, List<String>> exitPolicyV6Summary) { this.exit_policy_v6_summary = exitPolicyV6Summary; } + public Map<String, List<String>> getExitPolicyV6Summary() { return this.exit_policy_v6_summary; } private String contact; + public void setContact(String contact) { this.contact = escapeJSON(contact); } + public String getContact() { return unescapeJSON(this.contact); } private String platform; + public void setPlatform(String platform) { this.platform = escapeJSON(platform); } + public String getPlatform() { return unescapeJSON(this.platform); } private SortedSet<String> alleged_family; + public void setAllegedFamily(SortedSet<String> allegedFamily) { this.alleged_family = allegedFamily; } + public SortedSet<String> getAllegedFamily() { return this.alleged_family; } private SortedSet<String> effective_family; + public void setEffectiveFamily(SortedSet<String> effectiveFamily) { this.effective_family = effectiveFamily; } + public SortedSet<String> getEffectiveFamily() { return this.effective_family; } private SortedSet<String> indirect_family; + public void setIndirectFamily(SortedSet<String> indirectFamily) { this.indirect_family = indirectFamily; } + public SortedSet<String> getIndirectFamily() { return this.indirect_family; } private Float consensus_weight_fraction; + public void setConsensusWeightFraction(Float consensusWeightFraction) { - if (consensusWeightFraction == null || - consensusWeightFraction >= 0.0) { + if (consensusWeightFraction == null + || consensusWeightFraction >= 0.0) { this.consensus_weight_fraction = consensusWeightFraction; } } + public Float getConsensusWeightFraction() { return this.consensus_weight_fraction; } private Float guard_probability; + public void setGuardProbability(Float guardProbability) { if (guardProbability == null || guardProbability >= 0.0) { this.guard_probability = guardProbability; } } + public Float getGuardProbability() { return this.guard_probability; } private Float middle_probability; + public void setMiddleProbability(Float middleProbability) { if (middleProbability == null || middleProbability >= 0.0) { this.middle_probability = middleProbability; } } + public Float getMiddleProbability() { return this.middle_probability; } private Float exit_probability; + public void setExitProbability(Float exitProbability) { if (exitProbability == null || exitProbability >= 0.0) { this.exit_probability = exitProbability; } } + public Float getExitProbability() { return this.exit_probability; } private Boolean recommended_version; + public void setRecommendedVersion(Boolean recommendedVersion) { this.recommended_version = recommendedVersion; } + public Boolean getRecommendedVersion() { return this.recommended_version; } private Boolean hibernating; + public void setHibernating(Boolean hibernating) { this.hibernating = hibernating; } + public Boolean getHibernating() { return this.hibernating; } private List<String> transports; + public void setTransports(List<String> transports) { - this.transports = (transports != null && !transports.isEmpty()) ? - transports : null; + this.transports = (transports != null && !transports.isEmpty()) + ? transports : null; } + public List<String> getTransports() { return this.transports; } private Boolean measured; + public void setMeasured(Boolean measured) { this.measured = measured; } + public Boolean getMeasured() { return this.measured; } diff --git a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java index 460fb97..599bb59 100644 --- a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java @@ -1,14 +1,15 @@ /* Copyright 2013--2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.apache.commons.lang3.StringEscapeUtils; + import java.util.List; import java.util.Map; import java.util.SortedSet; import java.util.TreeSet; -import org.apache.commons.lang3.StringEscapeUtils; - public class DetailsStatus extends Document { /* We must ensure that details files only contain ASCII characters @@ -22,6 +23,7 @@ public class DetailsStatus extends Document { private static String escapeJSON(String s) { return StringEscapeUtils.escapeJava(s); } + private static String unescapeJSON(String s) { return StringEscapeUtils.unescapeJava(s); } @@ -29,116 +31,144 @@ public class DetailsStatus extends Document { /* From most recently published server descriptor: */ private String desc_published; + public void setDescPublished(Long descPublished) { this.desc_published = DateTimeHelper.format(descPublished); } + public Long getDescPublished() { return this.desc_published == null ? null : DateTimeHelper.parse(this.desc_published); } private String last_restarted; + public void setLastRestarted(Long lastRestarted) { this.last_restarted = DateTimeHelper.format(lastRestarted); } + public Long getLastRestarted() { return this.last_restarted == null ? null : DateTimeHelper.parse(this.last_restarted); } private Integer bandwidth_rate; + public void setBandwidthRate(Integer bandwidthRate) { this.bandwidth_rate = bandwidthRate; } + public Integer getBandwidthRate() { return this.bandwidth_rate; } private Integer bandwidth_burst; + public void setBandwidthBurst(Integer bandwidthBurst) { this.bandwidth_burst = bandwidthBurst; } + public Integer getBandwidthBurst() { return this.bandwidth_burst; } private Integer observed_bandwidth; + public void setObservedBandwidth(Integer observedBandwidth) { this.observed_bandwidth = observedBandwidth; } + public Integer getObservedBandwidth() { return this.observed_bandwidth; } private Integer advertised_bandwidth; + public void setAdvertisedBandwidth(Integer advertisedBandwidth) { this.advertised_bandwidth = advertisedBandwidth; } + public Integer getAdvertisedBandwidth() { return this.advertised_bandwidth; } private List<String> exit_policy; + public void setExitPolicy(List<String> exitPolicy) { this.exit_policy = exitPolicy; } + public List<String> getExitPolicy() { return this.exit_policy; } private String contact; + public void setContact(String contact) { this.contact = escapeJSON(contact); } + public String getContact() { return unescapeJSON(this.contact); } private String platform; + public void setPlatform(String platform) { this.platform = escapeJSON(platform); } + public String getPlatform() { return unescapeJSON(this.platform); } private SortedSet<String> alleged_family; + public void setAllegedFamily(SortedSet<String> allegedFamily) { this.alleged_family = allegedFamily; } + public SortedSet<String> getAllegedFamily() { return this.alleged_family; } private SortedSet<String> effective_family; + public void setEffectiveFamily(SortedSet<String> effectiveFamily) { this.effective_family = effectiveFamily; } + public SortedSet<String> getEffectiveFamily() { return this.effective_family; } private SortedSet<String> indirect_family; + public void setIndirectFamily(SortedSet<String> indirectFamily) { this.indirect_family = indirectFamily; } + public SortedSet<String> getIndirectFamily() { return this.indirect_family; } private Map<String, List<String>> exit_policy_v6_summary; + public void setExitPolicyV6Summary( Map<String, List<String>> exitPolicyV6Summary) { this.exit_policy_v6_summary = exitPolicyV6Summary; } + public Map<String, List<String>> getExitPolicyV6Summary() { return this.exit_policy_v6_summary; } private Boolean hibernating; + public void setHibernating(Boolean hibernating) { this.hibernating = hibernating; } + public Boolean getHibernating() { return this.hibernating; } @@ -146,18 +176,22 @@ public class DetailsStatus extends Document { /* From most recently published extra-info descriptor: */ private Long extra_info_desc_published; + public void setExtraInfoDescPublished(Long extraInfoDescPublished) { this.extra_info_desc_published = extraInfoDescPublished; } + public Long getExtraInfoDescPublished() { return this.extra_info_desc_published; } private List<String> transports; + public void setTransports(List<String> transports) { - this.transports = (transports != null && !transports.isEmpty()) ? - transports : null; + this.transports = (transports != null && !transports.isEmpty()) + ? transports : null; } + public List<String> getTransports() { return this.transports; } @@ -165,46 +199,57 @@ public class DetailsStatus extends Document { /* From network status entries: */ private boolean is_relay; + public void setRelay(boolean isRelay) { this.is_relay = isRelay; } + public boolean isRelay() { return this.is_relay; } private boolean running; + public void setRunning(boolean isRunning) { this.running = isRunning; } + public boolean isRunning() { return this.running; } private String nickname; + public void setNickname(String nickname) { this.nickname = nickname; } + public String getNickname() { return this.nickname; } private String address; + public void setAddress(String address) { this.address = address; } + public String getAddress() { return this.address; } private SortedSet<String> or_addresses_and_ports; + public void setOrAddressesAndPorts( SortedSet<String> orAddressesAndPorts) { this.or_addresses_and_ports = orAddressesAndPorts; } + public SortedSet<String> getOrAddressesAndPorts() { return this.or_addresses_and_ports == null ? new TreeSet<String>() : this.or_addresses_and_ports; } + public SortedSet<String> getOrAddresses() { SortedSet<String> orAddresses = new TreeSet<String>(); if (this.address != null) { @@ -223,90 +268,112 @@ public class DetailsStatus extends Document { } private long first_seen_millis; + public void setFirstSeenMillis(long firstSeenMillis) { this.first_seen_millis = firstSeenMillis; } + public long getFirstSeenMillis() { return this.first_seen_millis; } private long last_seen_millis; + public void setLastSeenMillis(long lastSeenMillis) { this.last_seen_millis = lastSeenMillis; } + public long getLastSeenMillis() { return this.last_seen_millis; } private int or_port; + public void setOrPort(int orPort) { this.or_port = orPort; } + public int getOrPort() { return this.or_port; } private int dir_port; + public void setDirPort(int dirPort) { this.dir_port = dirPort; } + public int getDirPort() { return this.dir_port; } private SortedSet<String> relay_flags; + public void setRelayFlags(SortedSet<String> relayFlags) { this.relay_flags = relayFlags; } + public SortedSet<String> getRelayFlags() { return this.relay_flags; } private long consensus_weight; + public void setConsensusWeight(long consensusWeight) { this.consensus_weight = consensusWeight; } + public long getConsensusWeight() { return this.consensus_weight; } private String default_policy; + public void setDefaultPolicy(String defaultPolicy) { this.default_policy = defaultPolicy; } + public String getDefaultPolicy() { return this.default_policy; } private String port_list; + public void setPortList(String portList) { this.port_list = portList; } + public String getPortList() { return this.port_list; } private long last_changed_or_address_or_port; + public void setLastChangedOrAddressOrPort( long lastChangedOrAddressOrPort) { this.last_changed_or_address_or_port = lastChangedOrAddressOrPort; } + public long getLastChangedOrAddressOrPort() { return this.last_changed_or_address_or_port; } private Boolean recommended_version; + public void setRecommendedVersion(Boolean recommendedVersion) { this.recommended_version = recommendedVersion; } + public Boolean getRecommendedVersion() { return this.recommended_version; } private Boolean measured; + public void setMeasured(Boolean measured) { this.measured = measured; } + public Boolean getMeasured() { return this.measured; } @@ -314,9 +381,11 @@ public class DetailsStatus extends Document { /* From exit lists: */ private Map<String, Long> exit_addresses; + public void setExitAddresses(Map<String, Long> exitAddresses) { this.exit_addresses = exitAddresses; } + public Map<String, Long> getExitAddresses() { return this.exit_addresses; } @@ -324,33 +393,41 @@ public class DetailsStatus extends Document { /* Calculated path-selection probabilities: */ private Float consensus_weight_fraction; + public void setConsensusWeightFraction(Float consensusWeightFraction) { this.consensus_weight_fraction = consensusWeightFraction; } + public Float getConsensusWeightFraction() { return this.consensus_weight_fraction; } private Float guard_probability; + public void setGuardProbability(Float guardProbability) { this.guard_probability = guardProbability; } + public Float getGuardProbability() { return this.guard_probability; } private Float middle_probability; + public void setMiddleProbability(Float middleProbability) { this.middle_probability = middleProbability; } + public Float getMiddleProbability() { return this.middle_probability; } private Float exit_probability; + public void setExitProbability(Float exitProbability) { this.exit_probability = exitProbability; } + public Float getExitProbability() { return this.exit_probability; } @@ -358,65 +435,81 @@ public class DetailsStatus extends Document { /* GeoIP lookup results: */ private Float latitude; + public void setLatitude(Float latitude) { this.latitude = latitude; } + public Float getLatitude() { return this.latitude; } private Float longitude; + public void setLongitude(Float longitude) { this.longitude = longitude; } + public Float getLongitude() { return this.longitude; } private String country_code; + public void setCountryCode(String countryCode) { this.country_code = countryCode; } + public String getCountryCode() { return this.country_code; } private String country_name; + public void setCountryName(String countryName) { this.country_name = escapeJSON(countryName); } + public String getCountryName() { return unescapeJSON(this.country_name); } private String region_name; + public void setRegionName(String regionName) { this.region_name = escapeJSON(regionName); } + public String getRegionName() { return unescapeJSON(this.region_name); } private String city_name; + public void setCityName(String cityName) { this.city_name = escapeJSON(cityName); } + public String getCityName() { return unescapeJSON(this.city_name); } private String as_name; + public void setASName(String aSName) { this.as_name = escapeJSON(aSName); } + public String getASName() { return unescapeJSON(this.as_name); } private String as_number; + public void setASNumber(String aSNumber) { this.as_number = escapeJSON(aSNumber); } + public String getASNumber() { return unescapeJSON(this.as_number); } @@ -424,10 +517,13 @@ public class DetailsStatus extends Document { /* Reverse DNS lookup result: */ private String host_name; + public void setHostName(String hostName) { this.host_name = escapeJSON(hostName); } + public String getHostName() { return unescapeJSON(this.host_name); } } + diff --git a/src/main/java/org/torproject/onionoo/docs/Document.java b/src/main/java/org/torproject/onionoo/docs/Document.java index 71068ff..4c3a7c3 100644 --- a/src/main/java/org/torproject/onionoo/docs/Document.java +++ b/src/main/java/org/torproject/onionoo/docs/Document.java @@ -1,13 +1,16 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; public abstract class Document { private transient String documentString; + public void setDocumentString(String documentString) { this.documentString = documentString; } + public String getDocumentString() { return this.documentString; } diff --git a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java index 8e2162c..8b71427 100644 --- a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java +++ b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java @@ -1,7 +1,20 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.torproject.onionoo.util.FormattingUtils; +import org.torproject.onionoo.util.Time; +import org.torproject.onionoo.util.TimeFactory; + +import com.google.gson.Gson; +import com.google.gson.GsonBuilder; +import com.google.gson.JsonParseException; + +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.BufferedInputStream; import java.io.BufferedOutputStream; import java.io.BufferedReader; @@ -21,17 +34,6 @@ import java.util.Stack; import java.util.TreeMap; import java.util.TreeSet; -import org.apache.commons.lang3.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.onionoo.util.FormattingUtils; -import org.torproject.onionoo.util.Time; -import org.torproject.onionoo.util.TimeFactory; - -import com.google.gson.Gson; -import com.google.gson.GsonBuilder; -import com.google.gson.JsonParseException; - // TODO For later migration from disk to database, do the following: // - read from database and then from disk if not found // - write only to database, delete from disk once in database @@ -45,6 +47,7 @@ public class DocumentStore { private final File statusDir = new File("status"); private File outDir = null; + public void setOutDir(File outDir) { this.outDir = outDir; } @@ -55,25 +58,38 @@ public class DocumentStore { this.time = TimeFactory.getTime(); } - private long listOperations = 0L, listedFiles = 0L, storedFiles = 0L, - storedBytes = 0L, retrievedFiles = 0L, retrievedBytes = 0L, - removedFiles = 0L; + private long listOperations = 0L; + + private long listedFiles = 0L; + + private long storedFiles = 0L; + + private long storedBytes = 0L; + + private long retrievedFiles = 0L; + + private long retrievedBytes = 0L; + + private long removedFiles = 0L; /* Node statuses and summary documents are cached in memory, as opposed * to all other document types. These caches are initialized when first * accessing or modifying a NodeStatus or SummaryDocument document, * respectively. */ private SortedMap<String, NodeStatus> cachedNodeStatuses; + private SortedMap<String, SummaryDocument> cachedSummaryDocuments; /* Last-modified timestamp of cached network statuses and summary * documents when reading them from disk. */ private long lastModifiedNodeStatuses = 0L; + private long lastModifiedSummaryDocuments = 0L; /* Fingerprints of updated node statuses and summary documents that are * not yet written to disk. */ private SortedSet<String> updatedNodeStatuses; + private SortedSet<String> updatedSummaryDocuments; public <T extends Document> SortedSet<String> list( @@ -226,8 +242,9 @@ public class DocumentStore { File file = files.pop(); if (file.isDirectory()) { files.addAll(Arrays.asList(file.listFiles())); - } else if (file.getName().length() == 40 && - (updatedAfter == 0L || file.lastModified() > updatedAfter)) { + } else if (file.getName().length() == 40 + && (updatedAfter == 0L + || file.lastModified() > updatedAfter)) { fingerprints.add(file.getName()); } } @@ -273,9 +290,11 @@ public class DocumentStore { return true; } - private static final long ONE_BYTE = 1L, - ONE_KIBIBYTE = 1024L * ONE_BYTE, - ONE_MIBIBYTE = 1024L * ONE_KIBIBYTE; + private static final long ONE_BYTE = 1L; + + private static final long ONE_KIBIBYTE = 1024L * ONE_BYTE; + + private static final long ONE_MIBIBYTE = 1024L * ONE_KIBIBYTE; private <T extends Document> boolean storeDocumentFile(T document, String fingerprint) { @@ -287,14 +306,14 @@ public class DocumentStore { String documentString; if (document.getDocumentString() != null) { documentString = document.getDocumentString(); - } else if (document instanceof BandwidthDocument || - document instanceof WeightsDocument || - document instanceof ClientsDocument || - document instanceof UptimeDocument) { + } else if (document instanceof BandwidthDocument + || document instanceof WeightsDocument + || document instanceof ClientsDocument + || document instanceof UptimeDocument) { Gson gson = new Gson(); documentString = gson.toJson(document); - } else if (document instanceof DetailsStatus || - document instanceof DetailsDocument) { + } else if (document instanceof DetailsStatus + || document instanceof DetailsDocument) { /* Don't escape HTML characters, like < and >, contained in * strings. */ Gson gson = new GsonBuilder().disableHtmlEscaping().create(); @@ -310,16 +329,16 @@ public class DocumentStore { /* Existing details statuses don't contain opening and closing curly * brackets, so we should remove them from new details statuses, * too. */ - if (document instanceof DetailsStatus) { - documentString = documentString.substring( - documentString.indexOf("{") + 1, - documentString.lastIndexOf("}")); - } - } else if (document instanceof BandwidthStatus || - document instanceof WeightsStatus || - document instanceof ClientsStatus || - document instanceof UptimeStatus || - document instanceof UpdateStatus) { + if (document instanceof DetailsStatus) { + documentString = documentString.substring( + documentString.indexOf("{") + 1, + documentString.lastIndexOf("}")); + } + } else if (document instanceof BandwidthStatus + || document instanceof WeightsStatus + || document instanceof ClientsStatus + || document instanceof UptimeStatus + || document instanceof UpdateStatus) { documentString = document.toDocumentString(); } else { log.error("Serializing is not supported for type " @@ -394,7 +413,9 @@ public class DocumentStore { boolean running = false; String nickname = detailsDocument.getNickname(); List<String> addresses = new ArrayList<String>(); - String countryCode = null, aSNumber = null, contact = null; + String countryCode = null; + String aSNumber = null; + String contact = null; for (String orAddressAndPort : detailsDocument.getOrAddresses()) { if (!orAddressAndPort.contains(":")) { log.warn("Attempt to create summary document from details " @@ -416,9 +437,11 @@ public class DocumentStore { } } } - SortedSet<String> relayFlags = new TreeSet<String>(), family = null; - long lastSeenMillis = -1L, consensusWeight = -1L, - firstSeenMillis = -1L; + SortedSet<String> relayFlags = new TreeSet<String>(); + SortedSet<String> family = null; + long lastSeenMillis = -1L; + long consensusWeight = -1L; + long firstSeenMillis = -1L; SummaryDocument summaryDocument = new SummaryDocument(isRelay, nickname, fingerprint, addresses, lastSeenMillis, running, relayFlags, consensusWeight, countryCode, firstSeenMillis, @@ -469,18 +492,18 @@ public class DocumentStore { if (!parse) { return this.retrieveUnparsedDocumentFile(documentType, documentString); - } else if (documentType.equals(DetailsDocument.class) || - documentType.equals(BandwidthDocument.class) || - documentType.equals(WeightsDocument.class) || - documentType.equals(ClientsDocument.class) || - documentType.equals(UptimeDocument.class)) { + } else if (documentType.equals(DetailsDocument.class) + || documentType.equals(BandwidthDocument.class) + || documentType.equals(WeightsDocument.class) + || documentType.equals(ClientsDocument.class) + || documentType.equals(UptimeDocument.class)) { return this.retrieveParsedDocumentFile(documentType, documentString); - } else if (documentType.equals(BandwidthStatus.class) || - documentType.equals(WeightsStatus.class) || - documentType.equals(ClientsStatus.class) || - documentType.equals(UptimeStatus.class) || - documentType.equals(UpdateStatus.class)) { + } else if (documentType.equals(BandwidthStatus.class) + || documentType.equals(WeightsStatus.class) + || documentType.equals(ClientsStatus.class) + || documentType.equals(UptimeStatus.class) + || documentType.equals(UpdateStatus.class)) { return this.retrieveParsedStatusFile(documentType, documentString); } else if (documentType.equals(DetailsStatus.class)) { return this.retrieveParsedDocumentFile(documentType, "{" @@ -595,8 +618,8 @@ public class DocumentStore { private <T extends Document> File getDocumentFile(Class<T> documentType, String fingerprint) { File documentFile = null; - if (fingerprint == null && !documentType.equals(UpdateStatus.class) && - !documentType.equals(UptimeStatus.class)) { + if (fingerprint == null && !documentType.equals(UpdateStatus.class) + && !documentType.equals(UptimeStatus.class)) { log.warn("Attempted to locate a document file of type " + documentType.getName() + " without providing a fingerprint. " + "Such a file does not exist."); @@ -663,8 +686,8 @@ public class DocumentStore { * containing current time. It's important to write the update file * now, not earlier, because the front-end should not read new node * statuses until all details, bandwidths, and weights are ready. */ - if (this.cachedNodeStatuses != null || - this.cachedSummaryDocuments != null) { + if (this.cachedNodeStatuses != null + || this.cachedSummaryDocuments != null) { if (this.cachedNodeStatuses != null) { this.writeNodeStatuses(); } @@ -692,9 +715,10 @@ public class DocumentStore { return; } File summaryFile = new File(directory, "summary"); - SortedMap<String, NodeStatus> - cachedRelays = new TreeMap<String, NodeStatus>(), - cachedBridges = new TreeMap<String, NodeStatus>(); + SortedMap<String, NodeStatus> cachedRelays = + new TreeMap<String, NodeStatus>(); + SortedMap<String, NodeStatus> cachedBridges = + new TreeMap<String, NodeStatus>(); for (Map.Entry<String, NodeStatus> e : this.cachedNodeStatuses.entrySet()) { if (e.getValue().isRelay()) { diff --git a/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java b/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java index ab62231..40916a7 100644 --- a/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java +++ b/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java @@ -1,13 +1,16 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; public class DocumentStoreFactory { private static DocumentStore documentStoreInstance; + public static void setDocumentStore(DocumentStore documentStore) { documentStoreInstance = documentStore; } + public static DocumentStore getDocumentStore() { if (documentStoreInstance == null) { documentStoreInstance = new DocumentStore(); diff --git a/src/main/java/org/torproject/onionoo/docs/GraphHistory.java b/src/main/java/org/torproject/onionoo/docs/GraphHistory.java index 115f9f9..165d059 100644 --- a/src/main/java/org/torproject/onionoo/docs/GraphHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/GraphHistory.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import java.util.List; @@ -7,50 +8,63 @@ import java.util.List; public class GraphHistory { private String first; + public void setFirst(long first) { this.first = DateTimeHelper.format(first); } + public long getFirst() { return DateTimeHelper.parse(this.first); } private String last; + public void setLast(long last) { this.last = DateTimeHelper.format(last); } + public long getLast() { return DateTimeHelper.parse(this.last); } private Integer interval; + public void setInterval(Integer interval) { this.interval = interval; } + public Integer getInterval() { return this.interval; } private Double factor; + public void setFactor(Double factor) { this.factor = factor; } + public Double getFactor() { return this.factor; } private Integer count; + public void setCount(Integer count) { this.count = count; } + public Integer getCount() { return this.count; } private List<Integer> values; + public void setValues(List<Integer> values) { this.values = values; } + public List<Integer> getValues() { return this.values; } } + diff --git a/src/main/java/org/torproject/onionoo/docs/NodeStatus.java b/src/main/java/org/torproject/onionoo/docs/NodeStatus.java index 6e51190..46cb946 100644 --- a/src/main/java/org/torproject/onionoo/docs/NodeStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/NodeStatus.java @@ -1,7 +1,12 @@ /* Copyright 2011, 2012 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.Arrays; import java.util.BitSet; import java.util.Collection; @@ -15,18 +20,15 @@ import java.util.SortedSet; import java.util.TreeMap; import java.util.TreeSet; -import org.apache.commons.lang3.StringUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - public class NodeStatus extends Document { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( NodeStatus.class); /* From most recently published server descriptor: */ private String contact; + public void setContact(String contact) { if (contact == null) { this.contact = null; @@ -42,14 +44,17 @@ public class NodeStatus extends Document { this.contact = sb.toString(); } } + public String getContact() { return this.contact; } private String[] declaredFamily; + public void setDeclaredFamily(SortedSet<String> declaredFamily) { this.declaredFamily = collectionToStringArray(declaredFamily); } + public SortedSet<String> getDeclaredFamily() { return stringArrayToSortedSet(this.declaredFamily); } @@ -66,6 +71,7 @@ public class NodeStatus extends Document { } return stringArray; } + private SortedSet<String> stringArrayToSortedSet(String[] stringArray) { SortedSet<String> sortedSet = new TreeSet<String>(); if (stringArray != null) { @@ -77,43 +83,53 @@ public class NodeStatus extends Document { /* From network status entries: */ private boolean isRelay; + public void setRelay(boolean isRelay) { this.isRelay = isRelay; } + public boolean isRelay() { return this.isRelay; } private final String fingerprint; + public String getFingerprint() { return this.fingerprint; } private String nickname; + public void setNickname(String nickname) { this.nickname = nickname; } + public String getNickname() { return this.nickname; } private String address; + public void setAddress(String address) { this.address = address; } + public String getAddress() { return this.address; } private SortedSet<String> orAddressesAndPorts; + public void setOrAddressesAndPorts( SortedSet<String> orAddressesAndPorts) { this.orAddressesAndPorts = orAddressesAndPorts; } + public SortedSet<String> getOrAddressesAndPorts() { - return this.orAddressesAndPorts == null ? new TreeSet<String>() : - this.orAddressesAndPorts; + return this.orAddressesAndPorts == null ? new TreeSet<String>() + : this.orAddressesAndPorts; } + public SortedSet<String> getOrAddresses() { SortedSet<String> orAddresses = new TreeSet<String>(); if (this.address != null) { @@ -132,43 +148,53 @@ public class NodeStatus extends Document { } private long firstSeenMillis; + public void setFirstSeenMillis(long firstSeenMillis) { this.firstSeenMillis = firstSeenMillis; } + public long getFirstSeenMillis() { return this.firstSeenMillis; } private long lastSeenMillis; + public void setLastSeenMillis(long lastSeenMillis) { this.lastSeenMillis = lastSeenMillis; } + public long getLastSeenMillis() { return this.lastSeenMillis; } private int orPort; + public void setOrPort(int orPort) { this.orPort = orPort; } + public int getOrPort() { return this.orPort; } private int dirPort; + public void setDirPort(int dirPort) { this.dirPort = dirPort; } + public int getDirPort() { return this.dirPort; } private static Map<String, Integer> relayFlagIndexes = new HashMap<String, Integer>(); + private static Map<Integer, String> relayFlagStrings = new HashMap<Integer, String>(); private BitSet relayFlags; + public void setRelayFlags(SortedSet<String> relayFlags) { BitSet newRelayFlags = new BitSet(relayFlagIndexes.size()); for (String relayFlag : relayFlags) { @@ -180,6 +206,7 @@ public class NodeStatus extends Document { } this.relayFlags = newRelayFlags; } + public SortedSet<String> getRelayFlags() { SortedSet<String> result = new TreeSet<String>(); if (this.relayFlags != null) { @@ -192,34 +219,42 @@ public class NodeStatus extends Document { } private long consensusWeight; + public void setConsensusWeight(long consensusWeight) { this.consensusWeight = consensusWeight; } + public long getConsensusWeight() { return this.consensusWeight; } private String defaultPolicy; + public void setDefaultPolicy(String defaultPolicy) { this.defaultPolicy = defaultPolicy; } + public String getDefaultPolicy() { return this.defaultPolicy; } private String portList; + public void setPortList(String portList) { this.portList = portList; } + public String getPortList() { return this.portList; } private SortedMap<Long, Set<String>> lastAddresses = new TreeMap<Long, Set<String>>(Collections.reverseOrder()); + public SortedMap<Long, Set<String>> getLastAddresses() { return new TreeMap<Long, Set<String>>(this.lastAddresses); } + public void addLastAddresses(long lastSeenMillis, String address, int orPort, int dirPort, SortedSet<String> orAddressesAndPorts) { Set<String> addressesAndPorts = new HashSet<String>(); @@ -234,6 +269,7 @@ public class NodeStatus extends Document { this.lastAddresses.put(lastSeenMillis, addressesAndPorts); } } + public long getLastChangedOrAddressOrPort() { long lastChangedAddressesMillis = -1L; if (this.lastAddresses != null) { @@ -255,9 +291,11 @@ public class NodeStatus extends Document { } private Boolean recommendedVersion; + public void setRecommendedVersion(Boolean recommendedVersion) { this.recommendedVersion = recommendedVersion; } + public Boolean getRecommendedVersion() { return this.recommendedVersion; } @@ -265,9 +303,11 @@ public class NodeStatus extends Document { /* From exit lists: */ private SortedSet<String> exitAddresses; + public void setExitAddresses(SortedSet<String> exitAddresses) { this.exitAddresses = exitAddresses; } + public SortedSet<String> getExitAddresses() { return new TreeSet<String>(this.exitAddresses); } @@ -275,17 +315,21 @@ public class NodeStatus extends Document { /* GeoIP lookup results: */ private String countryCode; + public void setCountryCode(String countryCode) { this.countryCode = countryCode; } + public String getCountryCode() { return this.countryCode; } private String aSNumber; + public void setASNumber(String aSNumber) { this.aSNumber = aSNumber; } + public String getASNumber() { return this.aSNumber; } @@ -293,9 +337,11 @@ public class NodeStatus extends Document { /* Reverse DNS lookup result */ private long lastRdnsLookup = -1L; + public void setLastRdnsLookup(long lastRdnsLookup) { this.lastRdnsLookup = lastRdnsLookup; } + public long getLastRdnsLookup() { return this.lastRdnsLookup; } @@ -304,17 +350,21 @@ public class NodeStatus extends Document { * and indirect family */ private String[] effectiveFamily; + public void setEffectiveFamily(SortedSet<String> effectiveFamily) { this.effectiveFamily = collectionToStringArray(effectiveFamily); } + public SortedSet<String> getEffectiveFamily() { return stringArrayToSortedSet(this.effectiveFamily); } private String[] extendedFamily; + public void setExtendedFamily(SortedSet<String> extendedFamily) { this.extendedFamily = collectionToStringArray(extendedFamily); } + public SortedSet<String> getExtendedFamily() { return stringArrayToSortedSet(this.extendedFamily); } @@ -353,7 +403,8 @@ public class NodeStatus extends Document { nodeStatus.setNickname(parts[1]); SortedSet<String> orAddressesAndPorts = new TreeSet<String>(); SortedSet<String> exitAddresses = new TreeSet<String>(); - String addresses = parts[3], address = null; + String addresses = parts[3]; + String address = null; if (addresses.contains(";")) { String[] addressParts = addresses.split(";", -1); if (addressParts.length != 3) { @@ -384,8 +435,8 @@ public class NodeStatus extends Document { return null; } nodeStatus.setLastSeenMillis(lastSeenMillis); - int orPort = Integer.parseInt(parts[6]), - dirPort = Integer.parseInt(parts[7]); + int orPort = Integer.parseInt(parts[6]); + int dirPort = Integer.parseInt(parts[7]); nodeStatus.setOrPort(orPort); nodeStatus.setDirPort(dirPort); nodeStatus.setRelayFlags(new TreeSet<String>( @@ -440,9 +491,9 @@ public class NodeStatus extends Document { * 2. indirect members that can be reached via others only. * Each group contains zero or more ';'-separated fingerprints. */ String[] groups = parts[22].split(":", -1); - SortedSet<String> allegedFamily = new TreeSet<String>(), - effectiveFamily = new TreeSet<String>(), - indirectFamily = new TreeSet<String>(); + SortedSet<String> allegedFamily = new TreeSet<String>(); + SortedSet<String> effectiveFamily = new TreeSet<String>(); + SortedSet<String> indirectFamily = new TreeSet<String>(); if (groups[0].length() > 0) { allegedFamily.addAll(Arrays.asList(groups[0].split(";"))); } diff --git a/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java b/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java index 0c217c1..d8d82ce 100644 --- a/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java @@ -1,7 +1,13 @@ /* Copyright 2013--2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.apache.commons.codec.DecoderException; +import org.apache.commons.codec.binary.Base64; +import org.apache.commons.codec.binary.Hex; +import org.apache.commons.codec.digest.DigestUtils; + import java.util.ArrayList; import java.util.Arrays; import java.util.Collection; @@ -10,22 +16,20 @@ import java.util.SortedSet; import java.util.TreeSet; import java.util.regex.Pattern; -import org.apache.commons.codec.DecoderException; -import org.apache.commons.codec.binary.Base64; -import org.apache.commons.codec.binary.Hex; -import org.apache.commons.codec.digest.DigestUtils; - public class SummaryDocument extends Document { private boolean t; + public void setRelay(boolean isRelay) { this.t = isRelay; } + public boolean isRelay() { return this.t; } private String f; + public void setFingerprint(String fingerprint) { if (fingerprint != null) { Pattern fingerprintPattern = Pattern.compile("^[0-9a-fA-F]{40}$"); @@ -38,11 +42,13 @@ public class SummaryDocument extends Document { this.hashedFingerprint = null; this.base64Fingerprint = null; } + public String getFingerprint() { return this.f; } private transient String hashedFingerprint = null; + public String getHashedFingerprint() { if (this.hashedFingerprint == null && this.f != null) { try { @@ -56,6 +62,7 @@ public class SummaryDocument extends Document { } private transient String base64Fingerprint = null; + public String getBase64Fingerprint() { if (this.base64Fingerprint == null && this.f != null) { try { @@ -69,6 +76,7 @@ public class SummaryDocument extends Document { } private transient String[] fingerprintSortedHexBlocks = null; + public String[] getFingerprintSortedHexBlocks() { if (this.fingerprintSortedHexBlocks == null && this.f != null) { String fingerprint = this.f.toUpperCase(); @@ -85,6 +93,7 @@ public class SummaryDocument extends Document { } private String n; + public void setNickname(String nickname) { if (nickname == null || nickname.equals("Unnamed")) { this.n = null; @@ -92,14 +101,17 @@ public class SummaryDocument extends Document { this.n = nickname; } } + public String getNickname() { return this.n == null ? "Unnamed" : this.n; } private String[] ad; + public void setAddresses(List<String> addresses) { this.ad = this.collectionToStringArray(addresses); } + public List<String> getAddresses() { return this.stringArrayToList(this.ad); } @@ -116,6 +128,7 @@ public class SummaryDocument extends Document { } return stringArray; } + private List<String> stringArrayToList(String[] stringArray) { List<String> list; if (stringArray == null) { @@ -125,6 +138,7 @@ public class SummaryDocument extends Document { } return list; } + private SortedSet<String> stringArrayToSortedSet(String[] stringArray) { SortedSet<String> sortedSet = new TreeSet<String>(); if (stringArray != null) { @@ -134,62 +148,77 @@ public class SummaryDocument extends Document { } private String cc; + public void setCountryCode(String countryCode) { this.cc = countryCode; } + public String getCountryCode() { return this.cc; } private String as; + public void setASNumber(String aSNumber) { this.as = aSNumber; } + public String getASNumber() { return this.as; } private String fs; + public void setFirstSeenMillis(long firstSeenMillis) { this.fs = DateTimeHelper.format(firstSeenMillis); } + public long getFirstSeenMillis() { return DateTimeHelper.parse(this.fs); } private String ls; + public void setLastSeenMillis(long lastSeenMillis) { this.ls = DateTimeHelper.format(lastSeenMillis); } + public long getLastSeenMillis() { return DateTimeHelper.parse(this.ls); } private String[] rf; + public void setRelayFlags(SortedSet<String> relayFlags) { this.rf = this.collectionToStringArray(relayFlags); } + public SortedSet<String> getRelayFlags() { return this.stringArrayToSortedSet(this.rf); } private long cw; + public void setConsensusWeight(long consensusWeight) { this.cw = consensusWeight; } + public long getConsensusWeight() { return this.cw; } private boolean r; + public void setRunning(boolean isRunning) { this.r = isRunning; } + public boolean isRunning() { return this.r; } private String c; + public void setContact(String contact) { if (contact != null && contact.length() == 0) { this.c = null; @@ -197,6 +226,7 @@ public class SummaryDocument extends Document { this.c = contact; } } + public String getContact() { return this.c; } @@ -205,18 +235,22 @@ public class SummaryDocument extends Document { * updater write effective families to summary documents at least once. * Remove this code after September 8, 2015. */ private String[] ff; + public void setFamilyFingerprints( SortedSet<String> familyFingerprints) { this.ff = this.collectionToStringArray(familyFingerprints); } + public SortedSet<String> getFamilyFingerprints() { return this.stringArrayToSortedSet(this.ff); } private String[] ef; + public void setEffectiveFamily(SortedSet<String> effectiveFamily) { this.ef = this.collectionToStringArray(effectiveFamily); } + public SortedSet<String> getEffectiveFamily() { return this.stringArrayToSortedSet(this.ef); } diff --git a/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java b/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java index efdcf24..6eee47d 100644 --- a/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java @@ -1,5 +1,6 @@ /* Copyright 2013, 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import org.slf4j.Logger; @@ -10,9 +11,11 @@ public class UpdateStatus extends Document { private static Logger log = LoggerFactory.getLogger(UpdateStatus.class); private long updatedMillis; + public void setUpdatedMillis(long updatedMillis) { this.updatedMillis = updatedMillis; } + public long getUpdatedMillis() { return this.updatedMillis; } diff --git a/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java b/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java index 9b84a31..c34fa52 100644 --- a/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import java.util.Map; @@ -9,23 +10,28 @@ public class UptimeDocument extends Document { @SuppressWarnings("unused") private String fingerprint; + public void setFingerprint(String fingerprint) { this.fingerprint = fingerprint; } private Map<String, GraphHistory> uptime; + public void setUptime(Map<String, GraphHistory> uptime) { this.uptime = uptime; } + public Map<String, GraphHistory> getUptime() { return this.uptime; } private SortedMap<String, Map<String, GraphHistory>> flags; + public void setFlags( SortedMap<String, Map<String, GraphHistory>> flags) { this.flags = flags; } + public SortedMap<String, Map<String, GraphHistory>> getFlags() { return this.flags; } diff --git a/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java b/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java index e98f72a..fc8c86a 100644 --- a/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java @@ -1,33 +1,36 @@ package org.torproject.onionoo.docs; -import java.util.SortedSet; -import java.util.TreeSet; - import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.util.SortedSet; +import java.util.TreeSet; public class UptimeHistory implements Comparable<UptimeHistory> { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( UptimeHistory.class); private boolean relay; + public boolean isRelay() { return this.relay; } private long startMillis; + public long getStartMillis() { return this.startMillis; } private int uptimeHours; + public int getUptimeHours() { return this.uptimeHours; } private SortedSet<String> flags; + public SortedSet<String> getFlags() { return this.flags; } @@ -108,17 +111,18 @@ public class UptimeHistory implements Comparable<UptimeHistory> { } else if (!this.relay && other.relay) { return 1; } - return this.startMillis < other.startMillis ? -1 : - this.startMillis > other.startMillis ? 1 : 0; + return this.startMillis < other.startMillis ? -1 + : this.startMillis > other.startMillis ? 1 : 0; } public boolean equals(Object other) { - return other instanceof UptimeHistory && - this.relay == ((UptimeHistory) other).relay && - this.startMillis == ((UptimeHistory) other).startMillis; + return other instanceof UptimeHistory + && this.relay == ((UptimeHistory) other).relay + && this.startMillis == ((UptimeHistory) other).startMillis; } public int hashCode() { return (int) this.startMillis + (this.relay ? 1 : 0); } -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java b/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java index 8c5542d..e0a25b3 100644 --- a/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java @@ -1,36 +1,41 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.NavigableSet; import java.util.Scanner; import java.util.SortedSet; import java.util.TreeSet; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - public class UptimeStatus extends Document { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( UptimeStatus.class); private transient boolean isDirty = false; + public boolean isDirty() { return this.isDirty; } + public void clearDirty() { this.isDirty = false; } private SortedSet<UptimeHistory> relayHistory = new TreeSet<UptimeHistory>(); + public SortedSet<UptimeHistory> getRelayHistory() { return this.relayHistory; } private SortedSet<UptimeHistory> bridgeHistory = new TreeSet<UptimeHistory>(); + public SortedSet<UptimeHistory> getBridgeHistory() { return this.bridgeHistory; } @@ -64,14 +69,14 @@ public class UptimeStatus extends Document { new TreeSet<UptimeHistory>(history.headSet(new UptimeHistory( relay, startMillis + DateTimeHelper.ONE_HOUR, 0, flags))); for (UptimeHistory prev : existingIntervals.descendingSet()) { - if (prev.isRelay() != interval.isRelay() || - prev.getStartMillis() + DateTimeHelper.ONE_HOUR + if (prev.isRelay() != interval.isRelay() + || prev.getStartMillis() + DateTimeHelper.ONE_HOUR * prev.getUptimeHours() <= interval.getStartMillis()) { break; } - if (prev.getFlags() == interval.getFlags() || - (prev.getFlags() != null && interval.getFlags() != null && - prev.getFlags().equals(interval.getFlags()))) { + if (prev.getFlags() == interval.getFlags() + || (prev.getFlags() != null && interval.getFlags() != null + && prev.getFlags().equals(interval.getFlags()))) { /* The exact same interval is already contained in history. */ return; } @@ -87,8 +92,8 @@ public class UptimeStatus extends Document { prev.getStartMillis(), hoursBefore, prev.getFlags())); } int hoursAfter = (int) (prev.getStartMillis() - / DateTimeHelper.ONE_HOUR + prev.getUptimeHours() - - interval.getStartMillis() / DateTimeHelper.ONE_HOUR - 1); + / DateTimeHelper.ONE_HOUR + prev.getUptimeHours() + - interval.getStartMillis() / DateTimeHelper.ONE_HOUR - 1); if (hoursAfter > 0) { history.add(new UptimeHistory(relay, interval.getStartMillis() + DateTimeHelper.ONE_HOUR, @@ -111,12 +116,13 @@ public class UptimeStatus extends Document { history.clear(); UptimeHistory lastInterval = null; for (UptimeHistory interval : uncompressedHistory) { - if (lastInterval != null && - lastInterval.getStartMillis() + DateTimeHelper.ONE_HOUR - * lastInterval.getUptimeHours() == interval.getStartMillis() && - lastInterval.isRelay() == interval.isRelay() && - (lastInterval.getFlags() == interval.getFlags() || - (lastInterval.getFlags() != null && interval.getFlags() != null + if (lastInterval != null + && lastInterval.getStartMillis() + DateTimeHelper.ONE_HOUR + * lastInterval.getUptimeHours() == interval.getStartMillis() + && lastInterval.isRelay() == interval.isRelay() + && (lastInterval.getFlags() == interval.getFlags() + || (lastInterval.getFlags() != null + && interval.getFlags() != null && lastInterval.getFlags().equals(interval.getFlags())))) { lastInterval.addUptime(interval); } else { diff --git a/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java b/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java index 64ce713..62dcf20 100644 --- a/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import java.util.Map; @@ -8,12 +9,14 @@ public class WeightsDocument extends Document { @SuppressWarnings("unused") private String fingerprint; + public void setFingerprint(String fingerprint) { this.fingerprint = fingerprint; } @SuppressWarnings("unused") private Map<String, GraphHistory> consensus_weight_fraction; + public void setConsensusWeightFraction( Map<String, GraphHistory> consensusWeightFraction) { this.consensus_weight_fraction = consensusWeightFraction; @@ -21,6 +24,7 @@ public class WeightsDocument extends Document { @SuppressWarnings("unused") private Map<String, GraphHistory> guard_probability; + public void setGuardProbability( Map<String, GraphHistory> guardProbability) { this.guard_probability = guardProbability; @@ -28,6 +32,7 @@ public class WeightsDocument extends Document { @SuppressWarnings("unused") private Map<String, GraphHistory> middle_probability; + public void setMiddleProbability( Map<String, GraphHistory> middleProbability) { this.middle_probability = middleProbability; @@ -35,6 +40,7 @@ public class WeightsDocument extends Document { @SuppressWarnings("unused") private Map<String, GraphHistory> exit_probability; + public void setExitProbability( Map<String, GraphHistory> exitProbability) { this.exit_probability = exitProbability; @@ -42,6 +48,7 @@ public class WeightsDocument extends Document { @SuppressWarnings("unused") private Map<String, GraphHistory> consensus_weight; + public void setConsensusWeight( Map<String, GraphHistory> consensusWeight) { this.consensus_weight = consensusWeight; diff --git a/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java b/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java index 5d7f23e..77ce583 100644 --- a/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java @@ -1,37 +1,44 @@ package org.torproject.onionoo.docs; +import org.torproject.onionoo.util.TimeFactory; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.Comparator; import java.util.Map; import java.util.Scanner; import java.util.SortedMap; import java.util.TreeMap; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.onionoo.util.TimeFactory; - public class WeightsStatus extends Document { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( WeightsStatus.class); private transient boolean isDirty = false; + public boolean isDirty() { return this.isDirty; } + public void clearDirty() { this.isDirty = false; } private SortedMap<long[], double[]> history = - new TreeMap<long[], double[]>(new Comparator<long[]>() { - public int compare(long[] a, long[] b) { - return a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0; - } - }); + new TreeMap<long[], double[]>( + new Comparator<long[]>() { + public int compare(long[] a, long[] b) { + return a[0] < b[0] ? -1 : a[0] > b[0] ? 1 : 0; + } + } + ); + public void setHistory(SortedMap<long[], double[]> history) { this.history = history; } + public SortedMap<long[], double[]> getHistory() { return this.history; } @@ -83,12 +90,12 @@ public class WeightsStatus extends Document { public void addToHistory(long validAfterMillis, long freshUntilMillis, double[] weights) { long[] interval = new long[] { validAfterMillis, freshUntilMillis }; - if ((this.history.headMap(interval).isEmpty() || - this.history.headMap(interval).lastKey()[1] <= - validAfterMillis) && - (this.history.tailMap(interval).isEmpty() || - this.history.tailMap(interval).firstKey()[0] >= - freshUntilMillis)) { + if ((this.history.headMap(interval).isEmpty() + || this.history.headMap(interval).lastKey()[1] + <= validAfterMillis) + && (this.history.tailMap(interval).isEmpty() + || this.history.tailMap(interval).firstKey()[0] + >= freshUntilMillis)) { this.history.put(interval, weights); this.isDirty = true; } @@ -98,25 +105,27 @@ public class WeightsStatus extends Document { SortedMap<long[], double[]> uncompressedHistory = new TreeMap<long[], double[]>(this.history); history.clear(); - long lastStartMillis = 0L, lastEndMillis = 0L; + long lastStartMillis = 0L; + long lastEndMillis = 0L; double[] lastWeights = null; String lastMonthString = "1970-01"; int lastMissingValues = -1; long now = TimeFactory.getTime().currentTimeMillis(); for (Map.Entry<long[], double[]> e : uncompressedHistory.entrySet()) { - long startMillis = e.getKey()[0], endMillis = e.getKey()[1]; + long startMillis = e.getKey()[0]; + long endMillis = e.getKey()[1]; double[] weights = e.getValue(); long intervalLengthMillis; if (now - endMillis <= DateTimeHelper.ONE_WEEK) { intervalLengthMillis = DateTimeHelper.ONE_HOUR; - } else if (now - endMillis <= - DateTimeHelper.ROUGHLY_ONE_MONTH) { + } else if (now - endMillis + <= DateTimeHelper.ROUGHLY_ONE_MONTH) { intervalLengthMillis = DateTimeHelper.FOUR_HOURS; - } else if (now - endMillis <= - DateTimeHelper.ROUGHLY_THREE_MONTHS) { + } else if (now - endMillis + <= DateTimeHelper.ROUGHLY_THREE_MONTHS) { intervalLengthMillis = DateTimeHelper.TWELVE_HOURS; - } else if (now - endMillis <= - DateTimeHelper.ROUGHLY_ONE_YEAR) { + } else if (now - endMillis + <= DateTimeHelper.ROUGHLY_ONE_YEAR) { intervalLengthMillis = DateTimeHelper.TWO_DAYS; } else { intervalLengthMillis = DateTimeHelper.TEN_DAYS; @@ -129,11 +138,11 @@ public class WeightsStatus extends Document { missingValues += 1 << i; } } - if (lastEndMillis == startMillis && - ((lastEndMillis - 1L) / intervalLengthMillis) == - ((endMillis - 1L) / intervalLengthMillis) && - lastMonthString.equals(monthString) && - lastMissingValues == missingValues) { + if (lastEndMillis == startMillis + && ((lastEndMillis - 1L) / intervalLengthMillis) + == ((endMillis - 1L) / intervalLengthMillis) + && lastMonthString.equals(monthString) + && lastMissingValues == missingValues) { double lastIntervalInHours = (double) ((lastEndMillis - lastStartMillis) / DateTimeHelper.ONE_HOUR); double currentIntervalInHours = (double) ((endMillis diff --git a/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java b/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java index 6cf022f..901eece 100644 --- a/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java +++ b/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java @@ -1,5 +1,6 @@ /* Copyright 2011, 2012 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; import java.util.Map; @@ -7,21 +8,28 @@ import java.util.Map; import javax.servlet.http.HttpServletRequest; public class HttpServletRequestWrapper { + private HttpServletRequest request; + protected HttpServletRequestWrapper(HttpServletRequest request) { this.request = request; } + protected String getRequestURI() { return this.request.getRequestURI(); } + @SuppressWarnings("rawtypes") protected Map getParameterMap() { return this.request.getParameterMap(); } + protected String[] getParameterValues(String parameterKey) { return this.request.getParameterValues(parameterKey); } + protected String getQueryString() { return this.request.getQueryString(); } -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java b/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java index 58d9f03..a3f3fff 100644 --- a/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java +++ b/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java @@ -1,5 +1,6 @@ /* Copyright 2011, 2012 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; import java.io.IOException; @@ -8,23 +9,31 @@ import java.io.PrintWriter; import javax.servlet.http.HttpServletResponse; public class HttpServletResponseWrapper { + private HttpServletResponse response = null; + protected HttpServletResponseWrapper(HttpServletResponse response) { this.response = response; } + protected void sendError(int errorStatusCode) throws IOException { this.response.sendError(errorStatusCode); } + protected void setHeader(String headerName, String headerValue) { this.response.setHeader(headerName, headerValue); } + protected void setContentType(String contentType) { this.response.setContentType(contentType); } + protected void setCharacterEncoding(String characterEncoding) { this.response.setCharacterEncoding(characterEncoding); } + protected PrintWriter getWriter() throws IOException { return this.response.getWriter(); } -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/server/NodeIndex.java b/src/main/java/org/torproject/onionoo/server/NodeIndex.java index fbc8896..82e5e08 100644 --- a/src/main/java/org/torproject/onionoo/server/NodeIndex.java +++ b/src/main/java/org/torproject/onionoo/server/NodeIndex.java @@ -1,5 +1,7 @@ package org.torproject.onionoo.server; +import org.torproject.onionoo.docs.SummaryDocument; + import java.text.SimpleDateFormat; import java.util.List; import java.util.Map; @@ -7,11 +9,10 @@ import java.util.Set; import java.util.SortedMap; import java.util.TimeZone; -import org.torproject.onionoo.docs.SummaryDocument; - class NodeIndex { private String relaysPublishedString; + public void setRelaysPublishedMillis(long relaysPublishedMillis) { SimpleDateFormat dateTimeFormat = new SimpleDateFormat( "yyyy-MM-dd HH:mm:ss"); @@ -19,11 +20,13 @@ class NodeIndex { this.relaysPublishedString = dateTimeFormat.format(relaysPublishedMillis); } + public String getRelaysPublishedString() { return relaysPublishedString; } private String bridgesPublishedString; + public void setBridgesPublishedMillis(long bridgesPublishedMillis) { SimpleDateFormat dateTimeFormat = new SimpleDateFormat( "yyyy-MM-dd HH:mm:ss"); @@ -31,122 +34,149 @@ class NodeIndex { this.bridgesPublishedString = dateTimeFormat.format(bridgesPublishedMillis); } + public String getBridgesPublishedString() { return bridgesPublishedString; } private List<String> relaysByConsensusWeight; + public void setRelaysByConsensusWeight( List<String> relaysByConsensusWeight) { this.relaysByConsensusWeight = relaysByConsensusWeight; } + public List<String> getRelaysByConsensusWeight() { return relaysByConsensusWeight; } - private Map<String, SummaryDocument> relayFingerprintSummaryLines; + public void setRelayFingerprintSummaryLines( Map<String, SummaryDocument> relayFingerprintSummaryLines) { this.relayFingerprintSummaryLines = relayFingerprintSummaryLines; } + public Map<String, SummaryDocument> getRelayFingerprintSummaryLines() { return this.relayFingerprintSummaryLines; } private Map<String, SummaryDocument> bridgeFingerprintSummaryLines; + public void setBridgeFingerprintSummaryLines( Map<String, SummaryDocument> bridgeFingerprintSummaryLines) { this.bridgeFingerprintSummaryLines = bridgeFingerprintSummaryLines; } + public Map<String, SummaryDocument> getBridgeFingerprintSummaryLines() { return this.bridgeFingerprintSummaryLines; } private Map<String, Set<String>> relaysByCountryCode = null; + public void setRelaysByCountryCode( Map<String, Set<String>> relaysByCountryCode) { this.relaysByCountryCode = relaysByCountryCode; } + public Map<String, Set<String>> getRelaysByCountryCode() { return relaysByCountryCode; } private Map<String, Set<String>> relaysByASNumber = null; + public void setRelaysByASNumber( Map<String, Set<String>> relaysByASNumber) { this.relaysByASNumber = relaysByASNumber; } + public Map<String, Set<String>> getRelaysByASNumber() { return relaysByASNumber; } private Map<String, Set<String>> relaysByFlag = null; + public void setRelaysByFlag(Map<String, Set<String>> relaysByFlag) { this.relaysByFlag = relaysByFlag; } + public Map<String, Set<String>> getRelaysByFlag() { return relaysByFlag; } private Map<String, Set<String>> bridgesByFlag = null; + public void setBridgesByFlag(Map<String, Set<String>> bridgesByFlag) { this.bridgesByFlag = bridgesByFlag; } + public Map<String, Set<String>> getBridgesByFlag() { return bridgesByFlag; } private Map<String, Set<String>> relaysByContact = null; + public void setRelaysByContact( Map<String, Set<String>> relaysByContact) { this.relaysByContact = relaysByContact; } + public Map<String, Set<String>> getRelaysByContact() { return relaysByContact; } private Map<String, Set<String>> relaysByFamily = null; + public void setRelaysByFamily(Map<String, Set<String>> relaysByFamily) { this.relaysByFamily = relaysByFamily; } + public Map<String, Set<String>> getRelaysByFamily() { return this.relaysByFamily; } private SortedMap<Integer, Set<String>> relaysByFirstSeenDays; + public void setRelaysByFirstSeenDays( SortedMap<Integer, Set<String>> relaysByFirstSeenDays) { this.relaysByFirstSeenDays = relaysByFirstSeenDays; } + public SortedMap<Integer, Set<String>> getRelaysByFirstSeenDays() { return relaysByFirstSeenDays; } private SortedMap<Integer, Set<String>> bridgesByFirstSeenDays; + public void setBridgesByFirstSeenDays( SortedMap<Integer, Set<String>> bridgesByFirstSeenDays) { this.bridgesByFirstSeenDays = bridgesByFirstSeenDays; } + public SortedMap<Integer, Set<String>> getBridgesByFirstSeenDays() { return bridgesByFirstSeenDays; } private SortedMap<Integer, Set<String>> relaysByLastSeenDays; + public void setRelaysByLastSeenDays( SortedMap<Integer, Set<String>> relaysByLastSeenDays) { this.relaysByLastSeenDays = relaysByLastSeenDays; } + public SortedMap<Integer, Set<String>> getRelaysByLastSeenDays() { return relaysByLastSeenDays; } private SortedMap<Integer, Set<String>> bridgesByLastSeenDays; + public void setBridgesByLastSeenDays( SortedMap<Integer, Set<String>> bridgesByLastSeenDays) { this.bridgesByLastSeenDays = bridgesByLastSeenDays; } + public SortedMap<Integer, Set<String>> getBridgesByLastSeenDays() { return bridgesByLastSeenDays; } -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/server/NodeIndexer.java b/src/main/java/org/torproject/onionoo/server/NodeIndexer.java index 347996b..4e123d9 100644 --- a/src/main/java/org/torproject/onionoo/server/NodeIndexer.java +++ b/src/main/java/org/torproject/onionoo/server/NodeIndexer.java @@ -1,5 +1,15 @@ package org.torproject.onionoo.server; +import org.torproject.onionoo.docs.DocumentStore; +import org.torproject.onionoo.docs.DocumentStoreFactory; +import org.torproject.onionoo.docs.SummaryDocument; +import org.torproject.onionoo.docs.UpdateStatus; +import org.torproject.onionoo.util.Time; +import org.torproject.onionoo.util.TimeFactory; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.File; import java.util.ArrayList; import java.util.Collections; @@ -16,15 +26,6 @@ import javax.servlet.ServletContext; import javax.servlet.ServletContextEvent; import javax.servlet.ServletContextListener; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.onionoo.docs.DocumentStore; -import org.torproject.onionoo.docs.DocumentStoreFactory; -import org.torproject.onionoo.docs.SummaryDocument; -import org.torproject.onionoo.docs.UpdateStatus; -import org.torproject.onionoo.util.Time; -import org.torproject.onionoo.util.TimeFactory; - public class NodeIndexer implements ServletContextListener, Runnable { private static final Logger log = LoggerFactory.getLogger( @@ -57,8 +58,8 @@ public class NodeIndexer implements ServletContextListener, Runnable { private Thread nodeIndexerThread = null; public synchronized long getLastIndexed(long timeoutMillis) { - if (this.lastIndexed == -1L && this.nodeIndexerThread != null && - timeoutMillis > 0L) { + if (this.lastIndexed == -1L && this.nodeIndexerThread != null + && timeoutMillis > 0L) { try { this.wait(timeoutMillis); } catch (InterruptedException e) { @@ -68,8 +69,8 @@ public class NodeIndexer implements ServletContextListener, Runnable { } public synchronized NodeIndex getLatestNodeIndex(long timeoutMillis) { - if (this.latestNodeIndex == null && this.nodeIndexerThread != null && - timeoutMillis > 0L) { + if (this.latestNodeIndex == null && this.nodeIndexerThread != null + && timeoutMillis > 0L) { try { this.wait(timeoutMillis); } catch (InterruptedException e) { @@ -86,8 +87,9 @@ public class NodeIndexer implements ServletContextListener, Runnable { } } - private static final long ONE_MINUTE = 60L * 1000L, - ONE_DAY = 24L * 60L * ONE_MINUTE; + private static final long ONE_MINUTE = 60L * 1000L; + + private static final long ONE_DAY = 24L * 60L * ONE_MINUTE; public void run() { while (this.nodeIndexerThread != null) { @@ -121,28 +123,36 @@ public class NodeIndexer implements ServletContextListener, Runnable { } documentStore.invalidateDocumentCache(); List<String> newRelaysByConsensusWeight = new ArrayList<String>(); - Map<String, SummaryDocument> - newRelayFingerprintSummaryLines = - new HashMap<String, SummaryDocument>(), - newBridgeFingerprintSummaryLines = + Map<String, SummaryDocument> newRelayFingerprintSummaryLines = + new HashMap<String, SummaryDocument>(); + Map<String, SummaryDocument> newBridgeFingerprintSummaryLines = new HashMap<String, SummaryDocument>(); - Map<String, Set<String>> - newRelaysByCountryCode = new HashMap<String, Set<String>>(), - newRelaysByASNumber = new HashMap<String, Set<String>>(), - newRelaysByFlag = new HashMap<String, Set<String>>(), - newBridgesByFlag = new HashMap<String, Set<String>>(), - newRelaysByContact = new HashMap<String, Set<String>>(), - newRelaysByFamily = new HashMap<String, Set<String>>(); - SortedMap<Integer, Set<String>> - newRelaysByFirstSeenDays = new TreeMap<Integer, Set<String>>(), - newBridgesByFirstSeenDays = new TreeMap<Integer, Set<String>>(), - newRelaysByLastSeenDays = new TreeMap<Integer, Set<String>>(), - newBridgesByLastSeenDays = new TreeMap<Integer, Set<String>>(); - Set<SummaryDocument> currentRelays = new HashSet<SummaryDocument>(), - currentBridges = new HashSet<SummaryDocument>(); + Map<String, Set<String>> newRelaysByCountryCode = + new HashMap<String, Set<String>>(); + Map<String, Set<String>> newRelaysByASNumber = + new HashMap<String, Set<String>>(); + Map<String, Set<String>> newRelaysByFlag = + new HashMap<String, Set<String>>(); + Map<String, Set<String>> newBridgesByFlag = + new HashMap<String, Set<String>>(); + Map<String, Set<String>> newRelaysByContact = + new HashMap<String, Set<String>>(); + Map<String, Set<String>> newRelaysByFamily = + new HashMap<String, Set<String>>(); + SortedMap<Integer, Set<String>> newRelaysByFirstSeenDays = + new TreeMap<Integer, Set<String>>(); + SortedMap<Integer, Set<String>> newBridgesByFirstSeenDays = + new TreeMap<Integer, Set<String>>(); + SortedMap<Integer, Set<String>> newRelaysByLastSeenDays = + new TreeMap<Integer, Set<String>>(); + SortedMap<Integer, Set<String>> newBridgesByLastSeenDays = + new TreeMap<Integer, Set<String>>(); + Set<SummaryDocument> currentRelays = new HashSet<SummaryDocument>(); + Set<SummaryDocument> currentBridges = new HashSet<SummaryDocument>(); SortedSet<String> fingerprints = documentStore.list( SummaryDocument.class); - long relaysLastValidAfterMillis = 0L, bridgesLastPublishedMillis = 0L; + long relaysLastValidAfterMillis = 0L; + long bridgesLastPublishedMillis = 0L; for (String fingerprint : fingerprints) { SummaryDocument node = documentStore.retrieve(SummaryDocument.class, true, fingerprint); @@ -165,8 +175,8 @@ public class NodeIndexer implements ServletContextListener, Runnable { new TreeMap<String, Set<String>>(); for (SummaryDocument entry : currentRelays) { String fingerprint = entry.getFingerprint().toUpperCase(); - String hashedFingerprint = entry.getHashedFingerprint(). - toUpperCase(); + String hashedFingerprint = entry.getHashedFingerprint() + .toUpperCase(); newRelayFingerprintSummaryLines.put(fingerprint, entry); newRelayFingerprintSummaryLines.put(hashedFingerprint, entry); long consensusWeight = entry.getConsensusWeight(); @@ -202,8 +212,8 @@ public class NodeIndexer implements ServletContextListener, Runnable { /* This condition can go away once all Onionoo services had their * hourly updater write effective families to summary documents at * least once. Remove this code after September 8, 2015. */ - if (entry.getFamilyFingerprints() != null && - !entry.getFamilyFingerprints().isEmpty()) { + if (entry.getFamilyFingerprints() != null + && !entry.getFamilyFingerprints().isEmpty()) { computedEffectiveFamilies.put(fingerprint, entry.getFamilyFingerprints()); } @@ -248,9 +258,9 @@ public class NodeIndexer implements ServletContextListener, Runnable { String fingerprint = e.getKey(); Set<String> inMutualFamilyRelation = new HashSet<String>(); for (String otherFingerprint : e.getValue()) { - if (computedEffectiveFamilies.containsKey(otherFingerprint) && - computedEffectiveFamilies.get(otherFingerprint).contains( - fingerprint)) { + if (computedEffectiveFamilies.containsKey(otherFingerprint) + && computedEffectiveFamilies.get(otherFingerprint).contains( + fingerprint)) { inMutualFamilyRelation.add(otherFingerprint); } } @@ -258,8 +268,8 @@ public class NodeIndexer implements ServletContextListener, Runnable { } for (SummaryDocument entry : currentBridges) { String hashedFingerprint = entry.getFingerprint().toUpperCase(); - String hashedHashedFingerprint = entry.getHashedFingerprint(). - toUpperCase(); + String hashedHashedFingerprint = entry.getHashedFingerprint() + .toUpperCase(); newBridgeFingerprintSummaryLines.put(hashedFingerprint, entry); newBridgeFingerprintSummaryLines.put(hashedHashedFingerprint, entry); diff --git a/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java b/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java index a605adf..cc045f7 100644 --- a/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java +++ b/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java @@ -1,13 +1,16 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; public class NodeIndexerFactory { private static NodeIndexer nodeIndexerInstance; + public static void setNodeIndexer(NodeIndexer nodeIndexer) { nodeIndexerInstance = nodeIndexer; } + public static NodeIndexer getNodeIndexer() { if (nodeIndexerInstance == null) { nodeIndexerInstance = new NodeIndexer(); diff --git a/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java b/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java index e764601..7bf85b6 100644 --- a/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java +++ b/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java @@ -1,7 +1,15 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; +import org.torproject.onionoo.docs.DateTimeHelper; +import org.torproject.onionoo.util.Time; +import org.torproject.onionoo.util.TimeFactory; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.text.SimpleDateFormat; import java.util.Arrays; import java.util.Collection; @@ -14,27 +22,27 @@ import java.util.TimeZone; import java.util.TreeMap; import java.util.TreeSet; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.onionoo.docs.DateTimeHelper; -import org.torproject.onionoo.util.Time; -import org.torproject.onionoo.util.TimeFactory; - class Counter { + int value = 0; + void increment() { this.value++; } + public String toString() { return String.valueOf(this.value); } + void clear() { this.value = 0; } } class MostFrequentString { + Map<String, Integer> stringFrequencies = new HashMap<String, Integer>(); + void addString(String string) { if (!this.stringFrequencies.containsKey(string)) { this.stringFrequencies.put(string, 1); @@ -43,6 +51,7 @@ class MostFrequentString { this.stringFrequencies.get(string) + 1); } } + public String toString() { SortedMap<Integer, SortedSet<String>> sortedFrequencies = new TreeMap<Integer, SortedSet<String>>( @@ -59,7 +68,8 @@ class MostFrequentString { } } StringBuilder sb = new StringBuilder(); - int stringsToAdd = 3, written = 0; + int stringsToAdd = 3; + int written = 0; for (Map.Entry<Integer, SortedSet<String>> e : sortedFrequencies.entrySet()) { for (String string : e.getValue()) { @@ -74,16 +84,20 @@ class MostFrequentString { } return sb.toString(); } + void clear() { this.stringFrequencies.clear(); } } class IntegerDistribution { + int[] logValues = new int[64]; + void addLong(long value) { logValues[64 - Long.numberOfLeadingZeros(value)]++; } + public String toString() { StringBuilder sb = new StringBuilder(); int totalValues = 0; @@ -95,8 +109,8 @@ class IntegerDistribution { int seenValues = 0; for (int i = 0, j = 0; i < logValues.length; i++) { seenValues += logValues[i]; - while (j < permilles.length && - (seenValues * 1000 > totalValues * permilles[j])) { + while (j < permilles.length + && (seenValues * 1000 > totalValues * permilles[j])) { sb.append((j > 0 ? ", " : "") + "." + permilles[j] + (i < logValues.length - 1 ? "<" + (1L << i) : ">=" + (1L << i - 1))); @@ -113,6 +127,7 @@ class IntegerDistribution { } return sb.toString(); } + void clear() { Arrays.fill(logValues, 0, logValues.length - 1, 0); } @@ -129,21 +144,33 @@ public class PerformanceMetrics { private static long lastLoggedMillis = -1L; - private static final long LOG_INTERVAL_SECONDS = 60L * 60L, - LOG_INTERVAL_MILLIS = LOG_INTERVAL_SECONDS * 1000L; + private static final long LOG_INTERVAL_SECONDS = 60L * 60L; + + private static final long LOG_INTERVAL_MILLIS = + LOG_INTERVAL_SECONDS * 1000L; private static Counter totalProcessedRequests = new Counter(); - private static MostFrequentString - requestsByResourceType = new MostFrequentString(), - requestsByParameters = new MostFrequentString(); + private static MostFrequentString requestsByResourceType = + new MostFrequentString(); + + private static MostFrequentString requestsByParameters = + new MostFrequentString(); + + private static IntegerDistribution matchingRelayDocuments = + new IntegerDistribution(); - private static IntegerDistribution - matchingRelayDocuments = new IntegerDistribution(), - matchingBridgeDocuments = new IntegerDistribution(), - writtenChars = new IntegerDistribution(), - handleRequestMillis = new IntegerDistribution(), - buildResponseMillis = new IntegerDistribution(); + private static IntegerDistribution matchingBridgeDocuments = + new IntegerDistribution(); + + private static IntegerDistribution writtenChars = + new IntegerDistribution(); + + private static IntegerDistribution handleRequestMillis = + new IntegerDistribution(); + + private static IntegerDistribution buildResponseMillis = + new IntegerDistribution(); public static void logStatistics(long receivedRequestMillis, String resourceType, Collection<String> parameterKeys, @@ -156,8 +183,8 @@ public class PerformanceMetrics { } if (lastLoggedMillis < 0L) { lastLoggedMillis = time.currentTimeMillis(); - } else if (receivedRequestMillis - lastLoggedMillis > - LOG_INTERVAL_MILLIS) { + } else if (receivedRequestMillis - lastLoggedMillis + > LOG_INTERVAL_MILLIS) { SimpleDateFormat dateTimeFormat = new SimpleDateFormat( "yyyy-MM-dd HH:mm:ss"); dateTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); @@ -191,8 +218,8 @@ public class PerformanceMetrics { buildResponseMillis.clear(); do { lastLoggedMillis += LOG_INTERVAL_MILLIS; - } while (receivedRequestMillis - lastLoggedMillis > - LOG_INTERVAL_MILLIS); + } while (receivedRequestMillis - lastLoggedMillis + > LOG_INTERVAL_MILLIS); } totalProcessedRequests.increment(); long handlingTime = parsedRequestMillis - receivedRequestMillis; @@ -216,6 +243,5 @@ public class PerformanceMetrics { buildResponseMillis.addLong(responseTime); } } - } diff --git a/src/main/java/org/torproject/onionoo/server/RequestHandler.java b/src/main/java/org/torproject/onionoo/server/RequestHandler.java index bd75992..cb71681 100644 --- a/src/main/java/org/torproject/onionoo/server/RequestHandler.java +++ b/src/main/java/org/torproject/onionoo/server/RequestHandler.java @@ -1,7 +1,12 @@ /* Copyright 2011--2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; +import org.torproject.onionoo.docs.DocumentStore; +import org.torproject.onionoo.docs.DocumentStoreFactory; +import org.torproject.onionoo.docs.SummaryDocument; + import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -12,10 +17,6 @@ import java.util.Map; import java.util.Set; import java.util.SortedMap; -import org.torproject.onionoo.docs.DocumentStore; -import org.torproject.onionoo.docs.DocumentStoreFactory; -import org.torproject.onionoo.docs.SummaryDocument; - public class RequestHandler { private NodeIndex nodeIndex; @@ -28,74 +29,88 @@ public class RequestHandler { } private String resourceType; + public void setResourceType(String resourceType) { this.resourceType = resourceType; } private String type; + public void setType(String type) { this.type = type; } private String running; + public void setRunning(String running) { this.running = running; } private String[] search; + public void setSearch(String[] search) { this.search = new String[search.length]; System.arraycopy(search, 0, this.search, 0, search.length); } private String lookup; + public void setLookup(String lookup) { this.lookup = lookup; } private String fingerprint; + public void setFingerprint(String fingerprint) { this.fingerprint = fingerprint; } private String country; + public void setCountry(String country) { this.country = country; } private String as; + public void setAs(String as) { this.as = as; } private String flag; + public void setFlag(String flag) { this.flag = flag; } private String[] contact; + public void setContact(String[] contact) { this.contact = new String[contact.length]; System.arraycopy(contact, 0, this.contact, 0, contact.length); } private String[] order; + public void setOrder(String[] order) { this.order = new String[order.length]; System.arraycopy(order, 0, this.order, 0, order.length); } private String offset; + public void setOffset(String offset) { this.offset = offset; } private String limit; + public void setLimit(String limit) { this.limit = limit; } private int[] firstSeenDays; + public void setFirstSeenDays(int[] firstSeenDays) { this.firstSeenDays = new int[firstSeenDays.length]; System.arraycopy(firstSeenDays, 0, this.firstSeenDays, 0, @@ -103,6 +118,7 @@ public class RequestHandler { } private int[] lastSeenDays; + public void setLastSeenDays(int[] lastSeenDays) { this.lastSeenDays = new int[lastSeenDays.length]; System.arraycopy(lastSeenDays, 0, this.lastSeenDays, 0, @@ -110,6 +126,7 @@ public class RequestHandler { } private String family; + public void setFamily(String family) { this.family = family; } @@ -143,7 +160,6 @@ public class RequestHandler { this.limit(); } - private void filterByResourceType() { if (this.resourceType.equals("clients")) { this.filteredRelays.clear(); @@ -171,8 +187,8 @@ public class RequestHandler { } boolean runningRequested = this.running.equals("true"); Set<String> removeRelays = new HashSet<String>(); - for (Map.Entry<String, SummaryDocument> e : - filteredRelays.entrySet()) { + for (Map.Entry<String, SummaryDocument> e + : filteredRelays.entrySet()) { if (e.getValue().isRunning() != runningRequested) { removeRelays.add(e.getKey()); } @@ -181,8 +197,8 @@ public class RequestHandler { this.filteredRelays.remove(fingerprint); } Set<String> removeBridges = new HashSet<String>(); - for (Map.Entry<String, SummaryDocument> e : - filteredBridges.entrySet()) { + for (Map.Entry<String, SummaryDocument> e + : filteredBridges.entrySet()) { if (e.getValue().isRunning() != runningRequested) { removeBridges.add(e.getKey()); } @@ -204,17 +220,17 @@ public class RequestHandler { private void filterBySearchTerm(String searchTerm) { Set<String> removeRelays = new HashSet<String>(); - for (Map.Entry<String, SummaryDocument> e : - filteredRelays.entrySet()) { + for (Map.Entry<String, SummaryDocument> e + : filteredRelays.entrySet()) { String fingerprint = e.getKey(); SummaryDocument entry = e.getValue(); - String base64Fingerprint = entry.isRelay() ? - entry.getBase64Fingerprint() : null; + String base64Fingerprint = entry.isRelay() + ? entry.getBase64Fingerprint() : null; String[] fingerprintSortedHexBlocks = entry.getFingerprintSortedHexBlocks(); boolean lineMatches = false; - String nickname = entry.getNickname() != null ? - entry.getNickname().toLowerCase() : "unnamed"; + String nickname = entry.getNickname() != null + ? entry.getNickname().toLowerCase() : "unnamed"; if (searchTerm.startsWith("$")) { /* Search is for $-prefixed fingerprint. */ if (fingerprint.startsWith( @@ -228,14 +244,14 @@ public class RequestHandler { } else if (fingerprint.startsWith(searchTerm.toUpperCase())) { /* Non-$-prefixed fingerprint matches. */ lineMatches = true; - } else if (base64Fingerprint != null && - base64Fingerprint.startsWith(searchTerm)) { + } else if (base64Fingerprint != null + && base64Fingerprint.startsWith(searchTerm)) { /* Base64-encoded fingerprint matches. */ lineMatches = true; - } else if (searchTerm.length() == 4 && - fingerprintSortedHexBlocks != null && - Arrays.binarySearch(fingerprintSortedHexBlocks, - searchTerm.toUpperCase()) >= 0) { + } else if (searchTerm.length() == 4 + && fingerprintSortedHexBlocks != null + && Arrays.binarySearch(fingerprintSortedHexBlocks, + searchTerm.toUpperCase()) >= 0) { /* 4-hex-character block of space-separated fingerprint * matches. */ lineMatches = true; @@ -262,8 +278,8 @@ public class RequestHandler { String hashedFingerprint = e.getKey(); SummaryDocument entry = e.getValue(); boolean lineMatches = false; - String nickname = entry.getNickname() != null ? - entry.getNickname().toLowerCase() : "unnamed"; + String nickname = entry.getNickname() != null + ? entry.getNickname().toLowerCase() : "unnamed"; if (searchTerm.startsWith("$")) { /* Search is for $-prefixed hashed fingerprint. */ if (hashedFingerprint.startsWith( @@ -464,8 +480,8 @@ public class RequestHandler { this.nodeIndex.getRelaysByContact().entrySet()) { String contact = e.getKey(); for (String contactPart : this.contact) { - if (contact == null || - !contact.contains(contactPart.toLowerCase())) { + if (contact == null + || !contact.contains(contactPart.toLowerCase())) { removeRelays.addAll(e.getValue()); break; } @@ -486,8 +502,8 @@ public class RequestHandler { this.filteredRelays.keySet()); removeRelays.remove(this.family); if (this.nodeIndex.getRelaysByFamily().containsKey(this.family)) { - removeRelays.removeAll(this.nodeIndex.getRelaysByFamily(). - get(this.family)); + removeRelays.removeAll(this.nodeIndex.getRelaysByFamily() + .get(this.family)); } for (String fingerprint : removeRelays) { this.filteredRelays.remove(fingerprint); @@ -503,8 +519,8 @@ public class RequestHandler { Collections.reverse(orderBy); } for (String relay : orderBy) { - if (this.filteredRelays.containsKey(relay) && - !this.orderedRelays.contains(filteredRelays.get(relay))) { + if (this.filteredRelays.containsKey(relay) + && !this.orderedRelays.contains(filteredRelays.get(relay))) { this.orderedRelays.add(this.filteredRelays.remove(relay)); } } @@ -532,9 +548,9 @@ public class RequestHandler { return; } int offsetValue = Integer.parseInt(this.offset); - while (offsetValue-- > 0 && - (!this.orderedRelays.isEmpty() || - !this.orderedBridges.isEmpty())) { + while (offsetValue-- > 0 + && (!this.orderedRelays.isEmpty() + || !this.orderedBridges.isEmpty())) { if (!this.orderedRelays.isEmpty()) { this.orderedRelays.remove(0); } else { @@ -549,25 +565,27 @@ public class RequestHandler { return; } int limitValue = Integer.parseInt(this.limit); - while (!this.orderedRelays.isEmpty() && - limitValue < this.orderedRelays.size()) { + while (!this.orderedRelays.isEmpty() + && limitValue < this.orderedRelays.size()) { this.orderedRelays.remove(this.orderedRelays.size() - 1); } limitValue -= this.orderedRelays.size(); - while (!this.orderedBridges.isEmpty() && - limitValue < this.orderedBridges.size()) { + while (!this.orderedBridges.isEmpty() + && limitValue < this.orderedBridges.size()) { this.orderedBridges.remove(this.orderedBridges.size() - 1); } } private List<SummaryDocument> orderedRelays = new ArrayList<SummaryDocument>(); + public List<SummaryDocument> getOrderedRelays() { return this.orderedRelays; } private List<SummaryDocument> orderedBridges = new ArrayList<SummaryDocument>(); + public List<SummaryDocument> getOrderedBridges() { return this.orderedBridges; } @@ -580,3 +598,4 @@ public class RequestHandler { return this.nodeIndex.getBridgesPublishedString(); } } + diff --git a/src/main/java/org/torproject/onionoo/server/ResourceServlet.java b/src/main/java/org/torproject/onionoo/server/ResourceServlet.java index 9aa460d..8430bdc 100644 --- a/src/main/java/org/torproject/onionoo/server/ResourceServlet.java +++ b/src/main/java/org/torproject/onionoo/server/ResourceServlet.java @@ -1,7 +1,11 @@ /* Copyright 2011, 2012 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; +import org.torproject.onionoo.util.Time; +import org.torproject.onionoo.util.TimeFactory; + import java.io.IOException; import java.io.PrintWriter; import java.util.ArrayList; @@ -20,9 +24,6 @@ import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; -import org.torproject.onionoo.util.Time; -import org.torproject.onionoo.util.TimeFactory; - public class ResourceServlet extends HttpServlet { private static final long serialVersionUID = 7236658979947465319L; @@ -32,9 +33,8 @@ public class ResourceServlet extends HttpServlet { /* Called by servlet container, not by test class. */ public void init(ServletConfig config) throws ServletException { super.init(config); - this.maintenanceMode = - config.getInitParameter("maintenance") != null && - config.getInitParameter("maintenance").equals("1"); + this.maintenanceMode = config.getInitParameter("maintenance") != null + && config.getInitParameter("maintenance").equals("1"); } private static final long INDEX_WAITING_TIME = 10L * 1000L; @@ -57,9 +57,11 @@ public class ResourceServlet extends HttpServlet { this.doGet(requestWrapper, responseWrapper); } - private static final long CACHE_MIN_TIME = 5L * 60L * 1000L, - CACHE_MAX_TIME = 45L * 60L * 1000L, - CACHE_INTERVAL = 5L * 60L * 1000L; + private static final long CACHE_MIN_TIME = 5L * 60L * 1000L; + + private static final long CACHE_MAX_TIME = 45L * 60L * 1000L; + + private static final long CACHE_INTERVAL = 5L * 60L * 1000L; private static Set<String> knownParameters = new HashSet<String>( Arrays.asList(("type,running,search,lookup,fingerprint,country,as," @@ -87,8 +89,8 @@ public class ResourceServlet extends HttpServlet { ((CACHE_MAX_TIME - indexAgeMillis) / CACHE_INTERVAL) * CACHE_INTERVAL); - NodeIndex nodeIndex = NodeIndexerFactory.getNodeIndexer(). - getLatestNodeIndex(INDEX_WAITING_TIME); + NodeIndex nodeIndex = NodeIndexerFactory.getNodeIndexer() + .getLatestNodeIndex(INDEX_WAITING_TIME); if (nodeIndex == null) { response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); return; @@ -153,8 +155,8 @@ public class ResourceServlet extends HttpServlet { if (searchTerm.contains(":") && !searchTerm.startsWith("[")) { String[] parts = searchTerm.split(":", 2); String parameterKey = parts[0]; - if (!knownParameters.contains(parameterKey) || - illegalSearchQualifiers.contains(parameterKey)) { + if (!knownParameters.contains(parameterKey) + || illegalSearchQualifiers.contains(parameterKey)) { response.sendError(HttpServletResponse.SC_BAD_REQUEST); return; } @@ -356,12 +358,14 @@ public class ResourceServlet extends HttpServlet { Pattern.compile("(?:.*[\\?&])*?" // lazily skip other parameters + "search=([0-9a-zA-Z+/\\.: \\$\\[\\]%]+)" // capture parameter + "(?:&.*)*"); // skip remaining parameters + private static Pattern searchParameterPattern = Pattern.compile("^\\$?[0-9a-fA-F]{1,40}$|" /* Hex fingerprint. */ + "^[0-9a-zA-Z+/]{1,27}$|" /* Base64 fingerprint. */ + "^[0-9a-zA-Z\\.]{1,19}$|" /* Nickname or IPv4 address. */ + "^\\[[0-9a-fA-F:\\.]{1,39}\\]?$|" /* IPv6 address. */ + "^[a-zA-Z_]+:[0-9a-zA-Z_,-]+$" /* Qualified search term. */); + protected static String[] parseSearchParameters(String queryString) { Matcher searchQueryStringMatcher = searchQueryStringPattern.matcher( queryString); @@ -383,6 +387,7 @@ public class ResourceServlet extends HttpServlet { private static Pattern fingerprintParameterPattern = Pattern.compile("^[0-9a-zA-Z]{1,40}$"); + private String parseFingerprintParameter(String parameter) { if (!fingerprintParameterPattern.matcher(parameter).matches()) { /* Fingerprint contains non-hex character(s). */ @@ -397,6 +402,7 @@ public class ResourceServlet extends HttpServlet { private static Pattern countryCodeParameterPattern = Pattern.compile("^[0-9a-zA-Z]{2}$"); + private String parseCountryCodeParameter(String parameter) { if (!countryCodeParameterPattern.matcher(parameter).matches()) { /* Country code contains illegal characters or is shorter/longer @@ -408,6 +414,7 @@ public class ResourceServlet extends HttpServlet { private static Pattern aSNumberParameterPattern = Pattern.compile("^[asAS]{0,2}[0-9]{1,10}$"); + private String parseASNumberParameter(String parameter) { if (!aSNumberParameterPattern.matcher(parameter).matches()) { /* AS number contains illegal character(s). */ @@ -418,6 +425,7 @@ public class ResourceServlet extends HttpServlet { private static Pattern flagPattern = Pattern.compile("^[a-zA-Z0-9]{1,20}$"); + private String parseFlagParameter(String parameter) { if (!flagPattern.matcher(parameter).matches()) { /* Flag contains illegal character(s). */ @@ -427,12 +435,14 @@ public class ResourceServlet extends HttpServlet { } private static Pattern daysPattern = Pattern.compile("^[0-9-]{1,10}$"); + private int[] parseDaysParameter(String parameter) { if (!daysPattern.matcher(parameter).matches()) { /* Days contain illegal character(s). */ return null; } - int x = 0, y = Integer.MAX_VALUE; + int x = 0; + int y = Integer.MAX_VALUE; try { if (!parameter.contains("-")) { x = Integer.parseInt(parameter); @@ -469,6 +479,7 @@ public class ResourceServlet extends HttpServlet { private static Pattern fieldsParameterPattern = Pattern.compile("^[0-9a-zA-Z_,]*$"); + private String[] parseFieldsParameter(String parameter) { if (!fieldsParameterPattern.matcher(parameter).matches()) { /* Fields contain illegal character(s). */ diff --git a/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java b/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java index 76790d5..96627d3 100644 --- a/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java +++ b/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java @@ -1,12 +1,8 @@ /* Copyright 2011--2013 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.server; -import java.io.PrintWriter; -import java.util.ArrayList; -import java.util.List; +package org.torproject.onionoo.server; -import org.apache.commons.lang3.StringUtils; import org.torproject.onionoo.docs.BandwidthDocument; import org.torproject.onionoo.docs.ClientsDocument; import org.torproject.onionoo.docs.DetailsDocument; @@ -19,6 +15,12 @@ import org.torproject.onionoo.docs.WeightsDocument; import com.google.gson.Gson; import com.google.gson.GsonBuilder; +import org.apache.commons.lang3.StringUtils; + +import java.io.PrintWriter; +import java.util.ArrayList; +import java.util.List; + public class ResponseBuilder { private DocumentStore documentStore; @@ -28,33 +30,39 @@ public class ResponseBuilder { } private String resourceType; + public void setResourceType(String resourceType) { this.resourceType = resourceType; } private String relaysPublishedString; + public void setRelaysPublishedString(String relaysPublishedString) { this.relaysPublishedString = relaysPublishedString; } private String bridgesPublishedString; + public void setBridgesPublishedString(String bridgesPublishedString) { this.bridgesPublishedString = bridgesPublishedString; } private List<SummaryDocument> orderedRelays = new ArrayList<SummaryDocument>(); + public void setOrderedRelays(List<SummaryDocument> orderedRelays) { this.orderedRelays = orderedRelays; } private List<SummaryDocument> orderedBridges = new ArrayList<SummaryDocument>(); + public void setOrderedBridges(List<SummaryDocument> orderedBridges) { this.orderedBridges = orderedBridges; } private String[] fields; + public void setFields(String[] fields) { this.fields = new String[fields.length]; System.arraycopy(fields, 0, this.fields, 0, fields.length); @@ -66,6 +74,7 @@ public class ResponseBuilder { } private int charsWritten = 0; + public int getCharsWritten() { return this.charsWritten; } @@ -144,8 +153,8 @@ public class ResponseBuilder { } private String writeRelaySummaryLine(SummaryDocument entry) { - String nickname = !entry.getNickname().equals("Unnamed") ? - entry.getNickname() : null; + String nickname = !entry.getNickname().equals("Unnamed") + ? entry.getNickname() : null; String fingerprint = entry.getFingerprint(); String running = entry.isRunning() ? "true" : "false"; List<String> addresses = entry.getAddresses(); @@ -161,8 +170,8 @@ public class ResponseBuilder { } private String writeBridgeSummaryLine(SummaryDocument entry) { - String nickname = !entry.getNickname().equals("Unnamed") ? - entry.getNickname() : null; + String nickname = !entry.getNickname().equals("Unnamed") + ? entry.getNickname() : null; String hashedFingerprint = entry.getFingerprint(); String running = entry.isRunning() ? "true" : "false"; return String.format("{%s\"h\":\"%s\",\"r\":%s}", @@ -304,8 +313,8 @@ public class ResponseBuilder { String fingerprint = entry.getFingerprint(); BandwidthDocument bandwidthDocument = this.documentStore.retrieve( BandwidthDocument.class, false, fingerprint); - if (bandwidthDocument != null && - bandwidthDocument.getDocumentString() != null) { + if (bandwidthDocument != null + && bandwidthDocument.getDocumentString() != null) { return bandwidthDocument.getDocumentString(); } else { return "{\"fingerprint\":\"" + fingerprint.toUpperCase() + "\"}"; @@ -316,8 +325,8 @@ public class ResponseBuilder { String fingerprint = entry.getFingerprint(); WeightsDocument weightsDocument = this.documentStore.retrieve( WeightsDocument.class, false, fingerprint); - if (weightsDocument != null && - weightsDocument.getDocumentString() != null) { + if (weightsDocument != null + && weightsDocument.getDocumentString() != null) { return weightsDocument.getDocumentString(); } else { return "{\"fingerprint\":\"" + fingerprint.toUpperCase() + "\"}"; @@ -328,8 +337,8 @@ public class ResponseBuilder { String fingerprint = entry.getFingerprint(); ClientsDocument clientsDocument = this.documentStore.retrieve( ClientsDocument.class, false, fingerprint); - if (clientsDocument != null && - clientsDocument.getDocumentString() != null) { + if (clientsDocument != null + && clientsDocument.getDocumentString() != null) { return clientsDocument.getDocumentString(); } else { return "{\"fingerprint\":\"" + fingerprint.toUpperCase() + "\"}"; @@ -340,11 +349,12 @@ public class ResponseBuilder { String fingerprint = entry.getFingerprint(); UptimeDocument uptimeDocument = this.documentStore.retrieve( UptimeDocument.class, false, fingerprint); - if (uptimeDocument != null && - uptimeDocument.getDocumentString() != null) { + if (uptimeDocument != null + && uptimeDocument.getDocumentString() != null) { return uptimeDocument.getDocumentString(); } else { return "{\"fingerprint\":\"" + fingerprint.toUpperCase() + "\"}"; } } } + diff --git a/src/main/java/org/torproject/onionoo/server/ServerMain.java b/src/main/java/org/torproject/onionoo/server/ServerMain.java index d109a32..de3a0d9 100644 --- a/src/main/java/org/torproject/onionoo/server/ServerMain.java +++ b/src/main/java/org/torproject/onionoo/server/ServerMain.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.server; import org.eclipse.jetty.server.Server; diff --git a/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java index 9a7717c..6286741 100644 --- a/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java @@ -1,5 +1,6 @@ /* Copyright 2011--2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; import org.torproject.descriptor.Descriptor; diff --git a/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java index 72f5a92..1b55273 100644 --- a/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java @@ -1,12 +1,7 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.updater; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeMap; -import java.util.TreeSet; +package org.torproject.onionoo.updater; import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.ExtraInfoDescriptor; @@ -17,6 +12,12 @@ import org.torproject.onionoo.docs.DocumentStore; import org.torproject.onionoo.docs.DocumentStoreFactory; import org.torproject.onionoo.util.FormattingUtils; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; + /* * Example extra-info descriptor used as input: * @@ -70,9 +71,9 @@ public class ClientsStatusUpdater implements DescriptorListener, descriptor.getDirreqStatsIntervalLength() * DateTimeHelper.ONE_SECOND; SortedMap<String, Integer> responses = descriptor.getDirreqV3Resp(); - if (dirreqStatsEndMillis < 0L || - dirreqStatsIntervalLengthMillis != DateTimeHelper.ONE_DAY || - responses == null || !responses.containsKey("ok")) { + if (dirreqStatsEndMillis < 0L + || dirreqStatsIntervalLengthMillis != DateTimeHelper.ONE_DAY + || responses == null || !responses.containsKey("ok")) { /* No directory request responses in the descriptor that we would * include in a clients document. */ return; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java b/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java index 2655b1a..64dec9b 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java @@ -1,5 +1,8 @@ package org.torproject.onionoo.updater; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.BufferedInputStream; import java.io.BufferedOutputStream; import java.io.BufferedReader; @@ -14,9 +17,6 @@ import java.util.SortedSet; import java.util.TreeSet; import java.util.zip.GZIPInputStream; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - class DescriptorDownloader { private static Logger log = LoggerFactory.getLogger( @@ -31,30 +31,30 @@ class DescriptorDownloader { public DescriptorDownloader(DescriptorType descriptorType) { switch (descriptorType) { - case RELAY_CONSENSUSES: - this.directory = "relay-descriptors/consensuses/"; - break; - case RELAY_SERVER_DESCRIPTORS: - this.directory = "relay-descriptors/server-descriptors/"; - break; - case RELAY_EXTRA_INFOS: - this.directory = "relay-descriptors/extra-infos/"; - break; - case EXIT_LISTS: - this.directory = "exit-lists/"; - break; - case BRIDGE_STATUSES: - this.directory = "bridge-descriptors/statuses/"; - break; - case BRIDGE_SERVER_DESCRIPTORS: - this.directory = "bridge-descriptors/server-descriptors/"; - break; - case BRIDGE_EXTRA_INFOS: - this.directory = "bridge-descriptors/extra-infos/"; - break; - default: - log.error("Unknown descriptor type."); - return; + case RELAY_CONSENSUSES: + this.directory = "relay-descriptors/consensuses/"; + break; + case RELAY_SERVER_DESCRIPTORS: + this.directory = "relay-descriptors/server-descriptors/"; + break; + case RELAY_EXTRA_INFOS: + this.directory = "relay-descriptors/extra-infos/"; + break; + case EXIT_LISTS: + this.directory = "exit-lists/"; + break; + case BRIDGE_STATUSES: + this.directory = "bridge-descriptors/statuses/"; + break; + case BRIDGE_SERVER_DESCRIPTORS: + this.directory = "bridge-descriptors/server-descriptors/"; + break; + case BRIDGE_EXTRA_INFOS: + this.directory = "bridge-descriptors/extra-infos/"; + break; + default: + log.error("Unknown descriptor type."); + return; } } @@ -90,8 +90,8 @@ class DescriptorDownloader { huc.getInputStream()))) { String line; while ((line = br.readLine()) != null) { - if (!line.trim().startsWith("<tr>") || - !line.contains("<a href=\"")) { + if (!line.trim().startsWith("<tr>") + || !line.contains("<a href=\"")) { continue; } String linePart = line.substring( @@ -108,7 +108,7 @@ class DescriptorDownloader { } } catch (IOException e) { log.error("Could not fetch or parse " + directoryUrl - + ". Skipping. Reason: " + e.getMessage()); + + ". Skipping. Reason: " + e.getMessage()); } return this.remoteFiles.size(); } @@ -139,8 +139,8 @@ class DescriptorDownloader { } long lastModified = huc.getHeaderFieldDate("Last-Modified", -1L); InputStream is; - if (huc.getContentEncoding() != null && - huc.getContentEncoding().equalsIgnoreCase("gzip")) { + if (huc.getContentEncoding() != null + && huc.getContentEncoding().equalsIgnoreCase("gzip")) { is = new GZIPInputStream(huc.getInputStream()); } else { is = huc.getInputStream(); @@ -178,3 +178,4 @@ class DescriptorDownloader { return deletedFiles; } } + diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java b/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java index e6d911e..f0f9eba 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java @@ -8,4 +8,5 @@ enum DescriptorHistory { BRIDGE_STATUS_HISTORY, BRIDGE_SERVER_HISTORY, BRIDGE_EXTRAINFO_HISTORY, -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java b/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java index 3613879..a93bef6 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java @@ -4,4 +4,5 @@ import org.torproject.descriptor.Descriptor; public interface DescriptorListener { abstract void processDescriptor(Descriptor descriptor, boolean relay); -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java b/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java index 6010bbf..7892f40 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java @@ -1,5 +1,13 @@ package org.torproject.onionoo.updater; +import org.torproject.descriptor.Descriptor; +import org.torproject.descriptor.DescriptorFile; +import org.torproject.descriptor.DescriptorReader; +import org.torproject.descriptor.DescriptorSourceFactory; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.BufferedReader; import java.io.BufferedWriter; import java.io.File; @@ -12,16 +20,9 @@ import java.util.Map; import java.util.SortedMap; import java.util.TreeMap; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorFile; -import org.torproject.descriptor.DescriptorReader; -import org.torproject.descriptor.DescriptorSourceFactory; - class DescriptorQueue { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( DescriptorQueue.class); private File statusDir; @@ -35,21 +36,25 @@ class DescriptorQueue { private List<Descriptor> descriptors; private int historySizeBefore; + public int getHistorySizeBefore() { return this.historySizeBefore; } private int historySizeAfter; + public int getHistorySizeAfter() { return this.historySizeAfter; } private long returnedDescriptors = 0L; + public long getReturnedDescriptors() { return this.returnedDescriptors; } private long returnedBytes = 0L; + public long getReturnedBytes() { return this.returnedBytes; } @@ -71,31 +76,31 @@ class DescriptorQueue { DescriptorType descriptorType) { String directoryName = null; switch (descriptorType) { - case RELAY_CONSENSUSES: - directoryName = "relay-descriptors/consensuses"; - break; - case RELAY_SERVER_DESCRIPTORS: - directoryName = "relay-descriptors/server-descriptors"; - break; - case RELAY_EXTRA_INFOS: - directoryName = "relay-descriptors/extra-infos"; - break; - case BRIDGE_STATUSES: - directoryName = "bridge-descriptors/statuses"; - break; - case BRIDGE_SERVER_DESCRIPTORS: - directoryName = "bridge-descriptors/server-descriptors"; - break; - case BRIDGE_EXTRA_INFOS: - directoryName = "bridge-descriptors/extra-infos"; - break; - case EXIT_LISTS: - directoryName = "exit-lists"; - break; - default: - log.error("Unknown descriptor type. Not adding directory " - + "to descriptor reader."); - return null; + case RELAY_CONSENSUSES: + directoryName = "relay-descriptors/consensuses"; + break; + case RELAY_SERVER_DESCRIPTORS: + directoryName = "relay-descriptors/server-descriptors"; + break; + case RELAY_EXTRA_INFOS: + directoryName = "relay-descriptors/extra-infos"; + break; + case BRIDGE_STATUSES: + directoryName = "bridge-descriptors/statuses"; + break; + case BRIDGE_SERVER_DESCRIPTORS: + directoryName = "bridge-descriptors/server-descriptors"; + break; + case BRIDGE_EXTRA_INFOS: + directoryName = "bridge-descriptors/extra-infos"; + break; + case EXIT_LISTS: + directoryName = "exit-lists"; + break; + default: + log.error("Unknown descriptor type. Not adding directory " + + "to descriptor reader."); + return null; } return new File(inDir, directoryName); } @@ -120,31 +125,31 @@ class DescriptorQueue { } String historyFileName = null; switch (descriptorHistory) { - case RELAY_EXTRAINFO_HISTORY: - historyFileName = "relay-extrainfo-history"; - break; - case BRIDGE_EXTRAINFO_HISTORY: - historyFileName = "bridge-extrainfo-history"; - break; - case EXIT_LIST_HISTORY: - historyFileName = "exit-list-history"; - break; - case RELAY_CONSENSUS_HISTORY: - historyFileName = "relay-consensus-history"; - break; - case BRIDGE_STATUS_HISTORY: - historyFileName = "bridge-status-history"; - break; - case RELAY_SERVER_HISTORY: - historyFileName = "relay-server-history"; - break; - case BRIDGE_SERVER_HISTORY: - historyFileName = "bridge-server-history"; - break; - default: - log.error("Unknown descriptor history. Not excluding " - + "files."); - return; + case RELAY_EXTRAINFO_HISTORY: + historyFileName = "relay-extrainfo-history"; + break; + case BRIDGE_EXTRAINFO_HISTORY: + historyFileName = "bridge-extrainfo-history"; + break; + case EXIT_LIST_HISTORY: + historyFileName = "exit-list-history"; + break; + case RELAY_CONSENSUS_HISTORY: + historyFileName = "relay-consensus-history"; + break; + case BRIDGE_STATUS_HISTORY: + historyFileName = "bridge-status-history"; + break; + case RELAY_SERVER_HISTORY: + historyFileName = "relay-server-history"; + break; + case BRIDGE_SERVER_HISTORY: + historyFileName = "bridge-server-history"; + break; + default: + log.error("Unknown descriptor history. Not excluding " + + "files."); + return; } this.historyFile = new File(this.statusDir, historyFileName); if (this.historyFile.exists() && this.historyFile.isFile()) { @@ -209,8 +214,8 @@ class DescriptorQueue { log.error("Could not parse " + descriptorFile.getFileName(), descriptorFile.getException()); } - if (descriptorFile.getDescriptors() != null && - !descriptorFile.getDescriptors().isEmpty()) { + if (descriptorFile.getDescriptors() != null + && !descriptorFile.getDescriptors().isEmpty()) { this.descriptors = descriptorFile.getDescriptors(); } } @@ -225,3 +230,4 @@ class DescriptorQueue { return nextDescriptor; } } + diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java b/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java index a1e489e..05e9bd8 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java @@ -1,7 +1,14 @@ /* Copyright 2013, 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; +import org.torproject.descriptor.Descriptor; +import org.torproject.onionoo.util.FormattingUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.File; import java.util.ArrayList; import java.util.HashMap; @@ -10,18 +17,14 @@ import java.util.List; import java.util.Map; import java.util.Set; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.descriptor.Descriptor; -import org.torproject.onionoo.util.FormattingUtils; - public class DescriptorSource { private static final Logger log = LoggerFactory.getLogger( DescriptorSource.class); - private final File inRecentDir = new File("in/recent"), - inArchiveDir = new File("in/archive"); + private final File inRecentDir = new File("in/recent"); + + private final File inArchiveDir = new File("in/archive"); private final File statusDir = new File("status"); @@ -66,8 +69,13 @@ public class DescriptorSource { } } - private int localFilesBefore = 0, foundRemoteFiles = 0, - downloadedFiles = 0, deletedLocalFiles = 0; + private int localFilesBefore = 0; + + private int foundRemoteFiles = 0; + + private int downloadedFiles = 0; + + private int deletedLocalFiles = 0; private void downloadDescriptors(DescriptorType descriptorType) { DescriptorDownloader descriptorDownloader = @@ -122,27 +130,27 @@ public class DescriptorSource { } } switch (descriptorType) { - case RELAY_CONSENSUSES: - log.info("Read recent relay network consensuses"); - break; - case RELAY_SERVER_DESCRIPTORS: - log.info("Read recent relay server descriptors"); - break; - case RELAY_EXTRA_INFOS: - log.info("Read recent relay extra-info descriptors"); - break; - case EXIT_LISTS: - log.info("Read recent exit lists"); - break; - case BRIDGE_STATUSES: - log.info("Read recent bridge network statuses"); - break; - case BRIDGE_SERVER_DESCRIPTORS: - log.info("Read recent bridge server descriptors"); - break; - case BRIDGE_EXTRA_INFOS: - log.info("Read recent bridge extra-info descriptors"); - break; + case RELAY_CONSENSUSES: + log.info("Read recent relay network consensuses"); + break; + case RELAY_SERVER_DESCRIPTORS: + log.info("Read recent relay server descriptors"); + break; + case RELAY_EXTRA_INFOS: + log.info("Read recent relay extra-info descriptors"); + break; + case EXIT_LISTS: + log.info("Read recent exit lists"); + break; + case BRIDGE_STATUSES: + log.info("Read recent bridge network statuses"); + break; + case BRIDGE_SERVER_DESCRIPTORS: + log.info("Read recent bridge server descriptors"); + break; + case BRIDGE_EXTRA_INFOS: + log.info("Read recent bridge extra-info descriptors"); + break; } } @@ -219,8 +227,10 @@ public class DescriptorSource { + "files deleted locally\n"); sb.append(" " + this.descriptorQueues.size() + " descriptor " + "queues created for recent descriptors\n"); - int historySizeBefore = 0, historySizeAfter = 0; - long descriptors = 0L, bytes = 0L; + int historySizeBefore = 0; + int historySizeAfter = 0; + long descriptors = 0L; + long bytes = 0L; for (DescriptorQueue descriptorQueue : this.descriptorQueues) { historySizeBefore += descriptorQueue.getHistorySizeBefore(); historySizeAfter += descriptorQueue.getHistorySizeAfter(); @@ -239,8 +249,8 @@ public class DescriptorSource { + "execution\n"); if (this.archiveDescriptorQueue != null) { sb.append(" " + FormattingUtils.formatDecimalNumber( - this.archiveDescriptorQueue.getReturnedDescriptors()) + - " archived descriptors provided\n"); + this.archiveDescriptorQueue.getReturnedDescriptors()) + + " archived descriptors provided\n"); sb.append(" " + FormattingUtils.formatBytes( this.archiveDescriptorQueue.getReturnedBytes()) + " of " + "archived descriptors provided\n"); diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java b/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java index b495120..a68cd9f 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java @@ -1,14 +1,17 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; public class DescriptorSourceFactory { private static DescriptorSource descriptorSourceInstance; + public static void setDescriptorSource( DescriptorSource descriptorSource) { descriptorSourceInstance = descriptorSource; } + public static DescriptorSource getDescriptorSource() { if (descriptorSourceInstance == null) { descriptorSourceInstance = new DescriptorSource(); diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorType.java b/src/main/java/org/torproject/onionoo/updater/DescriptorType.java index 60d2759..98c345a 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorType.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorType.java @@ -1,5 +1,6 @@ /* Copyright 2013, 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; public enum DescriptorType { diff --git a/src/main/java/org/torproject/onionoo/updater/LookupResult.java b/src/main/java/org/torproject/onionoo/updater/LookupResult.java index dcf3a2a..476783d 100644 --- a/src/main/java/org/torproject/onionoo/updater/LookupResult.java +++ b/src/main/java/org/torproject/onionoo/updater/LookupResult.java @@ -1,70 +1,88 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; public class LookupResult { private String countryCode; + public void setCountryCode(String countryCode) { this.countryCode = countryCode; } + public String getCountryCode() { return this.countryCode; } private String countryName; + public void setCountryName(String countryName) { this.countryName = countryName; } + public String getCountryName() { return this.countryName; } private String regionName; + public void setRegionName(String regionName) { this.regionName = regionName; } + public String getRegionName() { return this.regionName; } private String cityName; + public void setCityName(String cityName) { this.cityName = cityName; } + public String getCityName() { return this.cityName; } private Float latitude; + public void setLatitude(Float latitude) { this.latitude = latitude; } + public Float getLatitude() { return this.latitude; } private Float longitude; + public void setLongitude(Float longitude) { this.longitude = longitude; } + public Float getLongitude() { return this.longitude; } private String asNumber; + public void setAsNumber(String asNumber) { this.asNumber = asNumber; } + public String getAsNumber() { return this.asNumber; } private String asName; + public void setAsName(String asName) { this.asName = asName; } + public String getAsName() { return this.asName; } -} \ No newline at end of file +} + diff --git a/src/main/java/org/torproject/onionoo/updater/LookupService.java b/src/main/java/org/torproject/onionoo/updater/LookupService.java index 57a0885..357ff4f 100644 --- a/src/main/java/org/torproject/onionoo/updater/LookupService.java +++ b/src/main/java/org/torproject/onionoo/updater/LookupService.java @@ -1,7 +1,13 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; +import org.torproject.onionoo.util.FormattingUtils; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.BufferedReader; import java.io.File; import java.io.FileInputStream; @@ -22,20 +28,21 @@ import java.util.TreeMap; import java.util.TreeSet; import java.util.regex.Pattern; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.onionoo.util.FormattingUtils; - public class LookupService { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( LookupService.class); private File geoipDir; + private File geoLite2CityBlocksIPv4CsvFile; + private File geoLite2CityLocationsEnCsvFile; + private File geoIPASNum2CsvFile; + private boolean hasAllFiles = false; + public LookupService(File geoipDir) { this.geoipDir = geoipDir; this.findRequiredCsvFiles(); @@ -65,6 +72,7 @@ public class LookupService { } private Pattern ipv4Pattern = Pattern.compile("^[0-9\\.]{7,15}$"); + private long parseAddressString(String addressString) { long addressNumber = -1L; if (ipv4Pattern.matcher(addressString).matches()) { @@ -153,10 +161,10 @@ public class LookupService { } long endIpNum = startIpNum + (1 << (32 - networkMaskLength)) - 1; - for (long addressNumber : sortedAddressNumbers. - tailSet(startIpNum).headSet(endIpNum + 1L)) { - String blockString = parts[1].length() > 0 ? parts[1] : - parts[2]; + for (long addressNumber : sortedAddressNumbers + .tailSet(startIpNum).headSet(endIpNum + 1L)) { + String blockString = parts[1].length() > 0 ? parts[1] + : parts[2]; long blockNumber = Long.parseLong(blockString); addressNumberBlocks.put(addressNumber, blockNumber); if (parts[7].length() > 0 && parts[8].length() > 0) { @@ -240,8 +248,8 @@ public class LookupService { return lookupResults; } previousStartIpNum = startIpNum; - while (firstAddressNumber < startIpNum && - firstAddressNumber != -1L) { + while (firstAddressNumber < startIpNum + && firstAddressNumber != -1L) { sortedAddressNumbers.remove(firstAddressNumber); if (sortedAddressNumbers.isEmpty()) { firstAddressNumber = -1L; @@ -250,8 +258,8 @@ public class LookupService { } } long endIpNum = Long.parseLong(parts[1]); - while (firstAddressNumber <= endIpNum && - firstAddressNumber != -1L) { + while (firstAddressNumber <= endIpNum + && firstAddressNumber != -1L) { if (parts[2].startsWith("AS")) { addressNumberASN.put(firstAddressNumber, parts[2]); } @@ -284,17 +292,17 @@ public class LookupService { continue; } long addressNumber = addressStringNumbers.get(addressString); - if (!addressNumberBlocks.containsKey(addressNumber) && - !addressNumberLatLong.containsKey(addressNumber) && - !addressNumberASN.containsKey(addressNumber)) { + if (!addressNumberBlocks.containsKey(addressNumber) + && !addressNumberLatLong.containsKey(addressNumber) + && !addressNumberASN.containsKey(addressNumber)) { continue; } LookupResult lookupResult = new LookupResult(); if (addressNumberBlocks.containsKey(addressNumber)) { long blockNumber = addressNumberBlocks.get(addressNumber); if (blockLocations.containsKey(blockNumber)) { - String[] parts = blockLocations.get(blockNumber). - replaceAll("\"", "").split(",", -1); + String[] parts = blockLocations.get(blockNumber) + .replaceAll("\"", "").split(",", -1); if (parts[4].length() > 0) { lookupResult.setCountryCode(parts[4].toLowerCase()); } @@ -351,7 +359,9 @@ public class LookupService { new FileInputStream(file), dec)); } - private int addressesLookedUp = 0, addressesResolved = 0; + private int addressesLookedUp = 0; + + private int addressesResolved = 0; public String getStatsString() { StringBuilder sb = new StringBuilder(); @@ -362,3 +372,4 @@ public class LookupService { return sb.toString(); } } + diff --git a/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java index d282ab8..ef7246a 100644 --- a/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java @@ -1,20 +1,8 @@ /* Copyright 2011--2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.updater; -import java.util.Arrays; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeMap; -import java.util.TreeSet; +package org.torproject.onionoo.updater; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.descriptor.BridgeNetworkStatus; import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.ExitList; @@ -31,6 +19,20 @@ import org.torproject.onionoo.docs.NodeStatus; import org.torproject.onionoo.util.FormattingUtils; import org.torproject.onionoo.util.TimeFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; + /* * Status updater for both node and details statuses. * @@ -90,7 +92,9 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, private SortedMap<String, Integer> lastBandwidthWeights = null; - private int relayConsensusesProcessed = 0, bridgeStatusesProcessed = 0; + private int relayConsensusesProcessed = 0; + + private int bridgeStatusesProcessed = 0; public NodeDetailsStatusUpdater( ReverseDomainNameResolver reverseDomainNameResolver, @@ -150,9 +154,9 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, DetailsStatus.class, true, fingerprint); if (detailsStatus == null) { detailsStatus = new DetailsStatus(); - } else if (detailsStatus.getDescPublished() != null && - detailsStatus.getDescPublished() >= - descriptor.getPublishedMillis()) { + } else if (detailsStatus.getDescPublished() != null + && detailsStatus.getDescPublished() + >= descriptor.getPublishedMillis()) { /* Already parsed more recent server descriptor from this relay. */ return; } @@ -184,10 +188,10 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, } this.declaredFamilies.put(fingerprint, declaredFamily); } - if (descriptor.getIpv6DefaultPolicy() != null && - (descriptor.getIpv6DefaultPolicy().equals("accept") || - descriptor.getIpv6DefaultPolicy().equals("reject")) && - descriptor.getIpv6PortList() != null) { + if (descriptor.getIpv6DefaultPolicy() != null + && (descriptor.getIpv6DefaultPolicy().equals("accept") + || descriptor.getIpv6DefaultPolicy().equals("reject")) + && descriptor.getIpv6PortList() != null) { Map<String, List<String>> exitPolicyV6Summary = new HashMap<String, List<String>>(); List<String> portsOrPortRanges = Arrays.asList( @@ -225,9 +229,11 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, } } - private Map<String, Long> - lastSeenUnmeasured = new HashMap<String, Long>(), - lastSeenMeasured = new HashMap<String, Long>(); + private Map<String, Long> lastSeenUnmeasured = + new HashMap<String, Long>(); + + private Map<String, Long> lastSeenMeasured = + new HashMap<String, Long>(); private void processRelayNetworkStatusConsensus( RelayNetworkStatusConsensus consensus) { @@ -259,8 +265,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, entry.getOrAddresses()); nodeStatus.addLastAddresses(validAfterMillis, address, orPort, dirPort, orAddressesAndPorts); - if (nodeStatus.getFirstSeenMillis() == 0L || - validAfterMillis < nodeStatus.getFirstSeenMillis()) { + if (nodeStatus.getFirstSeenMillis() == 0L + || validAfterMillis < nodeStatus.getFirstSeenMillis()) { nodeStatus.setFirstSeenMillis(validAfterMillis); } if (validAfterMillis > nodeStatus.getLastSeenMillis()) { @@ -275,18 +281,20 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, nodeStatus.setConsensusWeight(entry.getBandwidth()); nodeStatus.setDefaultPolicy(entry.getDefaultPolicy()); nodeStatus.setPortList(entry.getPortList()); - nodeStatus.setRecommendedVersion((recommendedVersions == null || - entry.getVersion() == null) ? null : + nodeStatus.setRecommendedVersion((recommendedVersions == null + || entry.getVersion() == null) ? null : recommendedVersions.contains(entry.getVersion())); } if (entry.getUnmeasured()) { - if (!this.lastSeenUnmeasured.containsKey(fingerprint) || - this.lastSeenUnmeasured.get(fingerprint) < validAfterMillis) { + if (!this.lastSeenUnmeasured.containsKey(fingerprint) + || this.lastSeenUnmeasured.get(fingerprint) + < validAfterMillis) { this.lastSeenUnmeasured.put(fingerprint, validAfterMillis); } } else if (consensus.getConsensusMethod() >= 17) { - if (!this.lastSeenMeasured.containsKey(fingerprint) || - this.lastSeenMeasured.get(fingerprint) < validAfterMillis) { + if (!this.lastSeenMeasured.containsKey(fingerprint) + || this.lastSeenMeasured.get(fingerprint) + < validAfterMillis) { this.lastSeenMeasured.put(fingerprint, validAfterMillis); } } @@ -304,9 +312,9 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, DetailsStatus.class, true, fingerprint); if (detailsStatus == null) { detailsStatus = new DetailsStatus(); - } else if (detailsStatus.getDescPublished() != null && - detailsStatus.getDescPublished() >= - descriptor.getPublishedMillis()) { + } else if (detailsStatus.getDescPublished() != null + && detailsStatus.getDescPublished() + >= descriptor.getPublishedMillis()) { /* Already parsed more recent server descriptor from this bridge. */ return; } @@ -330,9 +338,9 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, DetailsStatus.class, true, fingerprint); if (detailsStatus == null) { detailsStatus = new DetailsStatus(); - } else if (null == detailsStatus.getExtraInfoDescPublished() || - descriptor.getPublishedMillis() > - detailsStatus.getExtraInfoDescPublished()) { + } else if (null == detailsStatus.getExtraInfoDescPublished() + || descriptor.getPublishedMillis() + > detailsStatus.getExtraInfoDescPublished()) { detailsStatus.setExtraInfoDescPublished( descriptor.getPublishedMillis()); detailsStatus.setTransports(descriptor.getTransports()); @@ -355,8 +363,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, nodeStatus = new NodeStatus(fingerprint); this.knownNodes.put(fingerprint, nodeStatus); } - if (nodeStatus.getFirstSeenMillis() == 0L || - publishedMillis < nodeStatus.getFirstSeenMillis()) { + if (nodeStatus.getFirstSeenMillis() == 0L + || publishedMillis < nodeStatus.getFirstSeenMillis()) { nodeStatus.setFirstSeenMillis(publishedMillis); } if (publishedMillis > nodeStatus.getLastSeenMillis()) { @@ -364,7 +372,7 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, nodeStatus.setNickname(entry.getNickname()); nodeStatus.setAddress(entry.getAddress()); nodeStatus.setOrAddressesAndPorts(new TreeSet<String>( - entry.getOrAddresses())); + entry.getOrAddresses())); nodeStatus.setOrPort(entry.getOrPort()); nodeStatus.setDirPort(entry.getDirPort()); nodeStatus.setRelayFlags(entry.getFlags()); @@ -393,34 +401,35 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, /* Step 2: read node statuses from disk. */ - private SortedSet<String> currentRelays = new TreeSet<String>(), - runningRelays = new TreeSet<String>(); + private SortedSet<String> currentRelays = new TreeSet<String>(); + + private SortedSet<String> runningRelays = new TreeSet<String>(); private void readNodeStatuses() { SortedSet<String> previouslyKnownNodes = this.documentStore.list( NodeStatus.class); - long previousRelaysLastValidAfterMillis = -1L, - previousBridgesLastValidAfterMillis = -1L; + long previousRelaysLastValidAfterMillis = -1L; + long previousBridgesLastValidAfterMillis = -1L; for (String fingerprint : previouslyKnownNodes) { NodeStatus nodeStatus = this.documentStore.retrieve( NodeStatus.class, true, fingerprint); - if (nodeStatus.isRelay() && nodeStatus.getLastSeenMillis() > - previousRelaysLastValidAfterMillis) { + if (nodeStatus.isRelay() && nodeStatus.getLastSeenMillis() + > previousRelaysLastValidAfterMillis) { previousRelaysLastValidAfterMillis = nodeStatus.getLastSeenMillis(); - } else if (!nodeStatus.isRelay() && nodeStatus.getLastSeenMillis() > - previousBridgesLastValidAfterMillis) { + } else if (!nodeStatus.isRelay() && nodeStatus.getLastSeenMillis() + > previousBridgesLastValidAfterMillis) { previousBridgesLastValidAfterMillis = nodeStatus.getLastSeenMillis(); } } - if (previousRelaysLastValidAfterMillis > - this.relaysLastValidAfterMillis) { + if (previousRelaysLastValidAfterMillis + > this.relaysLastValidAfterMillis) { this.relaysLastValidAfterMillis = previousRelaysLastValidAfterMillis; } - if (previousBridgesLastValidAfterMillis > - this.bridgesLastPublishedMillis) { + if (previousBridgesLastValidAfterMillis + > this.bridgesLastPublishedMillis) { this.bridgesLastPublishedMillis = previousBridgesLastValidAfterMillis; } @@ -430,11 +439,11 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, String fingerprint = e.getKey(); NodeStatus nodeStatus = e.getValue(); this.updatedNodes.add(fingerprint); - if (nodeStatus.isRelay() && - nodeStatus.getLastSeenMillis() >= cutoff) { + if (nodeStatus.isRelay() + && nodeStatus.getLastSeenMillis() >= cutoff) { this.currentRelays.add(fingerprint); - if (nodeStatus.getLastSeenMillis() == - this.relaysLastValidAfterMillis) { + if (nodeStatus.getLastSeenMillis() + == this.relaysLastValidAfterMillis) { this.runningRelays.add(fingerprint); } } @@ -459,8 +468,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, updatedNodeStatus.addLastAddresses( nodeStatus.getLastChangedOrAddressOrPort(), address, orPort, dirPort, orAddressesAndPorts); - if (nodeStatus.getLastSeenMillis() > - updatedNodeStatus.getLastSeenMillis()) { + if (nodeStatus.getLastSeenMillis() + > updatedNodeStatus.getLastSeenMillis()) { updatedNodeStatus.setNickname(nodeStatus.getNickname()); updatedNodeStatus.setAddress(address); updatedNodeStatus.setOrAddressesAndPorts(orAddressesAndPorts); @@ -479,8 +488,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, updatedNodeStatus.setRecommendedVersion( nodeStatus.getRecommendedVersion()); } - if (nodeStatus.getFirstSeenMillis() < - updatedNodeStatus.getFirstSeenMillis()) { + if (nodeStatus.getFirstSeenMillis() + < updatedNodeStatus.getFirstSeenMillis()) { updatedNodeStatus.setFirstSeenMillis( nodeStatus.getFirstSeenMillis()); } @@ -493,9 +502,9 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, } else { updatedNodeStatus = nodeStatus; this.knownNodes.put(fingerprint, nodeStatus); - if (nodeStatus.getLastSeenMillis() == (nodeStatus.isRelay() ? - previousRelaysLastValidAfterMillis : - previousBridgesLastValidAfterMillis)) { + if (nodeStatus.getLastSeenMillis() == (nodeStatus.isRelay() + ? previousRelaysLastValidAfterMillis + : previousBridgesLastValidAfterMillis)) { /* This relay or bridge was previously running, but we didn't * parse any descriptors with its fingerprint. Make sure to * update its details status file later on, so it has the @@ -503,11 +512,11 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, this.updatedNodes.add(fingerprint); } } - if (updatedNodeStatus.isRelay() && - updatedNodeStatus.getLastSeenMillis() >= cutoff) { + if (updatedNodeStatus.isRelay() + && updatedNodeStatus.getLastSeenMillis() >= cutoff) { this.currentRelays.add(fingerprint); - if (updatedNodeStatus.getLastSeenMillis() == - this.relaysLastValidAfterMillis) { + if (updatedNodeStatus.getLastSeenMillis() + == this.relaysLastValidAfterMillis) { this.runningRelays.add(fingerprint); } } @@ -569,16 +578,28 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, } } - private SortedMap<String, Float> - consensusWeightFractions = new TreeMap<String, Float>(), - guardProbabilities = new TreeMap<String, Float>(), - middleProbabilities = new TreeMap<String, Float>(), - exitProbabilities = new TreeMap<String, Float>(); + private SortedMap<String, Float> consensusWeightFractions = + new TreeMap<String, Float>(); + + private SortedMap<String, Float> guardProbabilities = + new TreeMap<String, Float>(); + + private SortedMap<String, Float> middleProbabilities = + new TreeMap<String, Float>(); + + private SortedMap<String, Float> exitProbabilities = + new TreeMap<String, Float>(); private void calculatePathSelectionProbabilities() { boolean consensusContainsBandwidthWeights = false; - double wgg = 0.0, wgd = 0.0, wmg = 0.0, wmm = 0.0, wme = 0.0, - wmd = 0.0, wee = 0.0, wed = 0.0; + double wgg = 0.0; + double wgd = 0.0; + double wmg = 0.0; + double wmm = 0.0; + double wme = 0.0; + double wmd = 0.0; + double wee = 0.0; + double wed = 0.0; if (this.lastBandwidthWeights != null) { SortedSet<String> weightKeys = new TreeSet<String>(Arrays.asList( "Wgg,Wgd,Wmg,Wmm,Wme,Wmd,Wee,Wed".split(","))); @@ -601,27 +622,30 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, + "this execution."); return; } - SortedMap<String, Double> - consensusWeights = new TreeMap<String, Double>(), - guardWeights = new TreeMap<String, Double>(), - middleWeights = new TreeMap<String, Double>(), - exitWeights = new TreeMap<String, Double>(); + SortedMap<String, Double> consensusWeights = + new TreeMap<String, Double>(); + SortedMap<String, Double> guardWeights = + new TreeMap<String, Double>(); + SortedMap<String, Double> middleWeights = + new TreeMap<String, Double>(); + SortedMap<String, Double> exitWeights = + new TreeMap<String, Double>(); double totalConsensusWeight = 0.0; double totalGuardWeight = 0.0; double totalMiddleWeight = 0.0; double totalExitWeight = 0.0; for (String fingerprint : this.runningRelays) { NodeStatus nodeStatus = this.knownNodes.get(fingerprint); - boolean isExit = nodeStatus.getRelayFlags().contains("Exit") && - !nodeStatus.getRelayFlags().contains("BadExit"); + boolean isExit = nodeStatus.getRelayFlags().contains("Exit") + && !nodeStatus.getRelayFlags().contains("BadExit"); boolean isGuard = nodeStatus.getRelayFlags().contains("Guard"); double consensusWeight = (double) nodeStatus.getConsensusWeight(); consensusWeights.put(fingerprint, consensusWeight); totalConsensusWeight += consensusWeight; if (consensusContainsBandwidthWeights) { - double guardWeight = consensusWeight, - middleWeight = consensusWeight, - exitWeight = consensusWeight; + double guardWeight = consensusWeight; + double middleWeight = consensusWeight; + double exitWeight = consensusWeight; if (isGuard && isExit) { guardWeight *= wgd; middleWeight *= wmd; @@ -676,8 +700,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, new TreeMap<String, SortedSet<String>>(); for (String fingerprint : this.currentRelays) { NodeStatus nodeStatus = this.knownNodes.get(fingerprint); - if (nodeStatus != null && nodeStatus.getDeclaredFamily() != null && - !nodeStatus.getDeclaredFamily().isEmpty()) { + if (nodeStatus != null && nodeStatus.getDeclaredFamily() != null + && !nodeStatus.getDeclaredFamily().isEmpty()) { declaredFamilies.put(fingerprint, nodeStatus.getDeclaredFamily()); } } @@ -689,8 +713,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, SortedSet<String> declaredFamily = e.getValue(); SortedSet<String> effectiveFamily = new TreeSet<String>(); for (String declaredFamilyMember : declaredFamily) { - if (declaredFamilies.containsKey(declaredFamilyMember) && - declaredFamilies.get(declaredFamilyMember).contains( + if (declaredFamilies.containsKey(declaredFamilyMember) + && declaredFamilies.get(declaredFamilyMember).contains( fingerprint)) { effectiveFamily.add(declaredFamilyMember); } @@ -737,15 +761,15 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, if (nodeStatus == null) { continue; } - if (effectiveFamilies.containsKey(fingerprint) || - extendedFamilies.containsKey(fingerprint)) { + if (effectiveFamilies.containsKey(fingerprint) + || extendedFamilies.containsKey(fingerprint)) { nodeStatus.setEffectiveFamily(effectiveFamilies.get(fingerprint)); nodeStatus.setExtendedFamily(extendedFamilies.get(fingerprint)); this.updatedNodes.add(fingerprint); - } else if ((nodeStatus.getEffectiveFamily() != null && - !nodeStatus.getEffectiveFamily().isEmpty()) || - (nodeStatus.getIndirectFamily() != null && - !nodeStatus.getIndirectFamily().isEmpty())) { + } else if ((nodeStatus.getEffectiveFamily() != null + && !nodeStatus.getEffectiveFamily().isEmpty()) + || (nodeStatus.getIndirectFamily() != null + && !nodeStatus.getIndirectFamily().isEmpty())) { nodeStatus.setEffectiveFamily(null); nodeStatus.setExtendedFamily(null); this.updatedNodes.add(fingerprint); @@ -804,8 +828,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, this.exitListEntries.get(fingerprint).entrySet()) { String exitAddress = e.getKey(); long scanMillis = e.getValue(); - if (!exitAddresses.containsKey(exitAddress) || - exitAddresses.get(exitAddress) < scanMillis) { + if (!exitAddresses.containsKey(exitAddress) + || exitAddresses.get(exitAddress) < scanMillis) { exitAddresses.put(exitAddress, scanMillis); } } @@ -859,12 +883,12 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, nodeStatus.setLastRdnsLookup(this.startedRdnsLookups); } - if (detailsStatus.getLastSeenMillis() < - nodeStatus.getLastSeenMillis()) { + if (detailsStatus.getLastSeenMillis() + < nodeStatus.getLastSeenMillis()) { if (this.lastSeenMeasured.containsKey(fingerprint)) { - if (this.lastSeenUnmeasured.containsKey(fingerprint) && - this.lastSeenUnmeasured.get(fingerprint) > - this.lastSeenMeasured.get(fingerprint)) { + if (this.lastSeenUnmeasured.containsKey(fingerprint) + && this.lastSeenUnmeasured.get(fingerprint) + > this.lastSeenMeasured.get(fingerprint)) { detailsStatus.setMeasured(false); } else { detailsStatus.setMeasured(true); @@ -875,9 +899,8 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, } detailsStatus.setRelay(nodeStatus.isRelay()); - detailsStatus.setRunning(nodeStatus.getLastSeenMillis() == - (nodeStatus.isRelay() - ? this.relaysLastValidAfterMillis + detailsStatus.setRunning(nodeStatus.getLastSeenMillis() + == (nodeStatus.isRelay() ? this.relaysLastValidAfterMillis : this.bridgesLastPublishedMillis)); detailsStatus.setNickname(nodeStatus.getNickname()); detailsStatus.setAddress(nodeStatus.getAddress()); diff --git a/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java b/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java index 4a06d20..39aba75 100644 --- a/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java +++ b/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java @@ -1,5 +1,6 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; import java.net.InetAddress; @@ -8,9 +9,17 @@ import java.net.UnknownHostException; class RdnsLookupRequest extends Thread { private final ReverseDomainNameResolver reverseDomainNameResolver; + private RdnsLookupWorker parent; - private String address, hostName; - private long lookupStartedMillis = -1L, lookupCompletedMillis = -1L; + + private String address; + + private String hostName; + + private long lookupStartedMillis = -1L; + + private long lookupCompletedMillis = -1L; + public RdnsLookupRequest( ReverseDomainNameResolver reverseDomainNameResolver, RdnsLookupWorker parent, String address) { @@ -18,6 +27,7 @@ class RdnsLookupRequest extends Thread { this.parent = parent; this.address = address; } + public void run() { this.lookupStartedMillis = this.reverseDomainNameResolver.time.currentTimeMillis(); @@ -33,9 +43,11 @@ class RdnsLookupRequest extends Thread { this.reverseDomainNameResolver.time.currentTimeMillis(); this.parent.interrupt(); } + public synchronized String getHostName() { return hostName; } + public synchronized long getLookupMillis() { return this.lookupCompletedMillis - this.lookupStartedMillis; } diff --git a/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java b/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java index cf7d580..c9af126 100644 --- a/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java +++ b/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java @@ -1,5 +1,6 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; class RdnsLookupWorker extends Thread { @@ -11,8 +12,8 @@ class RdnsLookupWorker extends Thread { } public void run() { - while (this.reverseDomainNameResolver.time.currentTimeMillis() - - ReverseDomainNameResolver.RDNS_LOOKUP_MAX_DURATION_MILLIS + while (this.reverseDomainNameResolver.time.currentTimeMillis() + - ReverseDomainNameResolver.RDNS_LOOKUP_MAX_DURATION_MILLIS <= this.reverseDomainNameResolver.startedRdnsLookups) { String rdnsLookupJob = null; synchronized (this.reverseDomainNameResolver.rdnsLookupJobs) { diff --git a/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java b/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java index 681a681..293ee82 100644 --- a/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java +++ b/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java @@ -1,7 +1,12 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; +import org.torproject.onionoo.util.FormattingUtils; +import org.torproject.onionoo.util.Time; +import org.torproject.onionoo.util.TimeFactory; + import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; @@ -10,10 +15,6 @@ import java.util.List; import java.util.Map; import java.util.Set; -import org.torproject.onionoo.util.FormattingUtils; -import org.torproject.onionoo.util.Time; -import org.torproject.onionoo.util.TimeFactory; - public class ReverseDomainNameResolver { Time time; @@ -23,9 +24,12 @@ public class ReverseDomainNameResolver { } static final long RDNS_LOOKUP_MAX_REQUEST_MILLIS = 10L * 1000L; + static final long RDNS_LOOKUP_MAX_DURATION_MILLIS = 5L * 60L * 1000L; + private static final long RDNS_LOOKUP_MAX_AGE_MILLIS = 12L * 60L * 60L * 1000L; + private static final int RDNS_LOOKUP_WORKERS_NUM = 5; private Map<String, Long> addressLastLookupTimes; diff --git a/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java b/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java index 50604bf..c33421b 100644 --- a/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java +++ b/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java @@ -1,12 +1,13 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.updater; -import java.io.File; +package org.torproject.onionoo.updater; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.File; + public class StatusUpdateRunner { private static final Logger log = LoggerFactory.getLogger( @@ -48,9 +49,10 @@ public class StatusUpdateRunner { su.getClass().getSimpleName() + "\n" + statsString); } } - LoggerFactory.getLogger("statistics").info("GeoIP lookup service\n" + - this.ls.getStatsString()); + LoggerFactory.getLogger("statistics").info("GeoIP lookup service\n" + + this.ls.getStatsString()); LoggerFactory.getLogger("statistics").info("Reverse domain name " + "resolver\n" + this.rdnr.getStatsString()); } } + diff --git a/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java index 9fc34d3..258d92f 100644 --- a/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; public interface StatusUpdater { diff --git a/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java index e2cee78..3edd20e 100644 --- a/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java @@ -1,14 +1,7 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.updater; -import java.util.BitSet; -import java.util.HashMap; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeMap; -import java.util.TreeSet; +package org.torproject.onionoo.updater; import org.torproject.descriptor.BridgeNetworkStatus; import org.torproject.descriptor.Descriptor; @@ -20,6 +13,14 @@ import org.torproject.onionoo.docs.DocumentStoreFactory; import org.torproject.onionoo.docs.UptimeStatus; import org.torproject.onionoo.util.FormattingUtils; +import java.util.BitSet; +import java.util.HashMap; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; + public class UptimeStatusUpdater implements DescriptorListener, StatusUpdater { @@ -85,9 +86,12 @@ public class UptimeStatusUpdater implements DescriptorListener, private SortedMap<Long, Flags> newRelayStatuses = new TreeMap<Long, Flags>(); + private SortedMap<String, SortedMap<Long, Flags>> newRunningRelays = new TreeMap<String, SortedMap<Long, Flags>>(); + private SortedSet<Long> newBridgeStatuses = new TreeSet<Long>(); + private SortedMap<String, SortedSet<Long>> newRunningBridges = new TreeMap<String, SortedSet<Long>>(); @@ -153,10 +157,10 @@ public class UptimeStatusUpdater implements DescriptorListener, private void updateStatus(boolean relay, String fingerprint, SortedMap<Long, Flags> dateHourMillisFlags) { - UptimeStatus uptimeStatus = (fingerprint == null) ? - this.documentStore.retrieve(UptimeStatus.class, true) : - this.documentStore.retrieve(UptimeStatus.class, true, - fingerprint); + UptimeStatus uptimeStatus = (fingerprint == null) + ? this.documentStore.retrieve(UptimeStatus.class, true) + : this.documentStore.retrieve(UptimeStatus.class, true, + fingerprint); if (uptimeStatus == null) { uptimeStatus = new UptimeStatus(); } diff --git a/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java index 8b6f1a2..ea7faed 100644 --- a/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java @@ -1,13 +1,7 @@ /* Copyright 2012--2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.updater; -import java.util.Arrays; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeMap; -import java.util.TreeSet; +package org.torproject.onionoo.updater; import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.NetworkStatusEntry; @@ -16,6 +10,13 @@ import org.torproject.onionoo.docs.DocumentStore; import org.torproject.onionoo.docs.DocumentStoreFactory; import org.torproject.onionoo.docs.WeightsStatus; +import java.util.Arrays; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; + public class WeightsStatusUpdater implements DescriptorListener, StatusUpdater { @@ -80,8 +81,14 @@ public class WeightsStatusUpdater implements DescriptorListener, private SortedMap<String, double[]> calculatePathSelectionProbabilities( RelayNetworkStatusConsensus consensus) { boolean containsBandwidthWeights = false; - double wgg = 1.0, wgd = 1.0, wmg = 1.0, wmm = 1.0, wme = 1.0, - wmd = 1.0, wee = 1.0, wed = 1.0; + double wgg = 1.0; + double wgd = 1.0; + double wmg = 1.0; + double wmm = 1.0; + double wme = 1.0; + double wmd = 1.0; + double wee = 1.0; + double wed = 1.0; SortedMap<String, Integer> bandwidthWeights = consensus.getBandwidthWeights(); if (bandwidthWeights != null) { @@ -100,11 +107,14 @@ public class WeightsStatusUpdater implements DescriptorListener, containsBandwidthWeights = true; } } - SortedMap<String, Double> - consensusWeights = new TreeMap<String, Double>(), - guardWeights = new TreeMap<String, Double>(), - middleWeights = new TreeMap<String, Double>(), - exitWeights = new TreeMap<String, Double>(); + SortedMap<String, Double> consensusWeights = + new TreeMap<String, Double>(); + SortedMap<String, Double> guardWeights = + new TreeMap<String, Double>(); + SortedMap<String, Double> middleWeights = + new TreeMap<String, Double>(); + SortedMap<String, Double> exitWeights = + new TreeMap<String, Double>(); double totalConsensusWeight = 0.0; double totalGuardWeight = 0.0; double totalMiddleWeight = 0.0; @@ -123,8 +133,8 @@ public class WeightsStatusUpdater implements DescriptorListener, double guardWeight = (double) relay.getBandwidth(); double middleWeight = (double) relay.getBandwidth(); double exitWeight = (double) relay.getBandwidth(); - boolean isExit = relay.getFlags().contains("Exit") && - !relay.getFlags().contains("BadExit"); + boolean isExit = relay.getFlags().contains("Exit") + && !relay.getFlags().contains("BadExit"); boolean isGuard = relay.getFlags().contains("Guard"); if (isGuard && isExit) { guardWeight *= wgd; @@ -157,26 +167,26 @@ public class WeightsStatusUpdater implements DescriptorListener, for (String fingerprint : consensusWeights.keySet()) { double[] probabilities = new double[] { -1.0, -1.0, -1.0, -1.0, -1.0, -1.0, -1.0 }; - if (consensusWeights.containsKey(fingerprint) && - totalConsensusWeight > 0.0) { - probabilities[1] = consensusWeights.get(fingerprint) / - totalConsensusWeight; + if (consensusWeights.containsKey(fingerprint) + && totalConsensusWeight > 0.0) { + probabilities[1] = consensusWeights.get(fingerprint) + / totalConsensusWeight; probabilities[6] = consensusWeights.get(fingerprint); } - if (guardWeights.containsKey(fingerprint) && - totalGuardWeight > 0.0) { - probabilities[2] = guardWeights.get(fingerprint) / - totalGuardWeight; + if (guardWeights.containsKey(fingerprint) + && totalGuardWeight > 0.0) { + probabilities[2] = guardWeights.get(fingerprint) + / totalGuardWeight; } - if (middleWeights.containsKey(fingerprint) && - totalMiddleWeight > 0.0) { - probabilities[3] = middleWeights.get(fingerprint) / - totalMiddleWeight; + if (middleWeights.containsKey(fingerprint) + && totalMiddleWeight > 0.0) { + probabilities[3] = middleWeights.get(fingerprint) + / totalMiddleWeight; } - if (exitWeights.containsKey(fingerprint) && - totalExitWeight > 0.0) { - probabilities[4] = exitWeights.get(fingerprint) / - totalExitWeight; + if (exitWeights.containsKey(fingerprint) + && totalExitWeight > 0.0) { + probabilities[4] = exitWeights.get(fingerprint) + / totalExitWeight; } pathSelectionProbabilities.put(fingerprint, probabilities); } diff --git a/src/main/java/org/torproject/onionoo/util/FormattingUtils.java b/src/main/java/org/torproject/onionoo/util/FormattingUtils.java index bda5a1d..615f67b 100644 --- a/src/main/java/org/torproject/onionoo/util/FormattingUtils.java +++ b/src/main/java/org/torproject/onionoo/util/FormattingUtils.java @@ -1,5 +1,6 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.util; public class FormattingUtils { @@ -7,8 +8,9 @@ public class FormattingUtils { private FormattingUtils() { } - private static final long ONE_SECOND = 1000L, - ONE_MINUTE = 60L * ONE_SECOND; + private static final long ONE_SECOND = 1000L; + + private static final long ONE_MINUTE = 60L * ONE_SECOND; public static String formatMillis(long millis) { return String.format("%02d:%02d.%03d minutes", millis / ONE_MINUTE, @@ -29,3 +31,4 @@ public class FormattingUtils { return String.format("%,d", decimalNumber); } } + diff --git a/src/main/java/org/torproject/onionoo/util/LockFile.java b/src/main/java/org/torproject/onionoo/util/LockFile.java index fc8a792..f7aeee8 100644 --- a/src/main/java/org/torproject/onionoo/util/LockFile.java +++ b/src/main/java/org/torproject/onionoo/util/LockFile.java @@ -1,18 +1,19 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.util; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - public class LockFile { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( LockFile.class); private final File lockFile = new File("lock"); @@ -22,7 +23,7 @@ public class LockFile { * milliseconds and return whether this operation was successful. * * @return <code>true</code> if the lock file did not exist and writing - * that file now succeeded, <code>false</code> otherwise. + * that file now succeeded, <code>false</code> otherwise. */ public boolean acquireLock() { Time time = TimeFactory.getTime(); @@ -54,7 +55,7 @@ public class LockFile { * whether the file was successfully deleted. * * @return <code>true</code> if the lock file does not exist anymore - * when returning, <code>false</code> otherwise. + * when returning, <code>false</code> otherwise. */ public boolean releaseLock() { if (this.lockFile.exists()) { diff --git a/src/main/java/org/torproject/onionoo/util/Time.java b/src/main/java/org/torproject/onionoo/util/Time.java index 126a910..4f6d0bb 100644 --- a/src/main/java/org/torproject/onionoo/util/Time.java +++ b/src/main/java/org/torproject/onionoo/util/Time.java @@ -1,5 +1,6 @@ /* Copyright 2013 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.util; /* @@ -7,6 +8,7 @@ package org.torproject.onionoo.util; * custom time source for testing. */ public class Time { + public long currentTimeMillis() { return System.currentTimeMillis(); } diff --git a/src/main/java/org/torproject/onionoo/util/TimeFactory.java b/src/main/java/org/torproject/onionoo/util/TimeFactory.java index b53b7a3..d52a584 100644 --- a/src/main/java/org/torproject/onionoo/util/TimeFactory.java +++ b/src/main/java/org/torproject/onionoo/util/TimeFactory.java @@ -1,13 +1,16 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.util; public class TimeFactory { private static Time timeInstance; + public static void setTime(Time time) { timeInstance = time; } + public static Time getTime() { if (timeInstance == null) { timeInstance = new Time(); diff --git a/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java index 551fc71..3762065 100644 --- a/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java @@ -1,16 +1,8 @@ /* Copyright 2011--2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.writer; -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; +package org.torproject.onionoo.writer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.onionoo.docs.BandwidthDocument; import org.torproject.onionoo.docs.BandwidthStatus; import org.torproject.onionoo.docs.DateTimeHelper; @@ -20,6 +12,16 @@ import org.torproject.onionoo.docs.GraphHistory; import org.torproject.onionoo.docs.UpdateStatus; import org.torproject.onionoo.util.TimeFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; + public class BandwidthDocumentWriter implements DocumentWriter { private static final Logger log = LoggerFactory.getLogger( @@ -37,8 +39,8 @@ public class BandwidthDocumentWriter implements DocumentWriter { public void writeDocuments() { UpdateStatus updateStatus = this.documentStore.retrieve( UpdateStatus.class, true); - long updatedMillis = updateStatus != null ? - updateStatus.getUpdatedMillis() : 0L; + long updatedMillis = updateStatus != null + ? updateStatus.getUpdatedMillis() : 0L; SortedSet<String> updateBandwidthDocuments = this.documentStore.list( BandwidthStatus.class, updatedMillis); for (String fingerprint : updateBandwidthDocuments) { @@ -101,9 +103,12 @@ public class BandwidthDocumentWriter implements DocumentWriter { List<Long> dataPoints = new ArrayList<Long>(); long intervalStartMillis = ((this.now - graphInterval) / dataPointInterval) * dataPointInterval; - long totalMillis = 0L, totalBandwidth = 0L; + long totalMillis = 0L; + long totalBandwidth = 0L; for (long[] v : history.values()) { - long startMillis = v[0], endMillis = v[1], bandwidth = v[2]; + long startMillis = v[0]; + long endMillis = v[1]; + long bandwidth = v[2]; if (endMillis < intervalStartMillis) { continue; } @@ -113,8 +118,8 @@ public class BandwidthDocumentWriter implements DocumentWriter { * one. */ continue; } - while ((intervalStartMillis / dataPointInterval) != - (endMillis / dataPointInterval)) { + while ((intervalStartMillis / dataPointInterval) + != (endMillis / dataPointInterval)) { dataPoints.add(totalMillis * 5L < dataPointInterval ? -1L : (totalBandwidth * DateTimeHelper.ONE_SECOND) / totalMillis); @@ -129,7 +134,8 @@ public class BandwidthDocumentWriter implements DocumentWriter { ? -1L : (totalBandwidth * DateTimeHelper.ONE_SECOND) / totalMillis); long maxValue = 1L; - int firstNonNullIndex = -1, lastNonNullIndex = -1; + int firstNonNullIndex = -1; + int lastNonNullIndex = -1; for (int j = 0; j < dataPoints.size(); j++) { long dataPoint = dataPoints.get(j); if (dataPoint >= 0L) { @@ -148,8 +154,8 @@ public class BandwidthDocumentWriter implements DocumentWriter { long firstDataPointMillis = (((this.now - graphInterval) / dataPointInterval) + firstNonNullIndex) * dataPointInterval + dataPointInterval / 2L; - if (i > 0 && !graphs.isEmpty() && - firstDataPointMillis >= this.now - graphIntervals[i - 1]) { + if (i > 0 && !graphs.isEmpty() + && firstDataPointMillis >= this.now - graphIntervals[i - 1]) { /* Skip bandwidth history object, because it doesn't contain * anything new that wasn't already contained in the last * bandwidth history object(s). Unless we did not include any of @@ -179,8 +185,8 @@ public class BandwidthDocumentWriter implements DocumentWriter { } previousNonNullIndex = j; } - values.add(dataPoint < 0L ? null : - (int) ((dataPoint * 999L) / maxValue)); + values.add(dataPoint < 0L ? null + : (int) ((dataPoint * 999L) / maxValue)); } graphHistory.setValues(values); if (foundTwoAdjacentDataPoints) { @@ -195,3 +201,4 @@ public class BandwidthDocumentWriter implements DocumentWriter { return null; } } + diff --git a/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java index 84d5a30..37d486e 100644 --- a/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java @@ -1,17 +1,8 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.writer; -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeMap; +package org.torproject.onionoo.writer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.onionoo.docs.ClientsDocument; import org.torproject.onionoo.docs.ClientsGraphHistory; import org.torproject.onionoo.docs.ClientsHistory; @@ -23,6 +14,17 @@ import org.torproject.onionoo.docs.UpdateStatus; import org.torproject.onionoo.util.FormattingUtils; import org.torproject.onionoo.util.TimeFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; + /* * Clients status file produced as intermediate output: * @@ -48,7 +50,7 @@ import org.torproject.onionoo.util.TimeFactory; */ public class ClientsDocumentWriter implements DocumentWriter { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( ClientsDocumentWriter.class); private DocumentStore documentStore; @@ -65,8 +67,8 @@ public class ClientsDocumentWriter implements DocumentWriter { public void writeDocuments() { UpdateStatus updateStatus = this.documentStore.retrieve( UpdateStatus.class, true); - long updatedMillis = updateStatus != null ? - updateStatus.getUpdatedMillis() : 0L; + long updatedMillis = updateStatus != null + ? updateStatus.getUpdatedMillis() : 0L; SortedSet<String> updateDocuments = this.documentStore.list( ClientsStatus.class, updatedMillis); for (String hashedFingerprint : updateDocuments) { @@ -111,8 +113,8 @@ public class ClientsDocumentWriter implements DocumentWriter { clientsDocument.setFingerprint(hashedFingerprint); Map<String, ClientsGraphHistory> averageClients = new LinkedHashMap<String, ClientsGraphHistory>(); - for (int graphIntervalIndex = 0; graphIntervalIndex < - this.graphIntervals.length; graphIntervalIndex++) { + for (int graphIntervalIndex = 0; graphIntervalIndex + < this.graphIntervals.length; graphIntervalIndex++) { String graphName = this.graphNames[graphIntervalIndex]; ClientsGraphHistory graphHistory = this.compileClientsHistory( graphIntervalIndex, history); @@ -133,17 +135,20 @@ public class ClientsDocumentWriter implements DocumentWriter { long intervalStartMillis = ((this.now - graphInterval) / dataPointInterval) * dataPointInterval; long millis = 0L; - double responses = 0.0, totalResponses = 0.0; - SortedMap<String, Double> - totalResponsesByCountry = new TreeMap<String, Double>(), - totalResponsesByTransport = new TreeMap<String, Double>(), - totalResponsesByVersion = new TreeMap<String, Double>(); + double responses = 0.0; + double totalResponses = 0.0; + SortedMap<String, Double> totalResponsesByCountry = + new TreeMap<String, Double>(); + SortedMap<String, Double> totalResponsesByTransport = + new TreeMap<String, Double>(); + SortedMap<String, Double> totalResponsesByVersion = + new TreeMap<String, Double>(); for (ClientsHistory hist : history) { if (hist.getEndMillis() < intervalStartMillis) { continue; } - while ((intervalStartMillis / dataPointInterval) != - (hist.getEndMillis() / dataPointInterval)) { + while ((intervalStartMillis / dataPointInterval) + != (hist.getEndMillis() / dataPointInterval)) { dataPoints.add(millis * 2L < dataPointInterval ? -1.0 : responses * ((double) DateTimeHelper.ONE_DAY) / (((double) millis) * 10.0)); @@ -183,7 +188,8 @@ public class ClientsDocumentWriter implements DocumentWriter { ? -1.0 : responses * ((double) DateTimeHelper.ONE_DAY) / (((double) millis) * 10.0)); double maxValue = 0.0; - int firstNonNullIndex = -1, lastNonNullIndex = -1; + int firstNonNullIndex = -1; + int lastNonNullIndex = -1; for (int dataPointIndex = 0; dataPointIndex < dataPoints.size(); dataPointIndex++) { double dataPoint = dataPoints.get(dataPointIndex); @@ -204,8 +210,8 @@ public class ClientsDocumentWriter implements DocumentWriter { long firstDataPointMillis = (((this.now - graphInterval) / dataPointInterval) + firstNonNullIndex) * dataPointInterval + dataPointInterval / 2L; - if (graphIntervalIndex > 0 && firstDataPointMillis >= - this.now - graphIntervals[graphIntervalIndex - 1]) { + if (graphIntervalIndex > 0 && firstDataPointMillis + >= this.now - graphIntervals[graphIntervalIndex - 1]) { /* Skip clients history object, because it doesn't contain * anything new that wasn't already contained in the last * clients history object(s). */ @@ -225,8 +231,8 @@ public class ClientsDocumentWriter implements DocumentWriter { int previousNonNullIndex = -2; boolean foundTwoAdjacentDataPoints = false; List<Integer> values = new ArrayList<Integer>(); - for (int dataPointIndex = firstNonNullIndex; dataPointIndex <= - lastNonNullIndex; dataPointIndex++) { + for (int dataPointIndex = firstNonNullIndex; dataPointIndex + <= lastNonNullIndex; dataPointIndex++) { double dataPoint = dataPoints.get(dataPointIndex); if (dataPoint >= 0.0) { if (dataPointIndex - previousNonNullIndex == 1) { @@ -287,3 +293,4 @@ public class ClientsDocumentWriter implements DocumentWriter { return sb.toString(); } } + diff --git a/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java index 8360a05..506067d 100644 --- a/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java @@ -1,5 +1,14 @@ package org.torproject.onionoo.writer; +import org.torproject.onionoo.docs.DetailsDocument; +import org.torproject.onionoo.docs.DetailsStatus; +import org.torproject.onionoo.docs.DocumentStore; +import org.torproject.onionoo.docs.DocumentStoreFactory; +import org.torproject.onionoo.docs.UpdateStatus; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; @@ -8,17 +17,9 @@ import java.util.Map; import java.util.SortedSet; import java.util.TreeSet; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.torproject.onionoo.docs.DetailsDocument; -import org.torproject.onionoo.docs.DetailsStatus; -import org.torproject.onionoo.docs.DocumentStore; -import org.torproject.onionoo.docs.DocumentStoreFactory; -import org.torproject.onionoo.docs.UpdateStatus; - public class DetailsDocumentWriter implements DocumentWriter { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( DetailsDocumentWriter.class); private DocumentStore documentStore; @@ -30,8 +31,8 @@ public class DetailsDocumentWriter implements DocumentWriter { public void writeDocuments() { UpdateStatus updateStatus = this.documentStore.retrieve( UpdateStatus.class, true); - long updatedMillis = updateStatus != null ? - updateStatus.getUpdatedMillis() : 0L; + long updatedMillis = updateStatus != null + ? updateStatus.getUpdatedMillis() : 0L; SortedSet<String> updatedDetailsStatuses = this.documentStore.list( DetailsStatus.class, updatedMillis); for (String fingerprint : updatedDetailsStatuses) { @@ -73,8 +74,8 @@ public class DetailsDocumentWriter implements DocumentWriter { detailsDocument.setHostName(detailsStatus.getHostName()); String defaultPolicy = detailsStatus.getDefaultPolicy(); String portList = detailsStatus.getPortList(); - if (defaultPolicy != null && (defaultPolicy.equals("accept") || - defaultPolicy.equals("reject")) && portList != null) { + if (defaultPolicy != null && (defaultPolicy.equals("accept") + || defaultPolicy.equals("reject")) && portList != null) { Map<String, List<String>> exitPolicySummary = new HashMap<String, List<String>>(); List<String> portsOrPortRanges = Arrays.asList(portList.split(",")); @@ -111,8 +112,8 @@ public class DetailsDocumentWriter implements DocumentWriter { detailsDocument.setExitPolicy(detailsStatus.getExitPolicy()); detailsDocument.setContact(detailsStatus.getContact()); detailsDocument.setPlatform(detailsStatus.getPlatform()); - if (detailsStatus.getAllegedFamily() != null && - !detailsStatus.getAllegedFamily().isEmpty()) { + if (detailsStatus.getAllegedFamily() != null + && !detailsStatus.getAllegedFamily().isEmpty()) { SortedSet<String> allegedFamily = new TreeSet<String>(); for (String familyMember : detailsStatus.getAllegedFamily()) { if (familyMember.length() >= 40) { @@ -123,16 +124,16 @@ public class DetailsDocumentWriter implements DocumentWriter { } detailsDocument.setAllegedFamily(allegedFamily); } - if (detailsStatus.getEffectiveFamily() != null && - !detailsStatus.getEffectiveFamily().isEmpty()) { + if (detailsStatus.getEffectiveFamily() != null + && !detailsStatus.getEffectiveFamily().isEmpty()) { SortedSet<String> effectiveFamily = new TreeSet<String>(); for (String familyMember : detailsStatus.getEffectiveFamily()) { effectiveFamily.add("$" + familyMember); } detailsDocument.setEffectiveFamily(effectiveFamily); } - if (detailsStatus.getIndirectFamily() != null && - !detailsStatus.getIndirectFamily().isEmpty()) { + if (detailsStatus.getIndirectFamily() != null + && !detailsStatus.getIndirectFamily().isEmpty()) { SortedSet<String> indirectFamily = new TreeSet<String>(); for (String familyMember : detailsStatus.getIndirectFamily()) { indirectFamily.add("$" + familyMember); @@ -187,3 +188,4 @@ public class DetailsDocumentWriter implements DocumentWriter { return null; } } + diff --git a/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java index c238170..fa3c264 100644 --- a/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.writer; public interface DocumentWriter { diff --git a/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java b/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java index 07bdf3c..fae481f 100644 --- a/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java +++ b/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java @@ -1,5 +1,6 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ + package org.torproject.onionoo.writer; import org.slf4j.Logger; @@ -7,7 +8,7 @@ import org.slf4j.LoggerFactory; public class DocumentWriterRunner { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( DocumentWriterRunner.class); private DocumentWriter[] documentWriters; @@ -34,9 +35,10 @@ public class DocumentWriterRunner { for (DocumentWriter dw : this.documentWriters) { String statsString = dw.getStatsString(); if (statsString != null) { - log.info(dw.getClass().getSimpleName() + "\n" + - statsString); + log.info(dw.getClass().getSimpleName() + "\n" + + statsString); } } } } + diff --git a/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java index 1be752a..89fb166 100644 --- a/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java @@ -1,13 +1,8 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.writer; -import java.util.ArrayList; -import java.util.List; -import java.util.SortedSet; +package org.torproject.onionoo.writer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.onionoo.docs.DateTimeHelper; import org.torproject.onionoo.docs.DocumentStore; import org.torproject.onionoo.docs.DocumentStoreFactory; @@ -15,9 +10,16 @@ import org.torproject.onionoo.docs.NodeStatus; import org.torproject.onionoo.docs.SummaryDocument; import org.torproject.onionoo.util.FormattingUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.List; +import java.util.SortedSet; + public class SummaryDocumentWriter implements DocumentWriter { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( SummaryDocumentWriter.class); private DocumentStore documentStore; @@ -26,11 +28,13 @@ public class SummaryDocumentWriter implements DocumentWriter { this.documentStore = DocumentStoreFactory.getDocumentStore(); } - private int writtenDocuments = 0, deletedDocuments = 0; + private int writtenDocuments = 0; + + private int deletedDocuments = 0; public void writeDocuments() { - long relaysLastValidAfterMillis = -1L, - bridgesLastPublishedMillis = -1L; + long relaysLastValidAfterMillis = -1L; + long bridgesLastPublishedMillis = -1L; for (String fingerprint : this.documentStore.list(NodeStatus.class)) { NodeStatus nodeStatus = this.documentStore.retrieve( NodeStatus.class, true, fingerprint); @@ -76,9 +80,9 @@ public class SummaryDocumentWriter implements DocumentWriter { } long lastSeenMillis = nodeStatus.getLastSeenMillis(); SortedSet<String> relayFlags = nodeStatus.getRelayFlags(); - boolean running = relayFlags.contains("Running") && (isRelay ? - lastSeenMillis == relaysLastValidAfterMillis : - lastSeenMillis == bridgesLastPublishedMillis); + boolean running = relayFlags.contains("Running") && (isRelay + ? lastSeenMillis == relaysLastValidAfterMillis + : lastSeenMillis == bridgesLastPublishedMillis); long consensusWeight = nodeStatus.getConsensusWeight(); String countryCode = nodeStatus.getCountryCode(); long firstSeenMillis = nodeStatus.getFirstSeenMillis(); @@ -92,7 +96,7 @@ public class SummaryDocumentWriter implements DocumentWriter { aSNumber, contact, declaredFamily, effectiveFamily); if (this.documentStore.store(summaryDocument, fingerprint)) { this.writtenDocuments++; - }; + } } log.info("Wrote summary document files"); } @@ -106,3 +110,4 @@ public class SummaryDocumentWriter implements DocumentWriter { return sb.toString(); } } + diff --git a/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java index d9eb91f..2a48ccb 100644 --- a/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java @@ -1,18 +1,8 @@ /* Copyright 2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.writer; -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeMap; -import java.util.TreeSet; +package org.torproject.onionoo.writer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.onionoo.docs.DateTimeHelper; import org.torproject.onionoo.docs.DocumentStore; import org.torproject.onionoo.docs.DocumentStoreFactory; @@ -24,9 +14,21 @@ import org.torproject.onionoo.docs.UptimeStatus; import org.torproject.onionoo.util.FormattingUtils; import org.torproject.onionoo.util.TimeFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; +import java.util.TreeMap; +import java.util.TreeSet; + public class UptimeDocumentWriter implements DocumentWriter { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( UptimeDocumentWriter.class); private DocumentStore documentStore; @@ -47,8 +49,8 @@ public class UptimeDocumentWriter implements DocumentWriter { } UpdateStatus updateStatus = this.documentStore.retrieve( UpdateStatus.class, true); - long updatedMillis = updateStatus != null ? - updateStatus.getUpdatedMillis() : 0L; + long updatedMillis = updateStatus != null + ? updateStatus.getUpdatedMillis() : 0L; SortedSet<String> updatedUptimeStatuses = this.documentStore.list( UptimeStatus.class, updatedMillis); for (String fingerprint : updatedUptimeStatuses) { @@ -106,8 +108,8 @@ public class UptimeDocumentWriter implements DocumentWriter { uptimeDocument.setFingerprint(fingerprint); Map<String, GraphHistory> uptime = new LinkedHashMap<String, GraphHistory>(); - for (int graphIntervalIndex = 0; graphIntervalIndex < - this.graphIntervals.length; graphIntervalIndex++) { + for (int graphIntervalIndex = 0; graphIntervalIndex + < this.graphIntervals.length; graphIntervalIndex++) { String graphName = this.graphNames[graphIntervalIndex]; GraphHistory graphHistory = this.compileUptimeHistory( graphIntervalIndex, relay, history, knownStatuses, null); @@ -127,8 +129,8 @@ public class UptimeDocumentWriter implements DocumentWriter { for (String flag : allFlags) { Map<String, GraphHistory> graphsForFlags = new LinkedHashMap<String, GraphHistory>(); - for (int graphIntervalIndex = 0; graphIntervalIndex < - this.graphIntervals.length; graphIntervalIndex++) { + for (int graphIntervalIndex = 0; graphIntervalIndex + < this.graphIntervals.length; graphIntervalIndex++) { String graphName = this.graphNames[graphIntervalIndex]; GraphHistory graphHistory = this.compileUptimeHistory( graphIntervalIndex, relay, history, knownStatuses, flag); @@ -160,9 +162,9 @@ public class UptimeDocumentWriter implements DocumentWriter { int uptimeHours = 0; long firstStatusStartMillis = -1L; for (UptimeHistory hist : history) { - if (hist.isRelay() != relay || - (flag != null && (hist.getFlags() == null || - !hist.getFlags().contains(flag)))) { + if (hist.isRelay() != relay + || (flag != null && (hist.getFlags() == null + || !hist.getFlags().contains(flag)))) { continue; } if (firstStatusStartMillis < 0L) { @@ -202,9 +204,9 @@ public class UptimeDocumentWriter implements DocumentWriter { / dataPointInterval) * dataPointInterval; int statusHours = -1; for (UptimeHistory hist : knownStatuses) { - if (hist.isRelay() != relay || - (flag != null && (hist.getFlags() == null || - !hist.getFlags().contains(flag)))) { + if (hist.isRelay() != relay + || (flag != null && (hist.getFlags() == null + || !hist.getFlags().contains(flag)))) { continue; } long histEndMillis = hist.getStartMillis() + DateTimeHelper.ONE_HOUR @@ -248,15 +250,16 @@ public class UptimeDocumentWriter implements DocumentWriter { dataPointIndex++) { if (dataPointIndex >= uptimeDataPoints.size()) { dataPoints.add(0.0); - } else if (uptimeDataPoints.get(dataPointIndex) >= 0 && - statusDataPoints.get(dataPointIndex) > 0) { + } else if (uptimeDataPoints.get(dataPointIndex) >= 0 + && statusDataPoints.get(dataPointIndex) > 0) { dataPoints.add(((double) uptimeDataPoints.get(dataPointIndex)) / ((double) statusDataPoints.get(dataPointIndex))); } else { dataPoints.add(-1.0); } } - int firstNonNullIndex = -1, lastNonNullIndex = -1; + int firstNonNullIndex = -1; + int lastNonNullIndex = -1; for (int dataPointIndex = 0; dataPointIndex < dataPoints.size(); dataPointIndex++) { double dataPoint = dataPoints.get(dataPointIndex); @@ -274,8 +277,8 @@ public class UptimeDocumentWriter implements DocumentWriter { long firstDataPointMillis = (((this.now - graphInterval) / dataPointInterval) + firstNonNullIndex) * dataPointInterval + dataPointInterval / 2L; - if (graphIntervalIndex > 0 && firstDataPointMillis >= - this.now - graphIntervals[graphIntervalIndex - 1]) { + if (graphIntervalIndex > 0 && firstDataPointMillis + >= this.now - graphIntervals[graphIntervalIndex - 1]) { /* Skip uptime history object, because it doesn't contain * anything new that wasn't already contained in the last * uptime history object(s). */ @@ -294,8 +297,8 @@ public class UptimeDocumentWriter implements DocumentWriter { int previousNonNullIndex = -2; boolean foundTwoAdjacentDataPoints = false; List<Integer> values = new ArrayList<Integer>(); - for (int dataPointIndex = firstNonNullIndex; dataPointIndex <= - lastNonNullIndex; dataPointIndex++) { + for (int dataPointIndex = firstNonNullIndex; dataPointIndex + <= lastNonNullIndex; dataPointIndex++) { double dataPoint = dataPoints.get(dataPointIndex); if (dataPoint >= 0.0) { if (dataPointIndex - previousNonNullIndex == 1) { diff --git a/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java index 62ff533..50001f3 100644 --- a/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java @@ -1,16 +1,8 @@ /* Copyright 2012--2014 The Tor Project * See LICENSE for licensing information */ -package org.torproject.onionoo.writer; -import java.util.ArrayList; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.Map; -import java.util.SortedMap; -import java.util.SortedSet; +package org.torproject.onionoo.writer; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; import org.torproject.onionoo.docs.DateTimeHelper; import org.torproject.onionoo.docs.DocumentStore; import org.torproject.onionoo.docs.DocumentStoreFactory; @@ -20,9 +12,19 @@ import org.torproject.onionoo.docs.WeightsDocument; import org.torproject.onionoo.docs.WeightsStatus; import org.torproject.onionoo.util.TimeFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.SortedMap; +import java.util.SortedSet; + public class WeightsDocumentWriter implements DocumentWriter { - private final static Logger log = LoggerFactory.getLogger( + private static final Logger log = LoggerFactory.getLogger( WeightsDocumentWriter.class); private DocumentStore documentStore; @@ -37,8 +39,8 @@ public class WeightsDocumentWriter implements DocumentWriter { public void writeDocuments() { UpdateStatus updateStatus = this.documentStore.retrieve( UpdateStatus.class, true); - long updatedMillis = updateStatus != null ? - updateStatus.getUpdatedMillis() : 0L; + long updatedMillis = updateStatus != null + ? updateStatus.getUpdatedMillis() : 0L; SortedSet<String> updateWeightsDocuments = this.documentStore.list( WeightsStatus.class, updatedMillis); for (String fingerprint : updateWeightsDocuments) { @@ -97,8 +99,8 @@ public class WeightsDocumentWriter implements DocumentWriter { SortedMap<long[], double[]> history, int graphTypeIndex) { Map<String, GraphHistory> graphs = new LinkedHashMap<String, GraphHistory>(); - for (int graphIntervalIndex = 0; graphIntervalIndex < - this.graphIntervals.length; graphIntervalIndex++) { + for (int graphIntervalIndex = 0; graphIntervalIndex + < this.graphIntervals.length; graphIntervalIndex++) { String graphName = this.graphNames[graphIntervalIndex]; GraphHistory graphHistory = this.compileWeightsHistory( graphTypeIndex, graphIntervalIndex, history); @@ -120,13 +122,14 @@ public class WeightsDocumentWriter implements DocumentWriter { long totalMillis = 0L; double totalWeightTimesMillis = 0.0; for (Map.Entry<long[], double[]> e : history.entrySet()) { - long startMillis = e.getKey()[0], endMillis = e.getKey()[1]; + long startMillis = e.getKey()[0]; + long endMillis = e.getKey()[1]; double weight = e.getValue()[graphTypeIndex]; if (endMillis < intervalStartMillis) { continue; } - while ((intervalStartMillis / dataPointInterval) != - (endMillis / dataPointInterval)) { + while ((intervalStartMillis / dataPointInterval) + != (endMillis / dataPointInterval)) { dataPoints.add(totalMillis * 5L < dataPointInterval ? -1.0 : totalWeightTimesMillis / (double) totalMillis); totalWeightTimesMillis = 0.0; @@ -142,7 +145,8 @@ public class WeightsDocumentWriter implements DocumentWriter { dataPoints.add(totalMillis * 5L < dataPointInterval ? -1.0 : totalWeightTimesMillis / (double) totalMillis); double maxValue = 0.0; - int firstNonNullIndex = -1, lastNonNullIndex = -1; + int firstNonNullIndex = -1; + int lastNonNullIndex = -1; for (int dataPointIndex = 0; dataPointIndex < dataPoints.size(); dataPointIndex++) { double dataPoint = dataPoints.get(dataPointIndex); @@ -163,8 +167,8 @@ public class WeightsDocumentWriter implements DocumentWriter { long firstDataPointMillis = (((this.now - graphInterval) / dataPointInterval) + firstNonNullIndex) * dataPointInterval + dataPointInterval / 2L; - if (graphIntervalIndex > 0 && firstDataPointMillis >= - this.now - graphIntervals[graphIntervalIndex - 1]) { + if (graphIntervalIndex > 0 && firstDataPointMillis + >= this.now - graphIntervals[graphIntervalIndex - 1]) { /* Skip weights history object, because it doesn't contain * anything new that wasn't already contained in the last * weights history object(s). */ @@ -184,8 +188,8 @@ public class WeightsDocumentWriter implements DocumentWriter { int previousNonNullIndex = -2; boolean foundTwoAdjacentDataPoints = false; List<Integer> values = new ArrayList<Integer>(); - for (int dataPointIndex = firstNonNullIndex; dataPointIndex <= - lastNonNullIndex; dataPointIndex++) { + for (int dataPointIndex = firstNonNullIndex; dataPointIndex + <= lastNonNullIndex; dataPointIndex++) { double dataPoint = dataPoints.get(dataPointIndex); if (dataPoint >= 0.0) { if (dataPointIndex - previousNonNullIndex == 1) { @@ -211,3 +215,4 @@ public class WeightsDocumentWriter implements DocumentWriter { return null; } } + diff --git a/src/test/java/org/torproject/onionoo/docs/BandwidthStatusTest.java b/src/test/java/org/torproject/onionoo/docs/BandwidthStatusTest.java index 96988d1..432f5d7 100644 --- a/src/test/java/org/torproject/onionoo/docs/BandwidthStatusTest.java +++ b/src/test/java/org/torproject/onionoo/docs/BandwidthStatusTest.java @@ -6,15 +6,16 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; -import java.util.SortedMap; -import java.util.TreeMap; - -import org.junit.Before; -import org.junit.Test; import org.torproject.descriptor.BandwidthHistory; import org.torproject.onionoo.util.DummyTime; import org.torproject.onionoo.util.TimeFactory; +import org.junit.Before; +import org.junit.Test; + +import java.util.SortedMap; +import java.util.TreeMap; + public class BandwidthStatusTest { private long currentTimeMillis = DateTimeHelper.parse( diff --git a/src/test/java/org/torproject/onionoo/docs/DummyDocumentStore.java b/src/test/java/org/torproject/onionoo/docs/DummyDocumentStore.java index aa987a1..34cd309 100644 --- a/src/test/java/org/torproject/onionoo/docs/DummyDocumentStore.java +++ b/src/test/java/org/torproject/onionoo/docs/DummyDocumentStore.java @@ -7,9 +7,6 @@ import java.util.SortedSet; import java.util.TreeMap; import java.util.TreeSet; -import org.torproject.onionoo.docs.Document; -import org.torproject.onionoo.docs.DocumentStore; - public class DummyDocumentStore extends DocumentStore { private Map<Class<? extends Document>, SortedMap<String, Document>> diff --git a/src/test/java/org/torproject/onionoo/docs/NodeStatusTest.java b/src/test/java/org/torproject/onionoo/docs/NodeStatusTest.java index e3a6fca..f001ea1 100644 --- a/src/test/java/org/torproject/onionoo/docs/NodeStatusTest.java +++ b/src/test/java/org/torproject/onionoo/docs/NodeStatusTest.java @@ -2,12 +2,12 @@ package org.torproject.onionoo.docs; import static org.junit.Assert.assertEquals; +import org.junit.Test; + import java.util.Arrays; import java.util.SortedSet; import java.util.TreeSet; -import org.junit.Test; - public class NodeStatusTest { private static final String GABELMOO_NODE_STATUS = diff --git a/src/test/java/org/torproject/onionoo/docs/SummaryDocumentTest.java b/src/test/java/org/torproject/onionoo/docs/SummaryDocumentTest.java index f25528c..3c097ae 100644 --- a/src/test/java/org/torproject/onionoo/docs/SummaryDocumentTest.java +++ b/src/test/java/org/torproject/onionoo/docs/SummaryDocumentTest.java @@ -4,11 +4,11 @@ package org.torproject.onionoo.docs; import static org.junit.Assert.assertTrue; +import org.junit.Test; + import java.util.Arrays; import java.util.TreeSet; -import org.junit.Test; - public class SummaryDocumentTest { @Test() diff --git a/src/test/java/org/torproject/onionoo/docs/UptimeStatusTest.java b/src/test/java/org/torproject/onionoo/docs/UptimeStatusTest.java index 7a48c25..44da144 100644 --- a/src/test/java/org/torproject/onionoo/docs/UptimeStatusTest.java +++ b/src/test/java/org/torproject/onionoo/docs/UptimeStatusTest.java @@ -6,12 +6,12 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; +import org.junit.Test; + import java.util.Arrays; import java.util.SortedSet; import java.util.TreeSet; -import org.junit.Test; - public class UptimeStatusTest { @Test() diff --git a/src/test/java/org/torproject/onionoo/server/ResourceServletTest.java b/src/test/java/org/torproject/onionoo/server/ResourceServletTest.java index 8370b16..ffea950 100644 --- a/src/test/java/org/torproject/onionoo/server/ResourceServletTest.java +++ b/src/test/java/org/torproject/onionoo/server/ResourceServletTest.java @@ -7,6 +7,19 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; +import org.torproject.onionoo.docs.DateTimeHelper; +import org.torproject.onionoo.docs.DocumentStoreFactory; +import org.torproject.onionoo.docs.DummyDocumentStore; +import org.torproject.onionoo.docs.UpdateStatus; +import org.torproject.onionoo.util.DummyTime; +import org.torproject.onionoo.util.Time; +import org.torproject.onionoo.util.TimeFactory; + +import com.google.gson.Gson; + +import org.junit.Before; +import org.junit.Test; + import java.io.IOException; import java.io.PrintWriter; import java.io.StringWriter; @@ -19,23 +32,6 @@ import java.util.SortedMap; import java.util.TreeMap; import java.util.TreeSet; -import org.junit.Before; -import org.junit.Test; -import org.torproject.onionoo.docs.DateTimeHelper; -import org.torproject.onionoo.docs.DocumentStoreFactory; -import org.torproject.onionoo.docs.DummyDocumentStore; -import org.torproject.onionoo.docs.UpdateStatus; -import org.torproject.onionoo.server.HttpServletRequestWrapper; -import org.torproject.onionoo.server.HttpServletResponseWrapper; -import org.torproject.onionoo.server.NodeIndexer; -import org.torproject.onionoo.server.NodeIndexerFactory; -import org.torproject.onionoo.server.ResourceServlet; -import org.torproject.onionoo.util.DummyTime; -import org.torproject.onionoo.util.Time; -import org.torproject.onionoo.util.TimeFactory; - -import com.google.gson.Gson; - /* TODO This test class could (should?) be split into ResponseBuilderTest * which tests ResponseBuilder and a much shorter ResourceServletTest * which tests servlet specifics. */ diff --git a/src/test/java/org/torproject/onionoo/updater/DummyBridgeStatus.java b/src/test/java/org/torproject/onionoo/updater/DummyBridgeStatus.java index ff6f104..f412de8 100644 --- a/src/test/java/org/torproject/onionoo/updater/DummyBridgeStatus.java +++ b/src/test/java/org/torproject/onionoo/updater/DummyBridgeStatus.java @@ -2,13 +2,13 @@ * See LICENSE for licensing information */ package org.torproject.onionoo.updater; +import org.torproject.descriptor.BridgeNetworkStatus; +import org.torproject.descriptor.NetworkStatusEntry; + import java.util.List; import java.util.SortedMap; import java.util.TreeMap; -import org.torproject.descriptor.BridgeNetworkStatus; -import org.torproject.descriptor.NetworkStatusEntry; - public class DummyBridgeStatus implements BridgeNetworkStatus { public byte[] getRawDescriptorBytes() { diff --git a/src/test/java/org/torproject/onionoo/updater/DummyConsensus.java b/src/test/java/org/torproject/onionoo/updater/DummyConsensus.java index a2d1e3e..0f01a5e 100644 --- a/src/test/java/org/torproject/onionoo/updater/DummyConsensus.java +++ b/src/test/java/org/torproject/onionoo/updater/DummyConsensus.java @@ -2,17 +2,17 @@ * See LICENSE for licensing information */ package org.torproject.onionoo.updater; +import org.torproject.descriptor.DirSourceEntry; +import org.torproject.descriptor.DirectorySignature; +import org.torproject.descriptor.NetworkStatusEntry; +import org.torproject.descriptor.RelayNetworkStatusConsensus; + import java.util.List; import java.util.SortedMap; import java.util.SortedSet; import java.util.TreeMap; import java.util.TreeSet; -import org.torproject.descriptor.DirSourceEntry; -import org.torproject.descriptor.DirectorySignature; -import org.torproject.descriptor.NetworkStatusEntry; -import org.torproject.descriptor.RelayNetworkStatusConsensus; - public class DummyConsensus implements RelayNetworkStatusConsensus { public byte[] getRawDescriptorBytes() { diff --git a/src/test/java/org/torproject/onionoo/updater/DummyDescriptorSource.java b/src/test/java/org/torproject/onionoo/updater/DummyDescriptorSource.java index 10df38a..40c1c65 100644 --- a/src/test/java/org/torproject/onionoo/updater/DummyDescriptorSource.java +++ b/src/test/java/org/torproject/onionoo/updater/DummyDescriptorSource.java @@ -1,16 +1,13 @@ package org.torproject.onionoo.updater; +import org.torproject.descriptor.Descriptor; + import java.util.Collection; import java.util.HashMap; import java.util.HashSet; import java.util.Map; import java.util.Set; -import org.torproject.descriptor.Descriptor; -import org.torproject.onionoo.updater.DescriptorListener; -import org.torproject.onionoo.updater.DescriptorSource; -import org.torproject.onionoo.updater.DescriptorType; - public class DummyDescriptorSource extends DescriptorSource { private Map<DescriptorType, Set<Descriptor>> descriptors = diff --git a/src/test/java/org/torproject/onionoo/updater/DummyStatusEntry.java b/src/test/java/org/torproject/onionoo/updater/DummyStatusEntry.java index b480c37..120fc28 100644 --- a/src/test/java/org/torproject/onionoo/updater/DummyStatusEntry.java +++ b/src/test/java/org/torproject/onionoo/updater/DummyStatusEntry.java @@ -2,13 +2,13 @@ * See LICENSE for licensing information */ package org.torproject.onionoo.updater; +import org.torproject.descriptor.NetworkStatusEntry; + import java.util.List; import java.util.Set; import java.util.SortedSet; import java.util.TreeSet; -import org.torproject.descriptor.NetworkStatusEntry; - public class DummyStatusEntry implements NetworkStatusEntry { public DummyStatusEntry(String fingerprint) { diff --git a/src/test/java/org/torproject/onionoo/updater/LookupServiceTest.java b/src/test/java/org/torproject/onionoo/updater/LookupServiceTest.java index 7faab7c..3b89b9a 100644 --- a/src/test/java/org/torproject/onionoo/updater/LookupServiceTest.java +++ b/src/test/java/org/torproject/onionoo/updater/LookupServiceTest.java @@ -6,6 +6,11 @@ package org.torproject.onionoo.updater; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + import java.io.BufferedWriter; import java.io.File; import java.io.FileOutputStream; @@ -19,13 +24,6 @@ import java.util.SortedMap; import java.util.SortedSet; import java.util.TreeSet; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import org.junit.rules.TemporaryFolder; -import org.torproject.onionoo.updater.LookupResult; -import org.torproject.onionoo.updater.LookupService; - public class LookupServiceTest { private List<String> geoLite2CityBlocksIPv4Lines, diff --git a/src/test/java/org/torproject/onionoo/updater/UptimeStatusUpdaterTest.java b/src/test/java/org/torproject/onionoo/updater/UptimeStatusUpdaterTest.java index 25399ab..278ed07 100644 --- a/src/test/java/org/torproject/onionoo/updater/UptimeStatusUpdaterTest.java +++ b/src/test/java/org/torproject/onionoo/updater/UptimeStatusUpdaterTest.java @@ -4,14 +4,15 @@ package org.torproject.onionoo.updater; import static org.junit.Assert.assertEquals; -import org.junit.Before; -import org.junit.Test; import org.torproject.onionoo.docs.DateTimeHelper; import org.torproject.onionoo.docs.DocumentStoreFactory; import org.torproject.onionoo.docs.DummyDocumentStore; import org.torproject.onionoo.docs.UptimeHistory; import org.torproject.onionoo.docs.UptimeStatus; +import org.junit.Before; +import org.junit.Test; + public class UptimeStatusUpdaterTest { private DummyDescriptorSource descriptorSource; diff --git a/src/test/java/org/torproject/onionoo/util/DummyTime.java b/src/test/java/org/torproject/onionoo/util/DummyTime.java index ec7a50f..907805c 100644 --- a/src/test/java/org/torproject/onionoo/util/DummyTime.java +++ b/src/test/java/org/torproject/onionoo/util/DummyTime.java @@ -2,8 +2,6 @@ * See LICENSE for licensing information */ package org.torproject.onionoo.util; -import org.torproject.onionoo.util.Time; - public class DummyTime extends Time { private long currentTimeMillis; public DummyTime(long currentTimeMillis) { diff --git a/src/test/java/org/torproject/onionoo/writer/UptimeDocumentWriterTest.java b/src/test/java/org/torproject/onionoo/writer/UptimeDocumentWriterTest.java index 517efc0..4e5b69c 100644 --- a/src/test/java/org/torproject/onionoo/writer/UptimeDocumentWriterTest.java +++ b/src/test/java/org/torproject/onionoo/writer/UptimeDocumentWriterTest.java @@ -5,11 +5,6 @@ package org.torproject.onionoo.writer; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; -import java.util.Arrays; -import java.util.List; - -import org.junit.Before; -import org.junit.Test; import org.torproject.onionoo.docs.DateTimeHelper; import org.torproject.onionoo.docs.DocumentStoreFactory; import org.torproject.onionoo.docs.DummyDocumentStore; @@ -21,6 +16,12 @@ import org.torproject.onionoo.updater.DummyDescriptorSource; import org.torproject.onionoo.util.DummyTime; import org.torproject.onionoo.util.TimeFactory; +import org.junit.Before; +import org.junit.Test; + +import java.util.Arrays; +import java.util.List; + public class UptimeDocumentWriterTest { private static final long TEST_TIME = DateTimeHelper.parse(

1 0

[onionoo/master] Update copyright notice to 2016.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit fb79b149ba96f49f758f82668e94633bfaca8ac9 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 16:32:34 2016 +0200 Update copyright notice to 2016. --- src/main/java/org/torproject/onionoo/cron/Main.java | 2 +- src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java | 2 +- src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java | 2 +- src/main/java/org/torproject/onionoo/docs/ClientsDocument.java | 2 +- src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java | 2 +- src/main/java/org/torproject/onionoo/docs/ClientsHistory.java | 2 +- src/main/java/org/torproject/onionoo/docs/ClientsStatus.java | 2 +- src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java | 2 +- src/main/java/org/torproject/onionoo/docs/DetailsDocument.java | 2 +- src/main/java/org/torproject/onionoo/docs/DetailsStatus.java | 2 +- src/main/java/org/torproject/onionoo/docs/Document.java | 2 +- src/main/java/org/torproject/onionoo/docs/DocumentStore.java | 2 +- src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java | 2 +- src/main/java/org/torproject/onionoo/docs/GraphHistory.java | 2 +- src/main/java/org/torproject/onionoo/docs/NodeStatus.java | 2 +- src/main/java/org/torproject/onionoo/docs/SummaryDocument.java | 2 +- src/main/java/org/torproject/onionoo/docs/UpdateStatus.java | 2 +- src/main/java/org/torproject/onionoo/docs/UptimeDocument.java | 2 +- src/main/java/org/torproject/onionoo/docs/UptimeHistory.java | 3 +++ src/main/java/org/torproject/onionoo/docs/UptimeStatus.java | 2 +- src/main/java/org/torproject/onionoo/docs/WeightsDocument.java | 2 +- src/main/java/org/torproject/onionoo/docs/WeightsStatus.java | 3 +++ .../java/org/torproject/onionoo/server/HttpServletRequestWrapper.java | 2 +- .../java/org/torproject/onionoo/server/HttpServletResponseWrapper.java | 2 +- src/main/java/org/torproject/onionoo/server/NodeIndex.java | 3 +++ src/main/java/org/torproject/onionoo/server/NodeIndexer.java | 3 +++ src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java | 2 +- src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java | 2 +- src/main/java/org/torproject/onionoo/server/RequestHandler.java | 2 +- src/main/java/org/torproject/onionoo/server/ResourceServlet.java | 2 +- src/main/java/org/torproject/onionoo/server/ResponseBuilder.java | 2 +- src/main/java/org/torproject/onionoo/server/ServerMain.java | 2 +- .../java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java | 2 +- src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java | 2 +- src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java | 3 +++ src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java | 3 +++ src/main/java/org/torproject/onionoo/updater/DescriptorListener.java | 3 +++ src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java | 3 +++ src/main/java/org/torproject/onionoo/updater/DescriptorSource.java | 2 +- .../java/org/torproject/onionoo/updater/DescriptorSourceFactory.java | 2 +- src/main/java/org/torproject/onionoo/updater/DescriptorType.java | 2 +- src/main/java/org/torproject/onionoo/updater/LookupResult.java | 2 +- src/main/java/org/torproject/onionoo/updater/LookupService.java | 2 +- .../java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java | 2 +- src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java | 2 +- src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java | 2 +- .../java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java | 2 +- src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java | 2 +- src/main/java/org/torproject/onionoo/updater/StatusUpdater.java | 2 +- src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java | 2 +- src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java | 2 +- src/main/java/org/torproject/onionoo/util/FormattingUtils.java | 2 +- src/main/java/org/torproject/onionoo/util/LockFile.java | 2 +- src/main/java/org/torproject/onionoo/util/Time.java | 2 +- src/main/java/org/torproject/onionoo/util/TimeFactory.java | 2 +- .../java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java | 2 +- src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java | 2 +- src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java | 3 +++ src/main/java/org/torproject/onionoo/writer/DocumentWriter.java | 2 +- src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java | 2 +- src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java | 2 +- src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java | 2 +- src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java | 2 +- 63 files changed, 81 insertions(+), 54 deletions(-) diff --git a/src/main/java/org/torproject/onionoo/cron/Main.java b/src/main/java/org/torproject/onionoo/cron/Main.java index 6392045..4a57f66 100644 --- a/src/main/java/org/torproject/onionoo/cron/Main.java +++ b/src/main/java/org/torproject/onionoo/cron/Main.java @@ -1,4 +1,4 @@ -/* Copyright 2011--2015 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.cron; diff --git a/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java b/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java index 9196fe9..6fe3240 100644 --- a/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/BandwidthDocument.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java b/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java index 828fbab..65d4cf5 100644 --- a/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/BandwidthStatus.java @@ -1,4 +1,4 @@ -/* Copyright 2013--2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java b/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java index 0b4be77..89729d2 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsDocument.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java b/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java index a3c0ef9..0826887 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsGraphHistory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java b/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java index b100344..0f4c148 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsHistory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java b/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java index bdd5666..4ef998e 100644 --- a/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/ClientsStatus.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java b/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java index 67a38c7..4e422e9 100644 --- a/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java +++ b/src/main/java/org/torproject/onionoo/docs/DateTimeHelper.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java b/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java index a80a504..8ffb358 100644 --- a/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/DetailsDocument.java @@ -1,4 +1,4 @@ -/* Copyright 2013--2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java index 599bb59..dcc6fcb 100644 --- a/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/DetailsStatus.java @@ -1,4 +1,4 @@ -/* Copyright 2013--2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/Document.java b/src/main/java/org/torproject/onionoo/docs/Document.java index 4c3a7c3..f2f9714 100644 --- a/src/main/java/org/torproject/onionoo/docs/Document.java +++ b/src/main/java/org/torproject/onionoo/docs/Document.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java index 8b71427..600115c 100644 --- a/src/main/java/org/torproject/onionoo/docs/DocumentStore.java +++ b/src/main/java/org/torproject/onionoo/docs/DocumentStore.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java b/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java index 40916a7..3fa28fc 100644 --- a/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java +++ b/src/main/java/org/torproject/onionoo/docs/DocumentStoreFactory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/GraphHistory.java b/src/main/java/org/torproject/onionoo/docs/GraphHistory.java index 165d059..5dde6a7 100644 --- a/src/main/java/org/torproject/onionoo/docs/GraphHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/GraphHistory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/NodeStatus.java b/src/main/java/org/torproject/onionoo/docs/NodeStatus.java index 46cb946..630171b 100644 --- a/src/main/java/org/torproject/onionoo/docs/NodeStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/NodeStatus.java @@ -1,4 +1,4 @@ -/* Copyright 2011, 2012 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java b/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java index d8d82ce..cebcf5e 100644 --- a/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/SummaryDocument.java @@ -1,4 +1,4 @@ -/* Copyright 2013--2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java b/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java index 6eee47d..07d5d20 100644 --- a/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/UpdateStatus.java @@ -1,4 +1,4 @@ -/* Copyright 2013, 2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java b/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java index c34fa52..505d66c 100644 --- a/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/UptimeDocument.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java b/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java index fc8c86a..6f1cb2e 100644 --- a/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java +++ b/src/main/java/org/torproject/onionoo/docs/UptimeHistory.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import org.slf4j.Logger; diff --git a/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java b/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java index e0a25b3..ce3a4d1 100644 --- a/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/UptimeStatus.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java b/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java index 62dcf20..6e2ee02 100644 --- a/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java +++ b/src/main/java/org/torproject/onionoo/docs/WeightsDocument.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.docs; diff --git a/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java b/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java index 77ce583..47eaed9 100644 --- a/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java +++ b/src/main/java/org/torproject/onionoo/docs/WeightsStatus.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.docs; import org.torproject.onionoo.util.TimeFactory; diff --git a/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java b/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java index 901eece..1aa964a 100644 --- a/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java +++ b/src/main/java/org/torproject/onionoo/server/HttpServletRequestWrapper.java @@ -1,4 +1,4 @@ -/* Copyright 2011, 2012 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java b/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java index a3f3fff..d804c3a 100644 --- a/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java +++ b/src/main/java/org/torproject/onionoo/server/HttpServletResponseWrapper.java @@ -1,4 +1,4 @@ -/* Copyright 2011, 2012 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/NodeIndex.java b/src/main/java/org/torproject/onionoo/server/NodeIndex.java index 82e5e08..c000bba 100644 --- a/src/main/java/org/torproject/onionoo/server/NodeIndex.java +++ b/src/main/java/org/torproject/onionoo/server/NodeIndex.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.server; import org.torproject.onionoo.docs.SummaryDocument; diff --git a/src/main/java/org/torproject/onionoo/server/NodeIndexer.java b/src/main/java/org/torproject/onionoo/server/NodeIndexer.java index 4e123d9..30a6f0e 100644 --- a/src/main/java/org/torproject/onionoo/server/NodeIndexer.java +++ b/src/main/java/org/torproject/onionoo/server/NodeIndexer.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.server; import org.torproject.onionoo.docs.DocumentStore; diff --git a/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java b/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java index cc045f7..1dfa859 100644 --- a/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java +++ b/src/main/java/org/torproject/onionoo/server/NodeIndexerFactory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java b/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java index 7bf85b6..a12e5c7 100644 --- a/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java +++ b/src/main/java/org/torproject/onionoo/server/PerformanceMetrics.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/RequestHandler.java b/src/main/java/org/torproject/onionoo/server/RequestHandler.java index cb71681..85a6ff1 100644 --- a/src/main/java/org/torproject/onionoo/server/RequestHandler.java +++ b/src/main/java/org/torproject/onionoo/server/RequestHandler.java @@ -1,4 +1,4 @@ -/* Copyright 2011--2014 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/ResourceServlet.java b/src/main/java/org/torproject/onionoo/server/ResourceServlet.java index 8430bdc..4fd38b7 100644 --- a/src/main/java/org/torproject/onionoo/server/ResourceServlet.java +++ b/src/main/java/org/torproject/onionoo/server/ResourceServlet.java @@ -1,4 +1,4 @@ -/* Copyright 2011, 2012 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java b/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java index 96627d3..cd0c633 100644 --- a/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java +++ b/src/main/java/org/torproject/onionoo/server/ResponseBuilder.java @@ -1,4 +1,4 @@ -/* Copyright 2011--2013 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/server/ServerMain.java b/src/main/java/org/torproject/onionoo/server/ServerMain.java index de3a0d9..22e315b 100644 --- a/src/main/java/org/torproject/onionoo/server/ServerMain.java +++ b/src/main/java/org/torproject/onionoo/server/ServerMain.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.server; diff --git a/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java index 6286741..1c0c181 100644 --- a/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/BandwidthStatusUpdater.java @@ -1,4 +1,4 @@ -/* Copyright 2011--2014 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java index 1b55273..492b67b 100644 --- a/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/ClientsStatusUpdater.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java b/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java index 64dec9b..71a43d2 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorDownloader.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; import org.slf4j.Logger; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java b/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java index f0f9eba..28eaec7 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorHistory.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; enum DescriptorHistory { diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java b/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java index a93bef6..328fd6f 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorListener.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; import org.torproject.descriptor.Descriptor; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java b/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java index 7892f40..c17d498 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorQueue.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.updater; import org.torproject.descriptor.Descriptor; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java b/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java index 05e9bd8..b9d07b4 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorSource.java @@ -1,4 +1,4 @@ -/* Copyright 2013, 2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java b/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java index a68cd9f..65c5622 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorSourceFactory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/DescriptorType.java b/src/main/java/org/torproject/onionoo/updater/DescriptorType.java index 98c345a..b7b6b8e 100644 --- a/src/main/java/org/torproject/onionoo/updater/DescriptorType.java +++ b/src/main/java/org/torproject/onionoo/updater/DescriptorType.java @@ -1,4 +1,4 @@ -/* Copyright 2013, 2014 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/LookupResult.java b/src/main/java/org/torproject/onionoo/updater/LookupResult.java index 476783d..930e2fb 100644 --- a/src/main/java/org/torproject/onionoo/updater/LookupResult.java +++ b/src/main/java/org/torproject/onionoo/updater/LookupResult.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/LookupService.java b/src/main/java/org/torproject/onionoo/updater/LookupService.java index 357ff4f..85d56e8 100644 --- a/src/main/java/org/torproject/onionoo/updater/LookupService.java +++ b/src/main/java/org/torproject/onionoo/updater/LookupService.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java index ef7246a..531b20e 100644 --- a/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java @@ -1,4 +1,4 @@ -/* Copyright 2011--2014 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java b/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java index 39aba75..e2ee5e1 100644 --- a/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java +++ b/src/main/java/org/torproject/onionoo/updater/RdnsLookupRequest.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java b/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java index c9af126..0c48be5 100644 --- a/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java +++ b/src/main/java/org/torproject/onionoo/updater/RdnsLookupWorker.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java b/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java index 293ee82..0f3caad 100644 --- a/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java +++ b/src/main/java/org/torproject/onionoo/updater/ReverseDomainNameResolver.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java b/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java index c33421b..2ba8401 100644 --- a/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java +++ b/src/main/java/org/torproject/onionoo/updater/StatusUpdateRunner.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java index 258d92f..f8474f4 100644 --- a/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/StatusUpdater.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java index 3edd20e..f2d7dc1 100644 --- a/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/UptimeStatusUpdater.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java index ea7faed..14e425b 100644 --- a/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/WeightsStatusUpdater.java @@ -1,4 +1,4 @@ -/* Copyright 2012--2014 The Tor Project +/* Copyright 2012--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.updater; diff --git a/src/main/java/org/torproject/onionoo/util/FormattingUtils.java b/src/main/java/org/torproject/onionoo/util/FormattingUtils.java index 615f67b..36fb838 100644 --- a/src/main/java/org/torproject/onionoo/util/FormattingUtils.java +++ b/src/main/java/org/torproject/onionoo/util/FormattingUtils.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.util; diff --git a/src/main/java/org/torproject/onionoo/util/LockFile.java b/src/main/java/org/torproject/onionoo/util/LockFile.java index f7aeee8..521b867 100644 --- a/src/main/java/org/torproject/onionoo/util/LockFile.java +++ b/src/main/java/org/torproject/onionoo/util/LockFile.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.util; diff --git a/src/main/java/org/torproject/onionoo/util/Time.java b/src/main/java/org/torproject/onionoo/util/Time.java index 4f6d0bb..1b7e121 100644 --- a/src/main/java/org/torproject/onionoo/util/Time.java +++ b/src/main/java/org/torproject/onionoo/util/Time.java @@ -1,4 +1,4 @@ -/* Copyright 2013 The Tor Project +/* Copyright 2013--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.util; diff --git a/src/main/java/org/torproject/onionoo/util/TimeFactory.java b/src/main/java/org/torproject/onionoo/util/TimeFactory.java index d52a584..e3067a4 100644 --- a/src/main/java/org/torproject/onionoo/util/TimeFactory.java +++ b/src/main/java/org/torproject/onionoo/util/TimeFactory.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.util; diff --git a/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java index 3762065..a844f7a 100644 --- a/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/BandwidthDocumentWriter.java @@ -1,4 +1,4 @@ -/* Copyright 2011--2014 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer; diff --git a/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java index 37d486e..6cbd2e0 100644 --- a/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/ClientsDocumentWriter.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer; diff --git a/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java index 506067d..44c675f 100644 --- a/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/DetailsDocumentWriter.java @@ -1,3 +1,6 @@ +/* Copyright 2016 The Tor Project + * See LICENSE for licensing information */ + package org.torproject.onionoo.writer; import org.torproject.onionoo.docs.DetailsDocument; diff --git a/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java index fa3c264..3fe63b9 100644 --- a/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/DocumentWriter.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer; diff --git a/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java b/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java index fae481f..4809ed9 100644 --- a/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java +++ b/src/main/java/org/torproject/onionoo/writer/DocumentWriterRunner.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer; diff --git a/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java index 89fb166..f0d7fe0 100644 --- a/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/SummaryDocumentWriter.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer; diff --git a/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java index 2a48ccb..e93627b 100644 --- a/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/UptimeDocumentWriter.java @@ -1,4 +1,4 @@ -/* Copyright 2014 The Tor Project +/* Copyright 2014--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer; diff --git a/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java b/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java index 50001f3..cabe964 100644 --- a/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java +++ b/src/main/java/org/torproject/onionoo/writer/WeightsDocumentWriter.java @@ -1,4 +1,4 @@ -/* Copyright 2012--2014 The Tor Project +/* Copyright 2012--2016 The Tor Project * See LICENSE for licensing information */ package org.torproject.onionoo.writer;

1 0

[onionoo/master] Avoid using the deprecated ExitListEntry type.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit e4effb03de01790f46b693d04dbe8b55fb2523e0 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 16:41:59 2016 +0200 Avoid using the deprecated ExitListEntry type. --- .../onionoo/updater/NodeDetailsStatusUpdater.java | 35 ++++++++++++---------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java b/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java index 531b20e..9bceaa1 100644 --- a/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java +++ b/src/main/java/org/torproject/onionoo/updater/NodeDetailsStatusUpdater.java @@ -6,7 +6,6 @@ package org.torproject.onionoo.updater; import org.torproject.descriptor.BridgeNetworkStatus; import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.ExitList; -import org.torproject.descriptor.ExitListEntry; import org.torproject.descriptor.ExtraInfoDescriptor; import org.torproject.descriptor.NetworkStatusEntry; import org.torproject.descriptor.RelayNetworkStatusConsensus; @@ -209,22 +208,26 @@ public class NodeDetailsStatusUpdater implements DescriptorListener, new HashMap<String, Map<String, Long>>(); private void processExitList(ExitList exitList) { - for (ExitListEntry exitListEntry : exitList.getExitListEntries()) { + for (ExitList.Entry exitListEntry : exitList.getEntries()) { String fingerprint = exitListEntry.getFingerprint(); - long scanMillis = exitListEntry.getScanMillis(); - if (scanMillis < this.now - DateTimeHelper.ONE_DAY) { - continue; - } - if (!this.exitListEntries.containsKey(fingerprint)) { - this.exitListEntries.put(fingerprint, - new HashMap<String, Long>()); - } - String exitAddress = exitListEntry.getExitAddress(); - if (!this.exitListEntries.get(fingerprint).containsKey(exitAddress) - || this.exitListEntries.get(fingerprint).get(exitAddress) - < scanMillis) { - this.exitListEntries.get(fingerprint).put(exitAddress, - scanMillis); + for (Map.Entry<String, Long> exitAddressScanMillis + : exitListEntry.getExitAddresses().entrySet()) { + long scanMillis = exitAddressScanMillis.getValue(); + if (scanMillis < this.now - DateTimeHelper.ONE_DAY) { + continue; + } + if (!this.exitListEntries.containsKey(fingerprint)) { + this.exitListEntries.put(fingerprint, + new HashMap<String, Long>()); + } + String exitAddress = exitAddressScanMillis.getKey(); + if (!this.exitListEntries.get(fingerprint).containsKey( + exitAddress) + || this.exitListEntries.get(fingerprint).get(exitAddress) + < scanMillis) { + this.exitListEntries.get(fingerprint).put(exitAddress, + scanMillis); + } } } }

1 0

[exonerator/master] Move Java sources to src/main/java/.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit 172e1498d328a62d0915ab1796ec63202bcdd12e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 10:59:08 2016 +0200 Move Java sources to src/main/java/. --- build.xml | 2 +- .../exonerator/ExoneraTorDatabaseImporter.java | 564 +++++++++++++ .../torproject/exonerator/ExoneraTorServlet.java | 911 +++++++++++++++++++++ .../exonerator/ExoneraTorDatabaseImporter.java | 564 ------------- .../torproject/exonerator/ExoneraTorServlet.java | 911 --------------------- 5 files changed, 1476 insertions(+), 1476 deletions(-) diff --git a/build.xml b/build.xml index 0326520..2e4f74d 100644 --- a/build.xml +++ b/build.xml @@ -1,7 +1,7 @@ <project default="run" name="exonerator" basedir=".">  - <property name="sources" value="src"/> + <property name="sources" value="src/main/java"/> <property name="resources" value="res"/> <property name="classes" value="classes"/> <property name="libs" value="lib"/> diff --git a/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java b/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java new file mode 100644 index 0000000..41751ca --- /dev/null +++ b/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java @@ -0,0 +1,564 @@ +/* Copyright 2011, 2012 The Tor Project + * See LICENSE for licensing information */ +package org.torproject.exonerator; + +import java.io.BufferedInputStream; +import java.io.BufferedReader; +import java.io.BufferedWriter; +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.FileInputStream; +import java.io.FileReader; +import java.io.FileWriter; +import java.io.IOException; +import java.io.StringReader; +import java.io.UnsupportedEncodingException; +import java.sql.CallableStatement; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Timestamp; +import java.sql.Types; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.Calendar; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Map; +import java.util.Set; +import java.util.Stack; +import java.util.TimeZone; + +import org.apache.commons.codec.binary.Base64; +import org.apache.commons.codec.binary.Hex; +import org.apache.commons.codec.digest.DigestUtils; +import org.torproject.descriptor.DescriptorCollector; +import org.torproject.descriptor.DescriptorSourceFactory; + +/* Import Tor descriptors into the ExoneraTor database. */ +public class ExoneraTorDatabaseImporter { + + /* Main function controlling the parsing process. */ + public static void main(String[] args) { + readConfiguration(); + openDatabaseConnection(); + prepareDatabaseStatements(); + createLockFile(); + fetchDescriptors(); + readImportHistoryToMemory(); + parseDescriptors(); + writeImportHistoryToDisk(); + closeDatabaseConnection(); + deleteLockFile(); + } + + /* JDBC string of the ExoneraTor database. */ + private static String jdbcString; + + /* Directory from which to import descriptors. */ + private static String importDirString; + + /* Learn JDBC string and directory to parse descriptors from. */ + private static void readConfiguration() { + File configFile = new File("config"); + if (!configFile.exists()) { + System.err.println("Could not find config file. Exiting."); + System.exit(1); + } + String line = null; + try { + BufferedReader br = new BufferedReader(new FileReader(configFile)); + while ((line = br.readLine()) != null) { + if (line.startsWith("#") || line.length() < 1) { + continue; + } else if (line.startsWith("ExoneraTorDatabaseJdbc")) { + jdbcString = line.split(" ")[1]; + } else if (line.startsWith("ExoneraTorImportDirectory")) { + importDirString = line.split(" ")[1]; + } else { + /* Ignore unrecognized configuration keys. */ + } + } + br.close(); + } catch (IOException e) { + System.err.println("Could not parse config file. Exiting."); + System.exit(1); + } + } + + /* Database connection. */ + private static Connection connection; + + /* Open a database connection using the JDBC string in the config. */ + private static void openDatabaseConnection() { + try { + connection = DriverManager.getConnection(jdbcString); + } catch (SQLException e) { + System.out.println("Could not connect to database. Exiting."); + System.exit(1); + } + } + + /* Callable statements to import data into the database. */ + private static CallableStatement insertStatusentryStatement; + private static CallableStatement insertExitlistentryStatement; + + /* Prepare statements for importing data into the database. */ + private static void prepareDatabaseStatements() { + try { + insertStatusentryStatement = connection.prepareCall( + "{call insert_statusentry(?, ?, ?, ?, ?, ?, ?)}"); + insertExitlistentryStatement = connection.prepareCall( + "{call insert_exitlistentry(?, ?, ?, ?, ?)}"); + } catch (SQLException e) { + System.out.println("Could not prepare callable statements to " + + "import data into the database. Exiting."); + System.exit(1); + } + } + + /* Create a local lock file to prevent other instances of this import + * tool to run concurrently. */ + private static void createLockFile() { + File lockFile = new File("exonerator-lock"); + try { + if (lockFile.exists()) { + BufferedReader br = new BufferedReader(new FileReader(lockFile)); + long runStarted = Long.parseLong(br.readLine()); + br.close(); + if (System.currentTimeMillis() - runStarted + < 6L * 60L * 60L * 1000L) { + System.out.println("File 'exonerator-lock' is less than 6 " + + "hours old. Exiting."); + System.exit(1); + } else { + System.out.println("File 'exonerator-lock' is at least 6 hours " + + "old. Overwriting and executing anyway."); + } + } + BufferedWriter bw = new BufferedWriter(new FileWriter( + "exonerator-lock")); + bw.append(String.valueOf(System.currentTimeMillis()) + "\n"); + bw.close(); + } catch (IOException e) { + System.out.println("Could not create 'exonerator-lock' file. " + + "Exiting."); + System.exit(1); + } + } + + /* Fetch recent descriptors from CollecTor. */ + private static void fetchDescriptors() { + DescriptorCollector collector = + DescriptorSourceFactory.createDescriptorCollector(); + collector.collectDescriptors("https://collector.torproject.org", + new String[] { "/recent/relay-descriptors/consensuses/", + "/recent/exit-lists/" }, 0L, new File(importDirString), true); + } + + /* Last and next parse histories containing paths of parsed files and + * last modified times. */ + private static Map<String, Long> + lastImportHistory = new HashMap<String, Long>(), + nextImportHistory = new HashMap<String, Long>(); + + /* Read stats/exonerator-import-history file from disk and remember + * locally when files were last parsed. */ + private static void readImportHistoryToMemory() { + File parseHistoryFile = new File("stats", + "exonerator-import-history"); + if (parseHistoryFile.exists()) { + try { + BufferedReader br = new BufferedReader(new FileReader( + parseHistoryFile)); + String line = null; + int lineNumber = 0; + while ((line = br.readLine()) != null) { + lineNumber++; + String[] parts = line.split(","); + if (parts.length != 2) { + System.out.println("File 'stats/exonerator-import-history' " + + "contains a corrupt entry in line " + lineNumber + + ". Ignoring parse history file entirely."); + lastImportHistory.clear(); + br.close(); + return; + } + long lastModified = Long.parseLong(parts[0]); + String filename = parts[1]; + lastImportHistory.put(filename, lastModified); + } + br.close(); + } catch (IOException e) { + System.out.println("Could not read import history. Ignoring."); + lastImportHistory.clear(); + } + } + } + + /* Parse descriptors in the import directory and its subdirectories. */ + private static void parseDescriptors() { + File file = new File(importDirString); + if (!file.exists()) { + System.out.println("File or directory " + importDirString + " does " + + "not exist. Exiting."); + return; + } + Stack<File> files = new Stack<File>(); + files.add(file); + while (!files.isEmpty()) { + file = files.pop(); + if (file.isDirectory()) { + for (File f : file.listFiles()) { + files.add(f); + } + } else { + parseFile(file); + } + } + } + + /* Import a file if it wasn't imported before, and add it to the import + * history for the next execution. */ + private static void parseFile(File file) { + long lastModified = file.lastModified(); + String filename = file.getName(); + nextImportHistory.put(filename, lastModified); + if (!lastImportHistory.containsKey(filename) || + lastImportHistory.get(filename) < lastModified) { + try { + FileInputStream fis = new FileInputStream(file); + BufferedInputStream bis = new BufferedInputStream(fis); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + int len; + byte[] bytes = new byte[1024]; + while ((len = bis.read(bytes, 0, 1024)) >= 0) { + baos.write(bytes, 0, len); + } + bis.close(); + byte[] allBytes = baos.toByteArray(); + splitFile(file, allBytes); + } catch (IOException e) { + System.out.println("Could not read '" + file + "' to memory. " + + "Skipping."); + nextImportHistory.remove(filename); + } + } + } + + /* Detect what descriptor type is contained in a file and split it to + * parse the single descriptors. */ + private static void splitFile(File file, byte[] bytes) { + try { + String asciiString = new String(bytes, "US-ASCII"); + BufferedReader br = new BufferedReader(new StringReader( + asciiString)); + String line = br.readLine(); + while (line != null && line.startsWith("@")) { + line = br.readLine(); + } + if (line == null) { + return; + } + br.close(); + String startToken = null; + if (line.equals("network-status-version 3")) { + startToken = "network-status-version 3"; + } else if (line.startsWith("Downloaded ") || + line.startsWith("ExitNode ")) { + startToken = "ExitNode "; + } else { + System.out.println("Unknown descriptor type in file '" + file + + "'. Ignoring."); + return; + } + String splitToken = "\n" + startToken; + int length = bytes.length, start = asciiString.indexOf(startToken); + while (start < length) { + int end = asciiString.indexOf(splitToken, start); + if (end < 0) { + end = length; + } else { + end += 1; + } + byte[] descBytes = new byte[end - start]; + System.arraycopy(bytes, start, descBytes, 0, end - start); + if (startToken.equals("network-status-version 3")) { + parseConsensus(file, descBytes); + } else if (startToken.equals("ExitNode ")) { + parseExitList(file, descBytes); + } + start = end; + } + } catch (IOException e) { + System.out.println("Could not parse descriptor '" + file + "'. " + + "Skipping."); + } + } + + /* Date format to parse UTC timestamps. */ + private static SimpleDateFormat parseFormat; + static { + parseFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); + parseFormat.setTimeZone(TimeZone.getTimeZone("UTC")); + } + + /* Parse a consensus. */ + private static void parseConsensus(File file, byte[] bytes) { + try { + BufferedReader br = new BufferedReader(new StringReader(new String( + bytes, "US-ASCII"))); + String line, fingerprint = null, descriptor = null; + Set<String> orAddresses = new HashSet<String>(); + long validAfterMillis = -1L; + StringBuilder rawStatusentryBuilder = null; + boolean isRunning = false; + while ((line = br.readLine()) != null) { + if (line.startsWith("vote-status ") && + !line.equals("vote-status consensus")) { + System.out.println("File '" + file + "' contains network status " + + "*votes*, not network status *consensuses*. Skipping."); + return; + } else if (line.startsWith("valid-after ")) { + String validAfterTime = line.substring("valid-after ".length()); + try { + validAfterMillis = parseFormat.parse(validAfterTime). + getTime(); + } catch (ParseException e) { + System.out.println("Could not parse valid-after timestamp in " + + "'" + file + "'. Skipping."); + return; + } + } else if (line.startsWith("r ") || + line.equals("directory-footer")) { + if (isRunning) { + byte[] rawStatusentry = rawStatusentryBuilder.toString(). + getBytes(); + importStatusentry(validAfterMillis, fingerprint, descriptor, + orAddresses, rawStatusentry); + orAddresses = new HashSet<String>(); + } + if (line.equals("directory-footer")) { + return; + } + rawStatusentryBuilder = new StringBuilder(line + "\n"); + String[] parts = line.split(" "); + if (parts.length < 9) { + System.out.println("Could not parse r line '" + line + + "'. Skipping."); + return; + } + fingerprint = Hex.encodeHexString(Base64.decodeBase64(parts[2] + + "=")).toLowerCase(); + descriptor = Hex.encodeHexString(Base64.decodeBase64(parts[3] + + "=")).toLowerCase(); + orAddresses.add(parts[6]); + } else if (line.startsWith("a ")) { + rawStatusentryBuilder.append(line + "\n"); + orAddresses.add(line.substring("a ".length(), + line.lastIndexOf(":"))); + } else if (line.startsWith("s ") || line.equals("s")) { + rawStatusentryBuilder.append(line + "\n"); + isRunning = line.contains(" Running"); + } else if (rawStatusentryBuilder != null) { + rawStatusentryBuilder.append(line + "\n"); + } + } + } catch (IOException e) { + System.out.println("Could not parse consensus. Skipping."); + return; + } + } + + /* UTC calendar for importing timestamps into the database. */ + private static Calendar calendarUTC = Calendar.getInstance( + TimeZone.getTimeZone("UTC")); + + /* Import a status entry with one or more OR addresses into the + * database. */ + private static void importStatusentry(long validAfterMillis, + String fingerprint, String descriptor, Set<String> orAddresses, + byte[] rawStatusentry) { + try { + for (String orAddress : orAddresses) { + insertStatusentryStatement.clearParameters(); + insertStatusentryStatement.setTimestamp(1, + new Timestamp(validAfterMillis), calendarUTC); + insertStatusentryStatement.setString(2, fingerprint); + insertStatusentryStatement.setString(3, descriptor); + if (!orAddress.contains(":")) { + String[] addressParts = orAddress.split("\\."); + byte[] address24Bytes = new byte[3]; + address24Bytes[0] = (byte) Integer.parseInt(addressParts[0]); + address24Bytes[1] = (byte) Integer.parseInt(addressParts[1]); + address24Bytes[2] = (byte) Integer.parseInt(addressParts[2]); + String orAddress24 = Hex.encodeHexString(address24Bytes); + insertStatusentryStatement.setString(4, orAddress24); + insertStatusentryStatement.setNull(5, Types.VARCHAR); + insertStatusentryStatement.setString(6, orAddress); + } else { + StringBuilder addressHex = new StringBuilder(); + int start = orAddress.startsWith("[::") ? 2 : 1; + int end = orAddress.length() + - (orAddress.endsWith("::]") ? 2 : 1); + String[] parts = orAddress.substring(start, end).split(":", -1); + for (int i = 0; i < parts.length; i++) { + String part = parts[i]; + if (part.length() == 0) { + addressHex.append("x"); + } else if (part.length() <= 4) { + addressHex.append(String.format("%4s", part)); + } else { + addressHex = null; + break; + } + } + String orAddress48 = null; + if (addressHex != null) { + String addressHexString = addressHex.toString(); + addressHexString = addressHexString.replaceFirst("x", + String.format("%" + (33 - addressHexString.length()) + + "s", "0")); + if (!addressHexString.contains("x") && + addressHexString.length() == 32) { + orAddress48 = addressHexString.replaceAll(" ", "0"). + toLowerCase().substring(0, 12); + } + } + if (orAddress48 != null) { + insertStatusentryStatement.setNull(4, Types.VARCHAR); + insertStatusentryStatement.setString(5, orAddress48); + insertStatusentryStatement.setString(6, + orAddress.replaceAll("[\\[\\]]", "")); + } else { + System.err.println("Could not import status entry with IPv6 " + + "address '" + orAddress + "'. Exiting."); + System.exit(1); + } + } + insertStatusentryStatement.setBytes(7, rawStatusentry); + insertStatusentryStatement.execute(); + } + } catch (SQLException e) { + System.out.println("Could not import status entry. Exiting."); + System.exit(1); + } + } + + /* Parse an exit list. */ + private static void parseExitList(File file, byte[] bytes) { + try { + BufferedReader br = new BufferedReader(new StringReader(new String( + bytes, "US-ASCII"))); + String fingerprint = null; + Set<String> exitAddressLines = new HashSet<String>(); + StringBuilder rawExitlistentryBuilder = new StringBuilder(); + while (true) { + String line = br.readLine(); + if ((line == null || line.startsWith("ExitNode ")) && + fingerprint != null) { + for (String exitAddressLine : exitAddressLines) { + String[] parts = exitAddressLine.split(" "); + String exitAddress = parts[1]; + /* TODO Extend the following code for IPv6 once the exit list + * format supports it. */ + String[] exitAddressParts = exitAddress.split("\\."); + byte[] exitAddress24Bytes = new byte[3]; + exitAddress24Bytes[0] = (byte) Integer.parseInt( + exitAddressParts[0]); + exitAddress24Bytes[1] = (byte) Integer.parseInt( + exitAddressParts[1]); + exitAddress24Bytes[2] = (byte) Integer.parseInt( + exitAddressParts[2]); + String exitAddress24 = Hex.encodeHexString( + exitAddress24Bytes); + String scannedTime = parts[2] + " " + parts[3]; + long scannedMillis = -1L; + try { + scannedMillis = parseFormat.parse(scannedTime).getTime(); + } catch (ParseException e) { + System.out.println("Could not parse timestamp in " + + "'" + file + "'. Skipping."); + return; + } + byte[] rawExitlistentry = rawExitlistentryBuilder.toString(). + getBytes(); + importExitlistentry(fingerprint, exitAddress24, exitAddress, + scannedMillis, rawExitlistentry); + } + exitAddressLines.clear(); + rawExitlistentryBuilder = new StringBuilder(); + } + if (line == null) { + break; + } + rawExitlistentryBuilder.append(line + "\n"); + if (line.startsWith("ExitNode ")) { + fingerprint = line.substring("ExitNode ".length()). + toLowerCase(); + } else if (line.startsWith("ExitAddress ")) { + exitAddressLines.add(line); + } + } + br.close(); + } catch (IOException e) { + System.out.println("Could not parse exit list. Skipping."); + return; + } + } + + /* Import an exit list entry into the database. */ + private static void importExitlistentry(String fingerprint, + String exitAddress24, String exitAddress, long scannedMillis, + byte[] rawExitlistentry) { + try { + insertExitlistentryStatement.clearParameters(); + insertExitlistentryStatement.setString(1, fingerprint); + insertExitlistentryStatement.setString(2, exitAddress24); + insertExitlistentryStatement.setString(3, exitAddress); + insertExitlistentryStatement.setTimestamp(4, + new Timestamp(scannedMillis), calendarUTC); + insertExitlistentryStatement.setBytes(5, rawExitlistentry); + insertExitlistentryStatement.execute(); + } catch (SQLException e) { + System.out.println("Could not import exit list entry. Exiting."); + System.exit(1); + } + } + + /* Write parse history from memory to disk for the next execution. */ + private static void writeImportHistoryToDisk() { + File parseHistoryFile = new File("stats/exonerator-import-history"); + parseHistoryFile.getParentFile().mkdirs(); + try { + BufferedWriter bw = new BufferedWriter(new FileWriter( + parseHistoryFile)); + for (Map.Entry<String, Long> historyEntry : + nextImportHistory.entrySet()) { + bw.write(String.valueOf(historyEntry.getValue()) + "," + + historyEntry.getKey() + "\n"); + } + bw.close(); + } catch (IOException e) { + System.out.println("File 'stats/exonerator-import-history' could " + + "not be written. Ignoring."); + } + } + + /* Close the database connection. */ + private static void closeDatabaseConnection() { + try { + connection.close(); + } catch (SQLException e) { + System.out.println("Could not close database connection. " + + "Ignoring."); + } + } + + /* Delete the exonerator-lock file to allow the next executing of this + * tool. */ + private static void deleteLockFile() { + new File("exonerator-lock").delete(); + } +} + diff --git a/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java b/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java new file mode 100644 index 0000000..68d79a3 --- /dev/null +++ b/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java @@ -0,0 +1,911 @@ +/* Copyright 2011--2015 The Tor Project + * See LICENSE for licensing information */ +package org.torproject.exonerator; + +import java.io.IOException; +import java.io.PrintWriter; +import java.sql.CallableStatement; +import java.sql.Connection; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Calendar; +import java.util.List; +import java.util.Locale; +import java.util.ResourceBundle; +import java.util.SortedSet; +import java.util.TimeZone; +import java.util.TreeSet; +import java.util.logging.Level; +import java.util.logging.Logger; +import java.util.regex.Pattern; + +import javax.naming.Context; +import javax.naming.InitialContext; +import javax.naming.NamingException; +import javax.servlet.ServletException; +import javax.servlet.http.HttpServlet; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; +import javax.sql.DataSource; + +import org.apache.commons.codec.binary.Hex; +import org.apache.commons.lang.StringEscapeUtils; + +public class ExoneraTorServlet extends HttpServlet { + + private static final long serialVersionUID = 1370088989739567509L; + + private DataSource ds; + + private Logger logger; + + public void init() { + + /* Initialize logger. */ + this.logger = Logger.getLogger(ExoneraTorServlet.class.toString()); + + /* Look up data source. */ + try { + Context cxt = new InitialContext(); + this.ds = (DataSource) cxt.lookup("java:comp/env/jdbc/exonerator"); + this.logger.info("Successfully looked up data source."); + } catch (NamingException e) { + this.logger.log(Level.WARNING, "Could not look up data source", e); + } + } + + public void doGet(HttpServletRequest request, + HttpServletResponse response) throws IOException, + ServletException { + + /* Set content type, or the page doesn't render in Chrome. */ + response.setContentType("text/html"); + + /* Start writing response. */ + PrintWriter out = response.getWriter(); + this.writeHeader(out); + + /* Find the right resource bundle for the user's locale. */ + Locale locale = request.getLocale(); + ResourceBundle rb = ResourceBundle.getBundle("ExoneraTor", locale); + + /* Open a database connection that we'll use to handle the whole + * request. */ + long requestedConnection = System.currentTimeMillis(); + Connection conn = this.connectToDatabase(); + if (conn == null) { + this.writeSummaryUnableToConnectToDatabase(out, rb); + this.writeFooter(out, rb); + return; + } + + /* Look up first and last date in the database. */ + long[] firstAndLastDates = this.queryFirstAndLastDatesFromDatabase( + conn); + if (firstAndLastDates == null) { + this.writeSummaryNoData(out, rb); + this.writeFooter(out, rb); + this.closeDatabaseConnection(conn, requestedConnection); + } + SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); + dateFormat.setTimeZone(TimeZone.getTimeZone("UTC")); + String firstDate = dateFormat.format(firstAndLastDates[0]); + String lastDate = dateFormat.format(firstAndLastDates[1]); + + /* Parse parameters. */ + String ipParameter = request.getParameter("ip"); + String relayIP = this.parseIpParameter(ipParameter); + boolean relayIPHasError = relayIP == null; + + /* Parse timestamp parameter. */ + String timestampParameter = request.getParameter("timestamp"); + String timestampStr = this.parseTimestampParameter( + timestampParameter); + boolean timestampHasError = timestampStr == null; + + /* Check that timestamp is within range. */ + long timestamp = 0L; + boolean timestampOutOfRange = false; + if (timestampStr != null && timestampStr.length() > 0) { + try { + timestamp = dateFormat.parse(timestampParameter).getTime(); + if (timestamp < firstAndLastDates[0] || + timestamp > firstAndLastDates[1]) { + timestampOutOfRange = true; + } + } catch (ParseException e) { + /* Already checked in parseTimestamp(). */ + } + } + + /* Write form. */ + this.writeForm(out, rb, relayIP, relayIPHasError || + ("".equals(relayIP) && !"".equals(timestampStr)), timestampStr, + !relayIPHasError && + !("".equals(relayIP) && !"".equals(timestampStr)) && + (timestampHasError || timestampOutOfRange || + (!"".equals(relayIP) && "".equals(timestampStr)))); + + /* If both parameters are empty, don't print any summary and exit. + * This is the start page. */ + if ("".equals(relayIP) && "".equals(timestampStr)) { + this.writeFooter(out, rb); + this.closeDatabaseConnection(conn, requestedConnection); + return; + } + + /* If either parameter is empty, print summary with warning message + * and exit. */ + if ("".equals(relayIP) || "".equals(timestampStr)) { + if ("".equals(relayIP)) { + writeSummaryNoIp(out, rb); + } else { + writeSummaryNoTimestamp(out, rb); + } + this.writeFooter(out, rb); + this.closeDatabaseConnection(conn, requestedConnection); + return; + } + + /* If there's a user error, print summary with exit message and + * exit. */ + if (relayIPHasError || timestampHasError || timestampOutOfRange) { + if (relayIPHasError) { + this.writeSummaryInvalidIp(out, rb, ipParameter); + } else if (timestampHasError) { + this.writeSummaryInvalidTimestamp(out, rb, timestampParameter); + } else if (timestampOutOfRange) { + this.writeSummaryTimestampOutsideRange(out, rb, timestampStr, + firstDate, lastDate); + } + this.writeFooter(out, rb); + this.closeDatabaseConnection(conn, requestedConnection); + return; + } + + /* Consider all consensuses published on or within a day of the given + * date. */ + long timestampFrom = timestamp - 24L * 60L * 60L * 1000L; + long timestampTo = timestamp + 2 * 24L * 60L * 60L * 1000L - 1L; + SimpleDateFormat validAfterTimeFormat = new SimpleDateFormat( + "yyyy-MM-dd HH:mm:ss"); + validAfterTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); + String fromValidAfter = validAfterTimeFormat.format(timestampFrom); + String toValidAfter = validAfterTimeFormat.format(timestampTo); + SortedSet<Long> relevantConsensuses = + this.queryKnownConsensusValidAfterTimes(conn, fromValidAfter, + toValidAfter); + if (relevantConsensuses == null || relevantConsensuses.isEmpty()) { + this.writeSummaryNoDataForThisInterval(out, rb); + this.writeFooter(out, rb); + this.closeDatabaseConnection(conn, requestedConnection); + return; + } + + /* Search for status entries with the given IP address as onion + * routing address, plus status entries of relays having an exit list + * entry with the given IP address as exit address. */ + List<String[]> statusEntries = this.queryStatusEntries(conn, relayIP, + timestamp, validAfterTimeFormat); + + /* If we didn't find anything, run another query to find out if there + * are relays running on other IP addresses in the same /24 or /48 + * network and tell the user about it. */ + List<String> addressesInSameNetwork = null; + if (statusEntries.isEmpty()) { + addressesInSameNetwork = new ArrayList<String>(); + if (!relayIP.contains(":")) { + String address24 = this.convertIPv4ToHex(relayIP).substring(0, 6); + if (address24 != null) { + addressesInSameNetwork = this.queryAddressesInSame24(conn, + address24, timestamp); + } + } else { + String address48 = this.convertIPv6ToHex(relayIP).substring( + 0, 12); + if (address48 != null) { + addressesInSameNetwork = this.queryAddressesInSame48(conn, + address48, timestamp); + } + } + } + + /* Print out result. */ + if (!statusEntries.isEmpty()) { + this.writeSummaryPositive(out, rb, relayIP, timestampStr); + this.writeTechnicalDetails(out, rb, relayIP, timestampStr, + statusEntries); + } else if (addressesInSameNetwork != null && + !addressesInSameNetwork.isEmpty()) { + this.writeSummaryAddressesInSameNetwork(out, rb, relayIP, + timestampStr, addressesInSameNetwork); + } else { + this.writeSummaryNegative(out, rb, relayIP, timestampStr); + } + + this.writePermanentLink(out, rb, relayIP, timestampStr); + + this.closeDatabaseConnection(conn, requestedConnection); + this.writeFooter(out, rb); + } + + /* Helper methods for handling the request. */ + + private String parseIpParameter(String passedIpParameter) { + String relayIP = null; + if (passedIpParameter != null && passedIpParameter.length() > 0) { + String ipParameter = passedIpParameter.trim(); + Pattern ipv4AddressPattern = Pattern.compile( + "^([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + + "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + + "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + + "([01]?\\d\\d?|2[0-4]\\d|25[0-5])$"); + Pattern ipv6AddressPattern = Pattern.compile( + "^\\[?[0-9a-fA-F:]{3,39}\\]?$"); + if (ipv4AddressPattern.matcher(ipParameter).matches()) { + String[] ipParts = ipParameter.split("\\."); + relayIP = Integer.parseInt(ipParts[0]) + "." + + Integer.parseInt(ipParts[1]) + "." + + Integer.parseInt(ipParts[2]) + "." + + Integer.parseInt(ipParts[3]); + } else if (ipv6AddressPattern.matcher(ipParameter).matches()) { + if (ipParameter.startsWith("[") && ipParameter.endsWith("]")) { + ipParameter = ipParameter.substring(1, + ipParameter.length() - 1); + } + StringBuilder addressHex = new StringBuilder(); + int start = ipParameter.startsWith("::") ? 1 : 0; + int end = ipParameter.length() + - (ipParameter.endsWith("::") ? 1 : 0); + String[] parts = ipParameter.substring(start, end).split(":", -1); + for (int i = 0; i < parts.length; i++) { + String part = parts[i]; + if (part.length() == 0) { + addressHex.append("x"); + } else if (part.length() <= 4) { + addressHex.append(String.format("%4s", part)); + } else { + addressHex = null; + break; + } + } + if (addressHex != null) { + String addressHexString = addressHex.toString(); + addressHexString = addressHexString.replaceFirst("x", + String.format("%" + (33 - addressHexString.length()) + "s", + "0")); + if (!addressHexString.contains("x") && + addressHexString.length() == 32) { + relayIP = ipParameter.toLowerCase(); + } + } + } + } else { + relayIP = ""; + } + return relayIP; + } + + private String convertIPv4ToHex(String relayIP) { + String[] relayIPParts = relayIP.split("\\."); + byte[] address24Bytes = new byte[4]; + for (int i = 0; i < address24Bytes.length; i++) { + address24Bytes[i] = (byte) Integer.parseInt(relayIPParts[i]); + } + String address24 = Hex.encodeHexString(address24Bytes); + return address24; + } + + private String convertIPv6ToHex(String relayIP) { + if (relayIP.startsWith("[") && relayIP.endsWith("]")) { + relayIP = relayIP.substring(1, relayIP.length() - 1); + } + StringBuilder addressHex = new StringBuilder(); + int start = relayIP.startsWith("::") ? 1 : 0; + int end = relayIP.length() - (relayIP.endsWith("::") ? 1 : 0); + String[] parts = relayIP.substring(start, end).split(":", -1); + for (int i = 0; i < parts.length; i++) { + String part = parts[i]; + if (part.length() == 0) { + addressHex.append("x"); + } else if (part.length() <= 4) { + addressHex.append(String.format("%4s", part)); + } else { + addressHex = null; + break; + } + } + String address48 = null; + if (addressHex != null) { + String addressHexString = addressHex.toString(); + addressHexString = addressHexString.replaceFirst("x", + String.format("%" + (33 - addressHexString.length()) + + "s", "0")); + if (!addressHexString.contains("x") && + addressHexString.length() == 32) { + address48 = addressHexString.replaceAll(" ", "0"). + toLowerCase(); + } + } + return address48; + } + + private String parseTimestampParameter( + String passedTimestampParameter) { + String timestampStr = ""; + SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); + dateFormat.setTimeZone(TimeZone.getTimeZone("UTC")); + dateFormat.setLenient(false); + if (passedTimestampParameter != null && + passedTimestampParameter.length() > 0) { + String timestampParameter = passedTimestampParameter.trim(); + try { + long timestamp = dateFormat.parse(timestampParameter).getTime(); + timestampStr = dateFormat.format(timestamp); + } catch (ParseException e) { + timestampStr = null; + } + } + return timestampStr; + } + + /* Helper methods for querying the database. */ + + private Connection connectToDatabase() { + Connection conn = null; + try { + conn = this.ds.getConnection(); + } catch (SQLException e) { + } + return conn; + } + + private long[] queryFirstAndLastDatesFromDatabase(Connection conn) { + long[] firstAndLastDates = null; + try { + Statement statement = conn.createStatement(); + String query = "SELECT DATE(MIN(validafter)) AS first, " + + "DATE(MAX(validafter)) AS last FROM statusentry"; + ResultSet rs = statement.executeQuery(query); + if (rs.next()) { + Calendar utcCalendar = Calendar.getInstance( + TimeZone.getTimeZone("UTC")); + firstAndLastDates = new long[] { + rs.getTimestamp(1, utcCalendar).getTime(), + rs.getTimestamp(2, utcCalendar).getTime() + }; + } + rs.close(); + statement.close(); + } catch (SQLException e) { + /* Looks like we don't have any consensuses. */ + firstAndLastDates = null; + } + return firstAndLastDates; + } + + private SortedSet<Long> queryKnownConsensusValidAfterTimes( + Connection conn, String fromValidAfter, String toValidAfter) { + SortedSet<Long> relevantConsensuses = new TreeSet<Long>(); + try { + Statement statement = conn.createStatement(); + String query = "SELECT DISTINCT validafter FROM statusentry " + + "WHERE validafter >= '" + fromValidAfter + + "' AND validafter <= '" + toValidAfter + "'"; + ResultSet rs = statement.executeQuery(query); + while (rs.next()) { + long consensusTime = rs.getTimestamp(1).getTime(); + relevantConsensuses.add(consensusTime); + } + rs.close(); + statement.close(); + } catch (SQLException e) { + /* Looks like we don't have any consensuses in the requested + * interval. */ + relevantConsensuses = null; + } + return relevantConsensuses; + } + + private List<String[]> queryStatusEntries(Connection conn, + String relayIP, long timestamp, + SimpleDateFormat validAfterTimeFormat) { + List<String[]> statusEntries = new ArrayList<String[]>(); + String addressHex = !relayIP.contains(":") + ? this.convertIPv4ToHex(relayIP) : this.convertIPv6ToHex(relayIP); + if (addressHex == null) { + return null; + } + String address24Or48Hex = !relayIP.contains(":") + ? addressHex.substring(0, 6) : addressHex.substring(0, 12); + try { + CallableStatement cs; + if (!relayIP.contains(":")) { + cs = conn.prepareCall("{call search_by_address24_date(?, ?)}"); + } else { + cs = conn.prepareCall("{call search_by_address48_date(?, ?)}"); + } + cs.setString(1, address24Or48Hex); + Calendar utcCalendar = Calendar.getInstance( + TimeZone.getTimeZone("UTC")); + cs.setDate(2, new java.sql.Date(timestamp), utcCalendar); + ResultSet rs = cs.executeQuery(); + while (rs.next()) { + byte[] rawstatusentry = rs.getBytes(1); + SortedSet<String> addresses = new TreeSet<String>(), + addressesHex = new TreeSet<String>(); + long validafter = rs.getTimestamp(2, utcCalendar).getTime(); + String validAfterString = validAfterTimeFormat.format(validafter); + String fingerprint = rs.getString(3).toUpperCase(); + String nickname = null; + String exit = "U"; + for (String line : new String(rawstatusentry).split("\n")) { + if (line.startsWith("r ")) { + String[] parts = line.split(" "); + nickname = parts[1]; + addresses.add(parts[6]); + addressesHex.add(this.convertIPv4ToHex(parts[6])); + } else if (line.startsWith("a ")) { + String address = line.substring("a ".length(), + line.lastIndexOf(":")); + addresses.add(address); + String orAddressHex = !address.contains(":") + ? this.convertIPv4ToHex(address) + : this.convertIPv6ToHex(address); + addressesHex.add(orAddressHex); + } else if (line.startsWith("p ")) { + exit = line.equals("p reject 1-65535") ? "N" : "Y"; + } + } + String exitaddress = rs.getString(4); + if (exitaddress != null && exitaddress.length() > 0) { + addresses.add(exitaddress); + addressesHex.add(this.convertIPv4ToHex(exitaddress)); + } + if (!addressesHex.contains(addressHex)) { + continue; + } + StringBuilder sb = new StringBuilder(); + int writtenAddresses = 0; + for (String address : addresses) { + sb.append((writtenAddresses++ > 0 ? ", " : "") + address); + } + String[] statusEntry = new String[] { validAfterString, + sb.toString(), fingerprint, nickname, exit }; + statusEntries.add(statusEntry); + } + rs.close(); + cs.close(); + } catch (SQLException e) { + /* Nothing found. */ + statusEntries = null; + } + return statusEntries; + } + + private List<String> queryAddressesInSame24(Connection conn, + String address24, long timestamp) { + List<String> addressesInSameNetwork = new ArrayList<String>(); + try { + CallableStatement cs = conn.prepareCall( + "{call search_addresses_in_same_24 (?, ?)}"); + cs.setString(1, address24); + cs.setDate(2, new java.sql.Date(timestamp)); + ResultSet rs = cs.executeQuery(); + while (rs.next()) { + String address = rs.getString(1); + if (!addressesInSameNetwork.contains(address)) { + addressesInSameNetwork.add(address); + } + } + rs.close(); + cs.close(); + } catch (SQLException e) { + /* No other addresses in the same /24 found. */ + addressesInSameNetwork = null; + } + return addressesInSameNetwork; + } + + private List<String> queryAddressesInSame48(Connection conn, + String address48, long timestamp) { + List<String> addressesInSameNetwork = new ArrayList<String>(); + try { + CallableStatement cs = conn.prepareCall( + "{call search_addresses_in_same_48 (?, ?)}"); + cs.setString(1, address48); + cs.setDate(2, new java.sql.Date(timestamp)); + ResultSet rs = cs.executeQuery(); + while (rs.next()) { + String address = rs.getString(1); + if (!addressesInSameNetwork.contains(address)) { + addressesInSameNetwork.add(address); + } + } + rs.close(); + cs.close(); + } catch (SQLException e) { + /* No other addresses in the same /48 found. */ + addressesInSameNetwork = null; + } + return addressesInSameNetwork; + } + + private void closeDatabaseConnection(Connection conn, + long requestedConnection) { + try { + conn.close(); + this.logger.info("Returned a database connection to the pool " + + "after " + (System.currentTimeMillis() + - requestedConnection) + " millis."); + } catch (SQLException e) { + } + return; + } + + /* Helper methods for writing the response. */ + + private void writeHeader(PrintWriter out) throws IOException { + out.println("<!DOCTYPE html>\n" + + "<html lang=\"en\">\n" + + " <head>\n" + + " <meta charset=\"utf-8\">\n" + + " <meta http-equiv=\"X-UA-Compatible\" " + + "content=\"IE=edge\">\n" + + " <meta name=\"viewport\" content=\"width=device-width, " + + "initial-scale=1\">\n" + + " <title>ExoneraTor</title>\n" + + " <link rel=\"stylesheet\" href=\"css/bootstrap.min.css\">\n" + + " <link rel=\"stylesheet\" href=\"css/exonerator.css\">\n" + + " <link href=\"images/favicon.ico\" type=\"image/x-icon\" " + + "rel=\"icon\">\n" + + " </head>\n" + + " <body>\n" + + " <div class=\"container\">\n" + + " <div class=\"row\">\n" + + " <div class=\"col-xs-12\">\n" + + " <div class=\"page-header\">\n" + + " <h1>\n" + + " <div class=\"text-center\">\n" + + " <a href=\"/\">" + + "<img src=\"images/exonerator-logo.png\" " + + "width=\"334\" height=\"252\" alt=\"ExoneraTor logo\">" + + "<img src=\"images/exonerator-wordmark.png\" width=\"428\" " + + "height=\"63\" alt=\"ExoneraTor wordmark\"></a>\n" + + " </div>\n" + + " </h1>\n" + + " </div>\n" + + " </div>\n" + + " </div>\n"); + } + + private void writeForm(PrintWriter out, ResourceBundle rb, + String relayIP, boolean relayIPHasError, String timestampStr, + boolean timestampHasError) throws IOException { + String ipValue = ""; + if (relayIP != null && relayIP.length() > 0) { + if (relayIP.contains(":")) { + ipValue = String.format(" value=\"[%s]\"", relayIP); + } else { + ipValue = String.format(" value=\"%s\"", relayIP); + } + } + out.printf(" <div class=\"row\">\n" + + " <div class=\"col-xs-12\">\n" + + " <div class=\"text-center\">\n" + + " <div class=\"row vbottom15\">\n" + + " <h4>%s</h4>\n" + + " </div> \n" + + " <form class=\"form-inline\">\n" + + " <div class=\"form-group%s\">\n" + + " <label for=\"inputIp\" " + + "class=\"control-label\">%s</label>\n" + + " <input type=\"text\" class=\"form-control\" " + + "name=\"ip\" id=\"inputIp\" placeholder=\"86.59.21.38\"%s " + + "required>\n" + + " </div>\n" + + " <div class=\"form-group%s\">\n" + + " <label for=\"inputTimestamp\" " + + "class=\"control-label\">%s</label>\n" + + " <input type=\"date\" class=\"form-control\" " + + "name=\"timestamp\" id=\"inputTimestamp\" " + + "placeholder=\"2010-01-01\"%s required>\n" + + " </div>\n" + + " <button type=\"submit\" " + + "class=\"btn btn-primary\">%s</button>\n" + + " </form>\n" + + " </div>\n" + + " </div>\n" + + " </div>\n", + rb.getString("form.explanation"), + relayIPHasError ? " has-error" : "", + rb.getString("form.ip.label"), + ipValue, + timestampHasError ? " has-error" : "", + rb.getString("form.timestamp.label"), + timestampStr != null && timestampStr.length() > 0 ? + " value=\"" + timestampStr + "\"" : "", + rb.getString("form.search.label")); + } + + private void writeSummaryUnableToConnectToDatabase(PrintWriter out, + ResourceBundle rb) throws IOException { + String contactLink = + "<a href=\"https://www.torproject.org/about/contact\">" + + rb.getString("summary.serverproblem.dbempty.body.link") + + "</a>"; + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.serverproblem.dbnoconnect.title"), null, + rb.getString("summary.serverproblem.dbnoconnect.body.text"), + contactLink); + } + + private void writeSummaryNoData(PrintWriter out, ResourceBundle rb) + throws IOException { + String contactLink = + "<a href=\"https://www.torproject.org/about/contact\">" + + rb.getString("summary.serverproblem.dbempty.body.link") + + "</a>"; + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.serverproblem.dbempty.title"), null, + rb.getString("summary.serverproblem.dbempty.body.text"), + contactLink); + } + + private void writeSummaryNoTimestamp(PrintWriter out, ResourceBundle rb) + throws IOException { + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.invalidparams.notimestamp.title"), null, + rb.getString("summary.invalidparams.notimestamp.body")); + } + + private void writeSummaryNoIp(PrintWriter out, ResourceBundle rb) + throws IOException { + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", rb.getString("summary.invalidparams.noip.title"), + null, rb.getString("summary.invalidparams.noip.body")); + } + + private void writeSummaryTimestampOutsideRange(PrintWriter out, + ResourceBundle rb, String timestampStr, String firstDate, + String lastDate) throws IOException { + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.invalidparams.timestamprange.title"), null, + rb.getString("summary.invalidparams.timestamprange.body"), + timestampStr, firstDate, lastDate); + } + + private void writeSummaryInvalidIp(PrintWriter out, ResourceBundle rb, + String ipParameter) throws IOException { + String escapedIpParameter = ipParameter.length() > 40 ? + StringEscapeUtils.escapeHtml(ipParameter.substring(0, 40)) + + "[...]" : StringEscapeUtils.escapeHtml(ipParameter); + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.invalidparams.invalidip.title"), null, + rb.getString("summary.invalidparams.invalidip.body"), + escapedIpParameter, "\"a.b.c.d\"", "\"[a:b:c:d:e:f:g:h]\""); + } + + private void writeSummaryInvalidTimestamp(PrintWriter out, + ResourceBundle rb, String timestampParameter) throws IOException { + String escapedTimestampParameter = timestampParameter.length() > 20 ? + StringEscapeUtils.escapeHtml(timestampParameter. + substring(0, 20)) + "[...]" : + StringEscapeUtils.escapeHtml(timestampParameter); + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.invalidparams.invalidtimestamp.title"), + null, rb.getString("summary.invalidparams.invalidtimestamp.body"), + escapedTimestampParameter, "\"YYYY-MM-DD\""); + } + + private void writeSummaryNoDataForThisInterval(PrintWriter out, + ResourceBundle rb) throws IOException { + String contactLink = + "<a href=\"https://www.torproject.org/about/contact\">" + + rb.getString("summary.serverproblem.dbempty.body.link") + + "</a>"; + this.writeSummary(out, rb.getString("summary.heading"), + "panel-danger", + rb.getString("summary.serverproblem.nodata.title"), null, + rb.getString("summary.serverproblem.nodata.body.text"), + contactLink); + } + + private void writeSummaryAddressesInSameNetwork(PrintWriter out, + ResourceBundle rb, String relayIP, String timestampStr, + List<String> addressesInSameNetwork) throws IOException { + Object[][] panelItems = new Object[addressesInSameNetwork.size()][]; + for (int i = 0; i < addressesInSameNetwork.size(); i++) { + String addressInSameNetwork = addressesInSameNetwork.get(i); + String link, address; + if (addressInSameNetwork.contains(":")) { + link = String.format("/?ip=[%s]&timestamp=%s", + addressInSameNetwork.replaceAll(":", "%3A"), timestampStr); + address = "[" + addressInSameNetwork + "]"; + } else { + link = String.format("/?ip=%s&timestamp=%s", + addressInSameNetwork, timestampStr); + address = addressInSameNetwork; + } + panelItems[i] = new Object[] { link, address }; + } + this.writeSummary(out, rb.getString("summary.heading"), + "panel-warning", + rb.getString("summary.negativesamenetwork.title"), panelItems, + rb.getString("summary.negativesamenetwork.body"), + relayIP, timestampStr, relayIP.contains(":") ? 48 : 24); + } + + private void writeSummaryPositive(PrintWriter out, ResourceBundle rb, + String relayIP, String timestampStr) throws IOException { + String formattedRelayIP = relayIP.contains(":") ? + "[" + relayIP + "]" : relayIP; + this.writeSummary(out, rb.getString("summary.heading"), + "panel-success", rb.getString("summary.positive.title"), null, + rb.getString("summary.positive.body"), formattedRelayIP, + timestampStr); + } + + private void writeSummaryNegative(PrintWriter out, ResourceBundle rb, + String relayIP, String timestampStr) throws IOException { + String formattedRelayIP = relayIP.contains(":") ? + "[" + relayIP + "]" : relayIP; + this.writeSummary(out, rb.getString("summary.heading"), + "panel-warning", rb.getString("summary.negative.title"), null, + rb.getString("summary.negative.body"), formattedRelayIP, + timestampStr); + } + + private void writeSummary(PrintWriter out, String heading, + String panelContext, String panelTitle, Object[][] panelItems, + String panelBodyTemplate, Object... panelBodyArgs) + throws IOException { + out.printf(" <div class=\"row\">\n" + + " <div class=\"col-xs-12\">\n" + + " <h2>%s</h2>\n" + + " <div class=\"panel %s\">\n" + + " <div class=\"panel-heading\">\n" + + " <h3 class=\"panel-title\">%s</h3>\n" + + " </div>\n" + + " <div class=\"panel-body\">\n" + + " <p>%s</p>\n", heading, panelContext, panelTitle, + String.format(panelBodyTemplate, panelBodyArgs)); + if (panelItems != null) { + out.print(" <ul>\n"); + for (Object[] panelItem : panelItems) { + out.printf(" <li><a href=\"%s\">%s</a></li>\n", + panelItem); + } + out.print(" </ul>\n"); + } + out.print(" </div>\n" + + " </div>\n" + + " </div>\n" + + " </div>\n"); + } + + private void writeTechnicalDetails(PrintWriter out, ResourceBundle rb, + String relayIP, String timestampStr, List<String[]> tableRows) + throws IOException { + String formattedRelayIP = relayIP.contains(":") ? + "[" + relayIP + "]" : relayIP; + out.printf(" <div class=\"row\">\n" + + " <div class=\"col-xs-12\">\n" + + " <h2>%s</h2>\n" + + " <p>%s</p>\n" + + " <table class=\"table\">\n" + + " <thead>\n" + + " <tr>\n" + + " <th>%s</th>\n" + + " <th>%s</th>\n" + + " <th>%s</th>\n" + + " <th>%s</th>\n" + + " <th>%s</th>\n" + + " </tr>\n" + + " </thead>\n" + + " <tbody>\n", + rb.getString("technicaldetails.heading"), + String.format(rb.getString("technicaldetails.pre"), + formattedRelayIP, timestampStr), + rb.getString("technicaldetails.colheader.timestamp"), + rb.getString("technicaldetails.colheader.ip"), + rb.getString("technicaldetails.colheader.fingerprint"), + rb.getString("technicaldetails.colheader.nickname"), + rb.getString("technicaldetails.colheader.exit")); + for (String[] tableRow : tableRows) { + out.print(" <tr>"); + for (int i = 0; i < tableRow.length; i++) { + String content = tableRow[i]; + if (i == 2) { + content = content.substring(0, 20) + "" + + content.substring(20, 40); + } else if (i == 3 && content == null) { + content = "(" + + rb.getString("technicaldetails.nickname.unknown") + ")"; + } else if (i == 4) { + if (content.equals("U")) { + content = rb.getString("technicaldetails.exit.unknown"); + } else if (content.equals("Y")) { + content = rb.getString("technicaldetails.exit.yes"); + } else { + content = rb.getString("technicaldetails.exit.no"); + } + } + out.print(" <td>" + content + "</td>"); + } + out.print(" </tr>\n"); + } + out.print(" </tbody>\n" + + " </table>\n" + + " </div>\n" + + " </div>\n"); + } + + private void writePermanentLink(PrintWriter out, ResourceBundle rb, + String relayIP, String timestampStr) throws IOException { + String encodedAddress = relayIP.contains(":") ? + "[" + relayIP.replaceAll(":", "%3A") + "]" : relayIP; + out.printf(" <div class=\"row\">\n" + + " <div class=\"col-xs-12\">\n" + + " <h2>%s</h2>\n" + + " <pre>https://exonerator.torproject.org/?ip=%s&" + + "timestamp=%s</pre>\n" + + " </div>\n" + + " </div>\n", + rb.getString("permanentlink.heading"), + encodedAddress, timestampStr); + } + + private void writeFooter(PrintWriter out, ResourceBundle rb) + throws IOException { + out.printf(" </div>\n" + + " <div class=\"footer\">\n" + + " <div class=\"container\">\n" + + " <div class=\"row\">\n" + + " <div class=\"col-xs-6\">\n" + + " <h3>%s</h3>\n" + + " <p class=\"small\">%s</p>\n" + + " </div>\n", + rb.getString("footer.abouttor.heading"), + String.format(rb.getString("footer.abouttor.body.text"), + "<a href=\"https://www.torproject.org/about/" + + "overview#thesolution\">" + + rb.getString("footer.abouttor.body.link1") + "</a>", + "<a href=\"https://www.torproject.org/about/overview\">" + + rb.getString("footer.abouttor.body.link2") + "</a>", + "<a href=\"https://www.torproject.org/about/contact\">" + + rb.getString("footer.abouttor.body.link3") + "</a>")); + out.printf(" <div class=\"col-xs-6\">\n" + + " <h3>%s</h3>\n" + + " <p class=\"small\">%s</p>\n" + + " </div>\n" + + " </div>\n" + + " <div class=\"row\">\n", + rb.getString("footer.aboutexonerator.heading"), + rb.getString("footer.aboutexonerator.body")); + out.printf(" <div class=\"col-xs-12\">\n" + + " <p class=\"text-center small\">%s</p>\n" + + " </div>\n" + + " </div>\n" + + " </div>\n" + + " </div>\n" + + " </body>\n" + + "</html>\n", + String.format(rb.getString("footer.trademark.text"), + "<a href=\"https://www.torproject.org/docs/" + + "trademark-faq.html.en\">" + + rb.getString("footer.trademark.link") + "</a>")); + out.close(); + } +} + diff --git a/src/org/torproject/exonerator/ExoneraTorDatabaseImporter.java b/src/org/torproject/exonerator/ExoneraTorDatabaseImporter.java deleted file mode 100644 index 41751ca..0000000 --- a/src/org/torproject/exonerator/ExoneraTorDatabaseImporter.java +++ /dev/null @@ -1,564 +0,0 @@ -/* Copyright 2011, 2012 The Tor Project - * See LICENSE for licensing information */ -package org.torproject.exonerator; - -import java.io.BufferedInputStream; -import java.io.BufferedReader; -import java.io.BufferedWriter; -import java.io.ByteArrayOutputStream; -import java.io.File; -import java.io.FileInputStream; -import java.io.FileReader; -import java.io.FileWriter; -import java.io.IOException; -import java.io.StringReader; -import java.io.UnsupportedEncodingException; -import java.sql.CallableStatement; -import java.sql.Connection; -import java.sql.DriverManager; -import java.sql.SQLException; -import java.sql.Timestamp; -import java.sql.Types; -import java.text.ParseException; -import java.text.SimpleDateFormat; -import java.util.Calendar; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Map; -import java.util.Set; -import java.util.Stack; -import java.util.TimeZone; - -import org.apache.commons.codec.binary.Base64; -import org.apache.commons.codec.binary.Hex; -import org.apache.commons.codec.digest.DigestUtils; -import org.torproject.descriptor.DescriptorCollector; -import org.torproject.descriptor.DescriptorSourceFactory; - -/* Import Tor descriptors into the ExoneraTor database. */ -public class ExoneraTorDatabaseImporter { - - /* Main function controlling the parsing process. */ - public static void main(String[] args) { - readConfiguration(); - openDatabaseConnection(); - prepareDatabaseStatements(); - createLockFile(); - fetchDescriptors(); - readImportHistoryToMemory(); - parseDescriptors(); - writeImportHistoryToDisk(); - closeDatabaseConnection(); - deleteLockFile(); - } - - /* JDBC string of the ExoneraTor database. */ - private static String jdbcString; - - /* Directory from which to import descriptors. */ - private static String importDirString; - - /* Learn JDBC string and directory to parse descriptors from. */ - private static void readConfiguration() { - File configFile = new File("config"); - if (!configFile.exists()) { - System.err.println("Could not find config file. Exiting."); - System.exit(1); - } - String line = null; - try { - BufferedReader br = new BufferedReader(new FileReader(configFile)); - while ((line = br.readLine()) != null) { - if (line.startsWith("#") || line.length() < 1) { - continue; - } else if (line.startsWith("ExoneraTorDatabaseJdbc")) { - jdbcString = line.split(" ")[1]; - } else if (line.startsWith("ExoneraTorImportDirectory")) { - importDirString = line.split(" ")[1]; - } else { - /* Ignore unrecognized configuration keys. */ - } - } - br.close(); - } catch (IOException e) { - System.err.println("Could not parse config file. Exiting."); - System.exit(1); - } - } - - /* Database connection. */ - private static Connection connection; - - /* Open a database connection using the JDBC string in the config. */ - private static void openDatabaseConnection() { - try { - connection = DriverManager.getConnection(jdbcString); - } catch (SQLException e) { - System.out.println("Could not connect to database. Exiting."); - System.exit(1); - } - } - - /* Callable statements to import data into the database. */ - private static CallableStatement insertStatusentryStatement; - private static CallableStatement insertExitlistentryStatement; - - /* Prepare statements for importing data into the database. */ - private static void prepareDatabaseStatements() { - try { - insertStatusentryStatement = connection.prepareCall( - "{call insert_statusentry(?, ?, ?, ?, ?, ?, ?)}"); - insertExitlistentryStatement = connection.prepareCall( - "{call insert_exitlistentry(?, ?, ?, ?, ?)}"); - } catch (SQLException e) { - System.out.println("Could not prepare callable statements to " - + "import data into the database. Exiting."); - System.exit(1); - } - } - - /* Create a local lock file to prevent other instances of this import - * tool to run concurrently. */ - private static void createLockFile() { - File lockFile = new File("exonerator-lock"); - try { - if (lockFile.exists()) { - BufferedReader br = new BufferedReader(new FileReader(lockFile)); - long runStarted = Long.parseLong(br.readLine()); - br.close(); - if (System.currentTimeMillis() - runStarted - < 6L * 60L * 60L * 1000L) { - System.out.println("File 'exonerator-lock' is less than 6 " - + "hours old. Exiting."); - System.exit(1); - } else { - System.out.println("File 'exonerator-lock' is at least 6 hours " - + "old. Overwriting and executing anyway."); - } - } - BufferedWriter bw = new BufferedWriter(new FileWriter( - "exonerator-lock")); - bw.append(String.valueOf(System.currentTimeMillis()) + "\n"); - bw.close(); - } catch (IOException e) { - System.out.println("Could not create 'exonerator-lock' file. " - + "Exiting."); - System.exit(1); - } - } - - /* Fetch recent descriptors from CollecTor. */ - private static void fetchDescriptors() { - DescriptorCollector collector = - DescriptorSourceFactory.createDescriptorCollector(); - collector.collectDescriptors("https://collector.torproject.org", - new String[] { "/recent/relay-descriptors/consensuses/", - "/recent/exit-lists/" }, 0L, new File(importDirString), true); - } - - /* Last and next parse histories containing paths of parsed files and - * last modified times. */ - private static Map<String, Long> - lastImportHistory = new HashMap<String, Long>(), - nextImportHistory = new HashMap<String, Long>(); - - /* Read stats/exonerator-import-history file from disk and remember - * locally when files were last parsed. */ - private static void readImportHistoryToMemory() { - File parseHistoryFile = new File("stats", - "exonerator-import-history"); - if (parseHistoryFile.exists()) { - try { - BufferedReader br = new BufferedReader(new FileReader( - parseHistoryFile)); - String line = null; - int lineNumber = 0; - while ((line = br.readLine()) != null) { - lineNumber++; - String[] parts = line.split(","); - if (parts.length != 2) { - System.out.println("File 'stats/exonerator-import-history' " - + "contains a corrupt entry in line " + lineNumber - + ". Ignoring parse history file entirely."); - lastImportHistory.clear(); - br.close(); - return; - } - long lastModified = Long.parseLong(parts[0]); - String filename = parts[1]; - lastImportHistory.put(filename, lastModified); - } - br.close(); - } catch (IOException e) { - System.out.println("Could not read import history. Ignoring."); - lastImportHistory.clear(); - } - } - } - - /* Parse descriptors in the import directory and its subdirectories. */ - private static void parseDescriptors() { - File file = new File(importDirString); - if (!file.exists()) { - System.out.println("File or directory " + importDirString + " does " - + "not exist. Exiting."); - return; - } - Stack<File> files = new Stack<File>(); - files.add(file); - while (!files.isEmpty()) { - file = files.pop(); - if (file.isDirectory()) { - for (File f : file.listFiles()) { - files.add(f); - } - } else { - parseFile(file); - } - } - } - - /* Import a file if it wasn't imported before, and add it to the import - * history for the next execution. */ - private static void parseFile(File file) { - long lastModified = file.lastModified(); - String filename = file.getName(); - nextImportHistory.put(filename, lastModified); - if (!lastImportHistory.containsKey(filename) || - lastImportHistory.get(filename) < lastModified) { - try { - FileInputStream fis = new FileInputStream(file); - BufferedInputStream bis = new BufferedInputStream(fis); - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - int len; - byte[] bytes = new byte[1024]; - while ((len = bis.read(bytes, 0, 1024)) >= 0) { - baos.write(bytes, 0, len); - } - bis.close(); - byte[] allBytes = baos.toByteArray(); - splitFile(file, allBytes); - } catch (IOException e) { - System.out.println("Could not read '" + file + "' to memory. " - + "Skipping."); - nextImportHistory.remove(filename); - } - } - } - - /* Detect what descriptor type is contained in a file and split it to - * parse the single descriptors. */ - private static void splitFile(File file, byte[] bytes) { - try { - String asciiString = new String(bytes, "US-ASCII"); - BufferedReader br = new BufferedReader(new StringReader( - asciiString)); - String line = br.readLine(); - while (line != null && line.startsWith("@")) { - line = br.readLine(); - } - if (line == null) { - return; - } - br.close(); - String startToken = null; - if (line.equals("network-status-version 3")) { - startToken = "network-status-version 3"; - } else if (line.startsWith("Downloaded ") || - line.startsWith("ExitNode ")) { - startToken = "ExitNode "; - } else { - System.out.println("Unknown descriptor type in file '" + file - + "'. Ignoring."); - return; - } - String splitToken = "\n" + startToken; - int length = bytes.length, start = asciiString.indexOf(startToken); - while (start < length) { - int end = asciiString.indexOf(splitToken, start); - if (end < 0) { - end = length; - } else { - end += 1; - } - byte[] descBytes = new byte[end - start]; - System.arraycopy(bytes, start, descBytes, 0, end - start); - if (startToken.equals("network-status-version 3")) { - parseConsensus(file, descBytes); - } else if (startToken.equals("ExitNode ")) { - parseExitList(file, descBytes); - } - start = end; - } - } catch (IOException e) { - System.out.println("Could not parse descriptor '" + file + "'. " - + "Skipping."); - } - } - - /* Date format to parse UTC timestamps. */ - private static SimpleDateFormat parseFormat; - static { - parseFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); - parseFormat.setTimeZone(TimeZone.getTimeZone("UTC")); - } - - /* Parse a consensus. */ - private static void parseConsensus(File file, byte[] bytes) { - try { - BufferedReader br = new BufferedReader(new StringReader(new String( - bytes, "US-ASCII"))); - String line, fingerprint = null, descriptor = null; - Set<String> orAddresses = new HashSet<String>(); - long validAfterMillis = -1L; - StringBuilder rawStatusentryBuilder = null; - boolean isRunning = false; - while ((line = br.readLine()) != null) { - if (line.startsWith("vote-status ") && - !line.equals("vote-status consensus")) { - System.out.println("File '" + file + "' contains network status " - + "*votes*, not network status *consensuses*. Skipping."); - return; - } else if (line.startsWith("valid-after ")) { - String validAfterTime = line.substring("valid-after ".length()); - try { - validAfterMillis = parseFormat.parse(validAfterTime). - getTime(); - } catch (ParseException e) { - System.out.println("Could not parse valid-after timestamp in " - + "'" + file + "'. Skipping."); - return; - } - } else if (line.startsWith("r ") || - line.equals("directory-footer")) { - if (isRunning) { - byte[] rawStatusentry = rawStatusentryBuilder.toString(). - getBytes(); - importStatusentry(validAfterMillis, fingerprint, descriptor, - orAddresses, rawStatusentry); - orAddresses = new HashSet<String>(); - } - if (line.equals("directory-footer")) { - return; - } - rawStatusentryBuilder = new StringBuilder(line + "\n"); - String[] parts = line.split(" "); - if (parts.length < 9) { - System.out.println("Could not parse r line '" + line - + "'. Skipping."); - return; - } - fingerprint = Hex.encodeHexString(Base64.decodeBase64(parts[2] - + "=")).toLowerCase(); - descriptor = Hex.encodeHexString(Base64.decodeBase64(parts[3] - + "=")).toLowerCase(); - orAddresses.add(parts[6]); - } else if (line.startsWith("a ")) { - rawStatusentryBuilder.append(line + "\n"); - orAddresses.add(line.substring("a ".length(), - line.lastIndexOf(":"))); - } else if (line.startsWith("s ") || line.equals("s")) { - rawStatusentryBuilder.append(line + "\n"); - isRunning = line.contains(" Running"); - } else if (rawStatusentryBuilder != null) { - rawStatusentryBuilder.append(line + "\n"); - } - } - } catch (IOException e) { - System.out.println("Could not parse consensus. Skipping."); - return; - } - } - - /* UTC calendar for importing timestamps into the database. */ - private static Calendar calendarUTC = Calendar.getInstance( - TimeZone.getTimeZone("UTC")); - - /* Import a status entry with one or more OR addresses into the - * database. */ - private static void importStatusentry(long validAfterMillis, - String fingerprint, String descriptor, Set<String> orAddresses, - byte[] rawStatusentry) { - try { - for (String orAddress : orAddresses) { - insertStatusentryStatement.clearParameters(); - insertStatusentryStatement.setTimestamp(1, - new Timestamp(validAfterMillis), calendarUTC); - insertStatusentryStatement.setString(2, fingerprint); - insertStatusentryStatement.setString(3, descriptor); - if (!orAddress.contains(":")) { - String[] addressParts = orAddress.split("\\."); - byte[] address24Bytes = new byte[3]; - address24Bytes[0] = (byte) Integer.parseInt(addressParts[0]); - address24Bytes[1] = (byte) Integer.parseInt(addressParts[1]); - address24Bytes[2] = (byte) Integer.parseInt(addressParts[2]); - String orAddress24 = Hex.encodeHexString(address24Bytes); - insertStatusentryStatement.setString(4, orAddress24); - insertStatusentryStatement.setNull(5, Types.VARCHAR); - insertStatusentryStatement.setString(6, orAddress); - } else { - StringBuilder addressHex = new StringBuilder(); - int start = orAddress.startsWith("[::") ? 2 : 1; - int end = orAddress.length() - - (orAddress.endsWith("::]") ? 2 : 1); - String[] parts = orAddress.substring(start, end).split(":", -1); - for (int i = 0; i < parts.length; i++) { - String part = parts[i]; - if (part.length() == 0) { - addressHex.append("x"); - } else if (part.length() <= 4) { - addressHex.append(String.format("%4s", part)); - } else { - addressHex = null; - break; - } - } - String orAddress48 = null; - if (addressHex != null) { - String addressHexString = addressHex.toString(); - addressHexString = addressHexString.replaceFirst("x", - String.format("%" + (33 - addressHexString.length()) - + "s", "0")); - if (!addressHexString.contains("x") && - addressHexString.length() == 32) { - orAddress48 = addressHexString.replaceAll(" ", "0"). - toLowerCase().substring(0, 12); - } - } - if (orAddress48 != null) { - insertStatusentryStatement.setNull(4, Types.VARCHAR); - insertStatusentryStatement.setString(5, orAddress48); - insertStatusentryStatement.setString(6, - orAddress.replaceAll("[\\[\\]]", "")); - } else { - System.err.println("Could not import status entry with IPv6 " - + "address '" + orAddress + "'. Exiting."); - System.exit(1); - } - } - insertStatusentryStatement.setBytes(7, rawStatusentry); - insertStatusentryStatement.execute(); - } - } catch (SQLException e) { - System.out.println("Could not import status entry. Exiting."); - System.exit(1); - } - } - - /* Parse an exit list. */ - private static void parseExitList(File file, byte[] bytes) { - try { - BufferedReader br = new BufferedReader(new StringReader(new String( - bytes, "US-ASCII"))); - String fingerprint = null; - Set<String> exitAddressLines = new HashSet<String>(); - StringBuilder rawExitlistentryBuilder = new StringBuilder(); - while (true) { - String line = br.readLine(); - if ((line == null || line.startsWith("ExitNode ")) && - fingerprint != null) { - for (String exitAddressLine : exitAddressLines) { - String[] parts = exitAddressLine.split(" "); - String exitAddress = parts[1]; - /* TODO Extend the following code for IPv6 once the exit list - * format supports it. */ - String[] exitAddressParts = exitAddress.split("\\."); - byte[] exitAddress24Bytes = new byte[3]; - exitAddress24Bytes[0] = (byte) Integer.parseInt( - exitAddressParts[0]); - exitAddress24Bytes[1] = (byte) Integer.parseInt( - exitAddressParts[1]); - exitAddress24Bytes[2] = (byte) Integer.parseInt( - exitAddressParts[2]); - String exitAddress24 = Hex.encodeHexString( - exitAddress24Bytes); - String scannedTime = parts[2] + " " + parts[3]; - long scannedMillis = -1L; - try { - scannedMillis = parseFormat.parse(scannedTime).getTime(); - } catch (ParseException e) { - System.out.println("Could not parse timestamp in " - + "'" + file + "'. Skipping."); - return; - } - byte[] rawExitlistentry = rawExitlistentryBuilder.toString(). - getBytes(); - importExitlistentry(fingerprint, exitAddress24, exitAddress, - scannedMillis, rawExitlistentry); - } - exitAddressLines.clear(); - rawExitlistentryBuilder = new StringBuilder(); - } - if (line == null) { - break; - } - rawExitlistentryBuilder.append(line + "\n"); - if (line.startsWith("ExitNode ")) { - fingerprint = line.substring("ExitNode ".length()). - toLowerCase(); - } else if (line.startsWith("ExitAddress ")) { - exitAddressLines.add(line); - } - } - br.close(); - } catch (IOException e) { - System.out.println("Could not parse exit list. Skipping."); - return; - } - } - - /* Import an exit list entry into the database. */ - private static void importExitlistentry(String fingerprint, - String exitAddress24, String exitAddress, long scannedMillis, - byte[] rawExitlistentry) { - try { - insertExitlistentryStatement.clearParameters(); - insertExitlistentryStatement.setString(1, fingerprint); - insertExitlistentryStatement.setString(2, exitAddress24); - insertExitlistentryStatement.setString(3, exitAddress); - insertExitlistentryStatement.setTimestamp(4, - new Timestamp(scannedMillis), calendarUTC); - insertExitlistentryStatement.setBytes(5, rawExitlistentry); - insertExitlistentryStatement.execute(); - } catch (SQLException e) { - System.out.println("Could not import exit list entry. Exiting."); - System.exit(1); - } - } - - /* Write parse history from memory to disk for the next execution. */ - private static void writeImportHistoryToDisk() { - File parseHistoryFile = new File("stats/exonerator-import-history"); - parseHistoryFile.getParentFile().mkdirs(); - try { - BufferedWriter bw = new BufferedWriter(new FileWriter( - parseHistoryFile)); - for (Map.Entry<String, Long> historyEntry : - nextImportHistory.entrySet()) { - bw.write(String.valueOf(historyEntry.getValue()) + "," - + historyEntry.getKey() + "\n"); - } - bw.close(); - } catch (IOException e) { - System.out.println("File 'stats/exonerator-import-history' could " - + "not be written. Ignoring."); - } - } - - /* Close the database connection. */ - private static void closeDatabaseConnection() { - try { - connection.close(); - } catch (SQLException e) { - System.out.println("Could not close database connection. " - + "Ignoring."); - } - } - - /* Delete the exonerator-lock file to allow the next executing of this - * tool. */ - private static void deleteLockFile() { - new File("exonerator-lock").delete(); - } -} - diff --git a/src/org/torproject/exonerator/ExoneraTorServlet.java b/src/org/torproject/exonerator/ExoneraTorServlet.java deleted file mode 100644 index 68d79a3..0000000 --- a/src/org/torproject/exonerator/ExoneraTorServlet.java +++ /dev/null @@ -1,911 +0,0 @@ -/* Copyright 2011--2015 The Tor Project - * See LICENSE for licensing information */ -package org.torproject.exonerator; - -import java.io.IOException; -import java.io.PrintWriter; -import java.sql.CallableStatement; -import java.sql.Connection; -import java.sql.ResultSet; -import java.sql.SQLException; -import java.sql.Statement; -import java.text.ParseException; -import java.text.SimpleDateFormat; -import java.util.ArrayList; -import java.util.Calendar; -import java.util.List; -import java.util.Locale; -import java.util.ResourceBundle; -import java.util.SortedSet; -import java.util.TimeZone; -import java.util.TreeSet; -import java.util.logging.Level; -import java.util.logging.Logger; -import java.util.regex.Pattern; - -import javax.naming.Context; -import javax.naming.InitialContext; -import javax.naming.NamingException; -import javax.servlet.ServletException; -import javax.servlet.http.HttpServlet; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; -import javax.sql.DataSource; - -import org.apache.commons.codec.binary.Hex; -import org.apache.commons.lang.StringEscapeUtils; - -public class ExoneraTorServlet extends HttpServlet { - - private static final long serialVersionUID = 1370088989739567509L; - - private DataSource ds; - - private Logger logger; - - public void init() { - - /* Initialize logger. */ - this.logger = Logger.getLogger(ExoneraTorServlet.class.toString()); - - /* Look up data source. */ - try { - Context cxt = new InitialContext(); - this.ds = (DataSource) cxt.lookup("java:comp/env/jdbc/exonerator"); - this.logger.info("Successfully looked up data source."); - } catch (NamingException e) { - this.logger.log(Level.WARNING, "Could not look up data source", e); - } - } - - public void doGet(HttpServletRequest request, - HttpServletResponse response) throws IOException, - ServletException { - - /* Set content type, or the page doesn't render in Chrome. */ - response.setContentType("text/html"); - - /* Start writing response. */ - PrintWriter out = response.getWriter(); - this.writeHeader(out); - - /* Find the right resource bundle for the user's locale. */ - Locale locale = request.getLocale(); - ResourceBundle rb = ResourceBundle.getBundle("ExoneraTor", locale); - - /* Open a database connection that we'll use to handle the whole - * request. */ - long requestedConnection = System.currentTimeMillis(); - Connection conn = this.connectToDatabase(); - if (conn == null) { - this.writeSummaryUnableToConnectToDatabase(out, rb); - this.writeFooter(out, rb); - return; - } - - /* Look up first and last date in the database. */ - long[] firstAndLastDates = this.queryFirstAndLastDatesFromDatabase( - conn); - if (firstAndLastDates == null) { - this.writeSummaryNoData(out, rb); - this.writeFooter(out, rb); - this.closeDatabaseConnection(conn, requestedConnection); - } - SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); - dateFormat.setTimeZone(TimeZone.getTimeZone("UTC")); - String firstDate = dateFormat.format(firstAndLastDates[0]); - String lastDate = dateFormat.format(firstAndLastDates[1]); - - /* Parse parameters. */ - String ipParameter = request.getParameter("ip"); - String relayIP = this.parseIpParameter(ipParameter); - boolean relayIPHasError = relayIP == null; - - /* Parse timestamp parameter. */ - String timestampParameter = request.getParameter("timestamp"); - String timestampStr = this.parseTimestampParameter( - timestampParameter); - boolean timestampHasError = timestampStr == null; - - /* Check that timestamp is within range. */ - long timestamp = 0L; - boolean timestampOutOfRange = false; - if (timestampStr != null && timestampStr.length() > 0) { - try { - timestamp = dateFormat.parse(timestampParameter).getTime(); - if (timestamp < firstAndLastDates[0] || - timestamp > firstAndLastDates[1]) { - timestampOutOfRange = true; - } - } catch (ParseException e) { - /* Already checked in parseTimestamp(). */ - } - } - - /* Write form. */ - this.writeForm(out, rb, relayIP, relayIPHasError || - ("".equals(relayIP) && !"".equals(timestampStr)), timestampStr, - !relayIPHasError && - !("".equals(relayIP) && !"".equals(timestampStr)) && - (timestampHasError || timestampOutOfRange || - (!"".equals(relayIP) && "".equals(timestampStr)))); - - /* If both parameters are empty, don't print any summary and exit. - * This is the start page. */ - if ("".equals(relayIP) && "".equals(timestampStr)) { - this.writeFooter(out, rb); - this.closeDatabaseConnection(conn, requestedConnection); - return; - } - - /* If either parameter is empty, print summary with warning message - * and exit. */ - if ("".equals(relayIP) || "".equals(timestampStr)) { - if ("".equals(relayIP)) { - writeSummaryNoIp(out, rb); - } else { - writeSummaryNoTimestamp(out, rb); - } - this.writeFooter(out, rb); - this.closeDatabaseConnection(conn, requestedConnection); - return; - } - - /* If there's a user error, print summary with exit message and - * exit. */ - if (relayIPHasError || timestampHasError || timestampOutOfRange) { - if (relayIPHasError) { - this.writeSummaryInvalidIp(out, rb, ipParameter); - } else if (timestampHasError) { - this.writeSummaryInvalidTimestamp(out, rb, timestampParameter); - } else if (timestampOutOfRange) { - this.writeSummaryTimestampOutsideRange(out, rb, timestampStr, - firstDate, lastDate); - } - this.writeFooter(out, rb); - this.closeDatabaseConnection(conn, requestedConnection); - return; - } - - /* Consider all consensuses published on or within a day of the given - * date. */ - long timestampFrom = timestamp - 24L * 60L * 60L * 1000L; - long timestampTo = timestamp + 2 * 24L * 60L * 60L * 1000L - 1L; - SimpleDateFormat validAfterTimeFormat = new SimpleDateFormat( - "yyyy-MM-dd HH:mm:ss"); - validAfterTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); - String fromValidAfter = validAfterTimeFormat.format(timestampFrom); - String toValidAfter = validAfterTimeFormat.format(timestampTo); - SortedSet<Long> relevantConsensuses = - this.queryKnownConsensusValidAfterTimes(conn, fromValidAfter, - toValidAfter); - if (relevantConsensuses == null || relevantConsensuses.isEmpty()) { - this.writeSummaryNoDataForThisInterval(out, rb); - this.writeFooter(out, rb); - this.closeDatabaseConnection(conn, requestedConnection); - return; - } - - /* Search for status entries with the given IP address as onion - * routing address, plus status entries of relays having an exit list - * entry with the given IP address as exit address. */ - List<String[]> statusEntries = this.queryStatusEntries(conn, relayIP, - timestamp, validAfterTimeFormat); - - /* If we didn't find anything, run another query to find out if there - * are relays running on other IP addresses in the same /24 or /48 - * network and tell the user about it. */ - List<String> addressesInSameNetwork = null; - if (statusEntries.isEmpty()) { - addressesInSameNetwork = new ArrayList<String>(); - if (!relayIP.contains(":")) { - String address24 = this.convertIPv4ToHex(relayIP).substring(0, 6); - if (address24 != null) { - addressesInSameNetwork = this.queryAddressesInSame24(conn, - address24, timestamp); - } - } else { - String address48 = this.convertIPv6ToHex(relayIP).substring( - 0, 12); - if (address48 != null) { - addressesInSameNetwork = this.queryAddressesInSame48(conn, - address48, timestamp); - } - } - } - - /* Print out result. */ - if (!statusEntries.isEmpty()) { - this.writeSummaryPositive(out, rb, relayIP, timestampStr); - this.writeTechnicalDetails(out, rb, relayIP, timestampStr, - statusEntries); - } else if (addressesInSameNetwork != null && - !addressesInSameNetwork.isEmpty()) { - this.writeSummaryAddressesInSameNetwork(out, rb, relayIP, - timestampStr, addressesInSameNetwork); - } else { - this.writeSummaryNegative(out, rb, relayIP, timestampStr); - } - - this.writePermanentLink(out, rb, relayIP, timestampStr); - - this.closeDatabaseConnection(conn, requestedConnection); - this.writeFooter(out, rb); - } - - /* Helper methods for handling the request. */ - - private String parseIpParameter(String passedIpParameter) { - String relayIP = null; - if (passedIpParameter != null && passedIpParameter.length() > 0) { - String ipParameter = passedIpParameter.trim(); - Pattern ipv4AddressPattern = Pattern.compile( - "^([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + - "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + - "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + - "([01]?\\d\\d?|2[0-4]\\d|25[0-5])$"); - Pattern ipv6AddressPattern = Pattern.compile( - "^\\[?[0-9a-fA-F:]{3,39}\\]?$"); - if (ipv4AddressPattern.matcher(ipParameter).matches()) { - String[] ipParts = ipParameter.split("\\."); - relayIP = Integer.parseInt(ipParts[0]) + "." - + Integer.parseInt(ipParts[1]) + "." - + Integer.parseInt(ipParts[2]) + "." - + Integer.parseInt(ipParts[3]); - } else if (ipv6AddressPattern.matcher(ipParameter).matches()) { - if (ipParameter.startsWith("[") && ipParameter.endsWith("]")) { - ipParameter = ipParameter.substring(1, - ipParameter.length() - 1); - } - StringBuilder addressHex = new StringBuilder(); - int start = ipParameter.startsWith("::") ? 1 : 0; - int end = ipParameter.length() - - (ipParameter.endsWith("::") ? 1 : 0); - String[] parts = ipParameter.substring(start, end).split(":", -1); - for (int i = 0; i < parts.length; i++) { - String part = parts[i]; - if (part.length() == 0) { - addressHex.append("x"); - } else if (part.length() <= 4) { - addressHex.append(String.format("%4s", part)); - } else { - addressHex = null; - break; - } - } - if (addressHex != null) { - String addressHexString = addressHex.toString(); - addressHexString = addressHexString.replaceFirst("x", - String.format("%" + (33 - addressHexString.length()) + "s", - "0")); - if (!addressHexString.contains("x") && - addressHexString.length() == 32) { - relayIP = ipParameter.toLowerCase(); - } - } - } - } else { - relayIP = ""; - } - return relayIP; - } - - private String convertIPv4ToHex(String relayIP) { - String[] relayIPParts = relayIP.split("\\."); - byte[] address24Bytes = new byte[4]; - for (int i = 0; i < address24Bytes.length; i++) { - address24Bytes[i] = (byte) Integer.parseInt(relayIPParts[i]); - } - String address24 = Hex.encodeHexString(address24Bytes); - return address24; - } - - private String convertIPv6ToHex(String relayIP) { - if (relayIP.startsWith("[") && relayIP.endsWith("]")) { - relayIP = relayIP.substring(1, relayIP.length() - 1); - } - StringBuilder addressHex = new StringBuilder(); - int start = relayIP.startsWith("::") ? 1 : 0; - int end = relayIP.length() - (relayIP.endsWith("::") ? 1 : 0); - String[] parts = relayIP.substring(start, end).split(":", -1); - for (int i = 0; i < parts.length; i++) { - String part = parts[i]; - if (part.length() == 0) { - addressHex.append("x"); - } else if (part.length() <= 4) { - addressHex.append(String.format("%4s", part)); - } else { - addressHex = null; - break; - } - } - String address48 = null; - if (addressHex != null) { - String addressHexString = addressHex.toString(); - addressHexString = addressHexString.replaceFirst("x", - String.format("%" + (33 - addressHexString.length()) - + "s", "0")); - if (!addressHexString.contains("x") && - addressHexString.length() == 32) { - address48 = addressHexString.replaceAll(" ", "0"). - toLowerCase(); - } - } - return address48; - } - - private String parseTimestampParameter( - String passedTimestampParameter) { - String timestampStr = ""; - SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); - dateFormat.setTimeZone(TimeZone.getTimeZone("UTC")); - dateFormat.setLenient(false); - if (passedTimestampParameter != null && - passedTimestampParameter.length() > 0) { - String timestampParameter = passedTimestampParameter.trim(); - try { - long timestamp = dateFormat.parse(timestampParameter).getTime(); - timestampStr = dateFormat.format(timestamp); - } catch (ParseException e) { - timestampStr = null; - } - } - return timestampStr; - } - - /* Helper methods for querying the database. */ - - private Connection connectToDatabase() { - Connection conn = null; - try { - conn = this.ds.getConnection(); - } catch (SQLException e) { - } - return conn; - } - - private long[] queryFirstAndLastDatesFromDatabase(Connection conn) { - long[] firstAndLastDates = null; - try { - Statement statement = conn.createStatement(); - String query = "SELECT DATE(MIN(validafter)) AS first, " - + "DATE(MAX(validafter)) AS last FROM statusentry"; - ResultSet rs = statement.executeQuery(query); - if (rs.next()) { - Calendar utcCalendar = Calendar.getInstance( - TimeZone.getTimeZone("UTC")); - firstAndLastDates = new long[] { - rs.getTimestamp(1, utcCalendar).getTime(), - rs.getTimestamp(2, utcCalendar).getTime() - }; - } - rs.close(); - statement.close(); - } catch (SQLException e) { - /* Looks like we don't have any consensuses. */ - firstAndLastDates = null; - } - return firstAndLastDates; - } - - private SortedSet<Long> queryKnownConsensusValidAfterTimes( - Connection conn, String fromValidAfter, String toValidAfter) { - SortedSet<Long> relevantConsensuses = new TreeSet<Long>(); - try { - Statement statement = conn.createStatement(); - String query = "SELECT DISTINCT validafter FROM statusentry " - + "WHERE validafter >= '" + fromValidAfter - + "' AND validafter <= '" + toValidAfter + "'"; - ResultSet rs = statement.executeQuery(query); - while (rs.next()) { - long consensusTime = rs.getTimestamp(1).getTime(); - relevantConsensuses.add(consensusTime); - } - rs.close(); - statement.close(); - } catch (SQLException e) { - /* Looks like we don't have any consensuses in the requested - * interval. */ - relevantConsensuses = null; - } - return relevantConsensuses; - } - - private List<String[]> queryStatusEntries(Connection conn, - String relayIP, long timestamp, - SimpleDateFormat validAfterTimeFormat) { - List<String[]> statusEntries = new ArrayList<String[]>(); - String addressHex = !relayIP.contains(":") - ? this.convertIPv4ToHex(relayIP) : this.convertIPv6ToHex(relayIP); - if (addressHex == null) { - return null; - } - String address24Or48Hex = !relayIP.contains(":") - ? addressHex.substring(0, 6) : addressHex.substring(0, 12); - try { - CallableStatement cs; - if (!relayIP.contains(":")) { - cs = conn.prepareCall("{call search_by_address24_date(?, ?)}"); - } else { - cs = conn.prepareCall("{call search_by_address48_date(?, ?)}"); - } - cs.setString(1, address24Or48Hex); - Calendar utcCalendar = Calendar.getInstance( - TimeZone.getTimeZone("UTC")); - cs.setDate(2, new java.sql.Date(timestamp), utcCalendar); - ResultSet rs = cs.executeQuery(); - while (rs.next()) { - byte[] rawstatusentry = rs.getBytes(1); - SortedSet<String> addresses = new TreeSet<String>(), - addressesHex = new TreeSet<String>(); - long validafter = rs.getTimestamp(2, utcCalendar).getTime(); - String validAfterString = validAfterTimeFormat.format(validafter); - String fingerprint = rs.getString(3).toUpperCase(); - String nickname = null; - String exit = "U"; - for (String line : new String(rawstatusentry).split("\n")) { - if (line.startsWith("r ")) { - String[] parts = line.split(" "); - nickname = parts[1]; - addresses.add(parts[6]); - addressesHex.add(this.convertIPv4ToHex(parts[6])); - } else if (line.startsWith("a ")) { - String address = line.substring("a ".length(), - line.lastIndexOf(":")); - addresses.add(address); - String orAddressHex = !address.contains(":") - ? this.convertIPv4ToHex(address) - : this.convertIPv6ToHex(address); - addressesHex.add(orAddressHex); - } else if (line.startsWith("p ")) { - exit = line.equals("p reject 1-65535") ? "N" : "Y"; - } - } - String exitaddress = rs.getString(4); - if (exitaddress != null && exitaddress.length() > 0) { - addresses.add(exitaddress); - addressesHex.add(this.convertIPv4ToHex(exitaddress)); - } - if (!addressesHex.contains(addressHex)) { - continue; - } - StringBuilder sb = new StringBuilder(); - int writtenAddresses = 0; - for (String address : addresses) { - sb.append((writtenAddresses++ > 0 ? ", " : "") + address); - } - String[] statusEntry = new String[] { validAfterString, - sb.toString(), fingerprint, nickname, exit }; - statusEntries.add(statusEntry); - } - rs.close(); - cs.close(); - } catch (SQLException e) { - /* Nothing found. */ - statusEntries = null; - } - return statusEntries; - } - - private List<String> queryAddressesInSame24(Connection conn, - String address24, long timestamp) { - List<String> addressesInSameNetwork = new ArrayList<String>(); - try { - CallableStatement cs = conn.prepareCall( - "{call search_addresses_in_same_24 (?, ?)}"); - cs.setString(1, address24); - cs.setDate(2, new java.sql.Date(timestamp)); - ResultSet rs = cs.executeQuery(); - while (rs.next()) { - String address = rs.getString(1); - if (!addressesInSameNetwork.contains(address)) { - addressesInSameNetwork.add(address); - } - } - rs.close(); - cs.close(); - } catch (SQLException e) { - /* No other addresses in the same /24 found. */ - addressesInSameNetwork = null; - } - return addressesInSameNetwork; - } - - private List<String> queryAddressesInSame48(Connection conn, - String address48, long timestamp) { - List<String> addressesInSameNetwork = new ArrayList<String>(); - try { - CallableStatement cs = conn.prepareCall( - "{call search_addresses_in_same_48 (?, ?)}"); - cs.setString(1, address48); - cs.setDate(2, new java.sql.Date(timestamp)); - ResultSet rs = cs.executeQuery(); - while (rs.next()) { - String address = rs.getString(1); - if (!addressesInSameNetwork.contains(address)) { - addressesInSameNetwork.add(address); - } - } - rs.close(); - cs.close(); - } catch (SQLException e) { - /* No other addresses in the same /48 found. */ - addressesInSameNetwork = null; - } - return addressesInSameNetwork; - } - - private void closeDatabaseConnection(Connection conn, - long requestedConnection) { - try { - conn.close(); - this.logger.info("Returned a database connection to the pool " - + "after " + (System.currentTimeMillis() - - requestedConnection) + " millis."); - } catch (SQLException e) { - } - return; - } - - /* Helper methods for writing the response. */ - - private void writeHeader(PrintWriter out) throws IOException { - out.println("<!DOCTYPE html>\n" - + "<html lang=\"en\">\n" - + " <head>\n" - + " <meta charset=\"utf-8\">\n" - + " <meta http-equiv=\"X-UA-Compatible\" " - + "content=\"IE=edge\">\n" - + " <meta name=\"viewport\" content=\"width=device-width, " - + "initial-scale=1\">\n" - + " <title>ExoneraTor</title>\n" - + " <link rel=\"stylesheet\" href=\"css/bootstrap.min.css\">\n" - + " <link rel=\"stylesheet\" href=\"css/exonerator.css\">\n" - + " <link href=\"images/favicon.ico\" type=\"image/x-icon\" " - + "rel=\"icon\">\n" - + " </head>\n" - + " <body>\n" - + " <div class=\"container\">\n" - + " <div class=\"row\">\n" - + " <div class=\"col-xs-12\">\n" - + " <div class=\"page-header\">\n" - + " <h1>\n" - + " <div class=\"text-center\">\n" - + " <a href=\"/\">" - + "<img src=\"images/exonerator-logo.png\" " - + "width=\"334\" height=\"252\" alt=\"ExoneraTor logo\">" - + "<img src=\"images/exonerator-wordmark.png\" width=\"428\" " - + "height=\"63\" alt=\"ExoneraTor wordmark\"></a>\n" - + " </div>\n" - + " </h1>\n" - + " </div>\n" - + " </div>\n" - + " </div>\n"); - } - - private void writeForm(PrintWriter out, ResourceBundle rb, - String relayIP, boolean relayIPHasError, String timestampStr, - boolean timestampHasError) throws IOException { - String ipValue = ""; - if (relayIP != null && relayIP.length() > 0) { - if (relayIP.contains(":")) { - ipValue = String.format(" value=\"[%s]\"", relayIP); - } else { - ipValue = String.format(" value=\"%s\"", relayIP); - } - } - out.printf(" <div class=\"row\">\n" - + " <div class=\"col-xs-12\">\n" - + " <div class=\"text-center\">\n" - + " <div class=\"row vbottom15\">\n" - + " <h4>%s</h4>\n" - + " </div> \n" - + " <form class=\"form-inline\">\n" - + " <div class=\"form-group%s\">\n" - + " <label for=\"inputIp\" " - + "class=\"control-label\">%s</label>\n" - + " <input type=\"text\" class=\"form-control\" " - + "name=\"ip\" id=\"inputIp\" placeholder=\"86.59.21.38\"%s " - + "required>\n" - + " </div>\n" - + " <div class=\"form-group%s\">\n" - + " <label for=\"inputTimestamp\" " - + "class=\"control-label\">%s</label>\n" - + " <input type=\"date\" class=\"form-control\" " - + "name=\"timestamp\" id=\"inputTimestamp\" " - + "placeholder=\"2010-01-01\"%s required>\n" - + " </div>\n" - + " <button type=\"submit\" " - + "class=\"btn btn-primary\">%s</button>\n" - + " </form>\n" - + " </div>\n" - + " </div>\n" - + " </div>\n", - rb.getString("form.explanation"), - relayIPHasError ? " has-error" : "", - rb.getString("form.ip.label"), - ipValue, - timestampHasError ? " has-error" : "", - rb.getString("form.timestamp.label"), - timestampStr != null && timestampStr.length() > 0 ? - " value=\"" + timestampStr + "\"" : "", - rb.getString("form.search.label")); - } - - private void writeSummaryUnableToConnectToDatabase(PrintWriter out, - ResourceBundle rb) throws IOException { - String contactLink = - "<a href=\"https://www.torproject.org/about/contact\">" - + rb.getString("summary.serverproblem.dbempty.body.link") - + "</a>"; - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.serverproblem.dbnoconnect.title"), null, - rb.getString("summary.serverproblem.dbnoconnect.body.text"), - contactLink); - } - - private void writeSummaryNoData(PrintWriter out, ResourceBundle rb) - throws IOException { - String contactLink = - "<a href=\"https://www.torproject.org/about/contact\">" - + rb.getString("summary.serverproblem.dbempty.body.link") - + "</a>"; - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.serverproblem.dbempty.title"), null, - rb.getString("summary.serverproblem.dbempty.body.text"), - contactLink); - } - - private void writeSummaryNoTimestamp(PrintWriter out, ResourceBundle rb) - throws IOException { - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.invalidparams.notimestamp.title"), null, - rb.getString("summary.invalidparams.notimestamp.body")); - } - - private void writeSummaryNoIp(PrintWriter out, ResourceBundle rb) - throws IOException { - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", rb.getString("summary.invalidparams.noip.title"), - null, rb.getString("summary.invalidparams.noip.body")); - } - - private void writeSummaryTimestampOutsideRange(PrintWriter out, - ResourceBundle rb, String timestampStr, String firstDate, - String lastDate) throws IOException { - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.invalidparams.timestamprange.title"), null, - rb.getString("summary.invalidparams.timestamprange.body"), - timestampStr, firstDate, lastDate); - } - - private void writeSummaryInvalidIp(PrintWriter out, ResourceBundle rb, - String ipParameter) throws IOException { - String escapedIpParameter = ipParameter.length() > 40 ? - StringEscapeUtils.escapeHtml(ipParameter.substring(0, 40)) - + "[...]" : StringEscapeUtils.escapeHtml(ipParameter); - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.invalidparams.invalidip.title"), null, - rb.getString("summary.invalidparams.invalidip.body"), - escapedIpParameter, "\"a.b.c.d\"", "\"[a:b:c:d:e:f:g:h]\""); - } - - private void writeSummaryInvalidTimestamp(PrintWriter out, - ResourceBundle rb, String timestampParameter) throws IOException { - String escapedTimestampParameter = timestampParameter.length() > 20 ? - StringEscapeUtils.escapeHtml(timestampParameter. - substring(0, 20)) + "[...]" : - StringEscapeUtils.escapeHtml(timestampParameter); - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.invalidparams.invalidtimestamp.title"), - null, rb.getString("summary.invalidparams.invalidtimestamp.body"), - escapedTimestampParameter, "\"YYYY-MM-DD\""); - } - - private void writeSummaryNoDataForThisInterval(PrintWriter out, - ResourceBundle rb) throws IOException { - String contactLink = - "<a href=\"https://www.torproject.org/about/contact\">" - + rb.getString("summary.serverproblem.dbempty.body.link") - + "</a>"; - this.writeSummary(out, rb.getString("summary.heading"), - "panel-danger", - rb.getString("summary.serverproblem.nodata.title"), null, - rb.getString("summary.serverproblem.nodata.body.text"), - contactLink); - } - - private void writeSummaryAddressesInSameNetwork(PrintWriter out, - ResourceBundle rb, String relayIP, String timestampStr, - List<String> addressesInSameNetwork) throws IOException { - Object[][] panelItems = new Object[addressesInSameNetwork.size()][]; - for (int i = 0; i < addressesInSameNetwork.size(); i++) { - String addressInSameNetwork = addressesInSameNetwork.get(i); - String link, address; - if (addressInSameNetwork.contains(":")) { - link = String.format("/?ip=[%s]&timestamp=%s", - addressInSameNetwork.replaceAll(":", "%3A"), timestampStr); - address = "[" + addressInSameNetwork + "]"; - } else { - link = String.format("/?ip=%s&timestamp=%s", - addressInSameNetwork, timestampStr); - address = addressInSameNetwork; - } - panelItems[i] = new Object[] { link, address }; - } - this.writeSummary(out, rb.getString("summary.heading"), - "panel-warning", - rb.getString("summary.negativesamenetwork.title"), panelItems, - rb.getString("summary.negativesamenetwork.body"), - relayIP, timestampStr, relayIP.contains(":") ? 48 : 24); - } - - private void writeSummaryPositive(PrintWriter out, ResourceBundle rb, - String relayIP, String timestampStr) throws IOException { - String formattedRelayIP = relayIP.contains(":") ? - "[" + relayIP + "]" : relayIP; - this.writeSummary(out, rb.getString("summary.heading"), - "panel-success", rb.getString("summary.positive.title"), null, - rb.getString("summary.positive.body"), formattedRelayIP, - timestampStr); - } - - private void writeSummaryNegative(PrintWriter out, ResourceBundle rb, - String relayIP, String timestampStr) throws IOException { - String formattedRelayIP = relayIP.contains(":") ? - "[" + relayIP + "]" : relayIP; - this.writeSummary(out, rb.getString("summary.heading"), - "panel-warning", rb.getString("summary.negative.title"), null, - rb.getString("summary.negative.body"), formattedRelayIP, - timestampStr); - } - - private void writeSummary(PrintWriter out, String heading, - String panelContext, String panelTitle, Object[][] panelItems, - String panelBodyTemplate, Object... panelBodyArgs) - throws IOException { - out.printf(" <div class=\"row\">\n" - + " <div class=\"col-xs-12\">\n" - + " <h2>%s</h2>\n" - + " <div class=\"panel %s\">\n" - + " <div class=\"panel-heading\">\n" - + " <h3 class=\"panel-title\">%s</h3>\n" - + " </div>\n" - + " <div class=\"panel-body\">\n" - + " <p>%s</p>\n", heading, panelContext, panelTitle, - String.format(panelBodyTemplate, panelBodyArgs)); - if (panelItems != null) { - out.print(" <ul>\n"); - for (Object[] panelItem : panelItems) { - out.printf(" <li><a href=\"%s\">%s</a></li>\n", - panelItem); - } - out.print(" </ul>\n"); - } - out.print(" </div>\n" - + " </div>\n" - + " </div>\n" - + " </div>\n"); - } - - private void writeTechnicalDetails(PrintWriter out, ResourceBundle rb, - String relayIP, String timestampStr, List<String[]> tableRows) - throws IOException { - String formattedRelayIP = relayIP.contains(":") ? - "[" + relayIP + "]" : relayIP; - out.printf(" <div class=\"row\">\n" - + " <div class=\"col-xs-12\">\n" - + " <h2>%s</h2>\n" - + " <p>%s</p>\n" - + " <table class=\"table\">\n" - + " <thead>\n" - + " <tr>\n" - + " <th>%s</th>\n" - + " <th>%s</th>\n" - + " <th>%s</th>\n" - + " <th>%s</th>\n" - + " <th>%s</th>\n" - + " </tr>\n" - + " </thead>\n" - + " <tbody>\n", - rb.getString("technicaldetails.heading"), - String.format(rb.getString("technicaldetails.pre"), - formattedRelayIP, timestampStr), - rb.getString("technicaldetails.colheader.timestamp"), - rb.getString("technicaldetails.colheader.ip"), - rb.getString("technicaldetails.colheader.fingerprint"), - rb.getString("technicaldetails.colheader.nickname"), - rb.getString("technicaldetails.colheader.exit")); - for (String[] tableRow : tableRows) { - out.print(" <tr>"); - for (int i = 0; i < tableRow.length; i++) { - String content = tableRow[i]; - if (i == 2) { - content = content.substring(0, 20) + "" - + content.substring(20, 40); - } else if (i == 3 && content == null) { - content = "(" - + rb.getString("technicaldetails.nickname.unknown") + ")"; - } else if (i == 4) { - if (content.equals("U")) { - content = rb.getString("technicaldetails.exit.unknown"); - } else if (content.equals("Y")) { - content = rb.getString("technicaldetails.exit.yes"); - } else { - content = rb.getString("technicaldetails.exit.no"); - } - } - out.print(" <td>" + content + "</td>"); - } - out.print(" </tr>\n"); - } - out.print(" </tbody>\n" - + " </table>\n" - + " </div>\n" - + " </div>\n"); - } - - private void writePermanentLink(PrintWriter out, ResourceBundle rb, - String relayIP, String timestampStr) throws IOException { - String encodedAddress = relayIP.contains(":") ? - "[" + relayIP.replaceAll(":", "%3A") + "]" : relayIP; - out.printf(" <div class=\"row\">\n" - + " <div class=\"col-xs-12\">\n" - + " <h2>%s</h2>\n" - + " <pre>https://exonerator.torproject.org/?ip=%s&" - + "timestamp=%s</pre>\n" - + " </div>\n" - + " </div>\n", - rb.getString("permanentlink.heading"), - encodedAddress, timestampStr); - } - - private void writeFooter(PrintWriter out, ResourceBundle rb) - throws IOException { - out.printf(" </div>\n" - + " <div class=\"footer\">\n" - + " <div class=\"container\">\n" - + " <div class=\"row\">\n" - + " <div class=\"col-xs-6\">\n" - + " <h3>%s</h3>\n" - + " <p class=\"small\">%s</p>\n" - + " </div>\n", - rb.getString("footer.abouttor.heading"), - String.format(rb.getString("footer.abouttor.body.text"), - "<a href=\"https://www.torproject.org/about/" - + "overview#thesolution\">" - + rb.getString("footer.abouttor.body.link1") + "</a>", - "<a href=\"https://www.torproject.org/about/overview\">" - + rb.getString("footer.abouttor.body.link2") + "</a>", - "<a href=\"https://www.torproject.org/about/contact\">" - + rb.getString("footer.abouttor.body.link3") + "</a>")); - out.printf(" <div class=\"col-xs-6\">\n" - + " <h3>%s</h3>\n" - + " <p class=\"small\">%s</p>\n" - + " </div>\n" - + " </div>\n" - + " <div class=\"row\">\n", - rb.getString("footer.aboutexonerator.heading"), - rb.getString("footer.aboutexonerator.body")); - out.printf(" <div class=\"col-xs-12\">\n" - + " <p class=\"text-center small\">%s</p>\n" - + " </div>\n" - + " </div>\n" - + " </div>\n" - + " </div>\n" - + " </body>\n" - + "</html>\n", - String.format(rb.getString("footer.trademark.text"), - "<a href=\"https://www.torproject.org/docs/" - + "trademark-faq.html.en\">" - + rb.getString("footer.trademark.link") + "</a>")); - out.close(); - } -} -

1 0

[exonerator/master] Add checkstyle task.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit 43ea75698740196a3898d6b54b393f3e2c08ed36 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 11:01:45 2016 +0200 Add checkstyle task. Copied over from CollecTor. --- build.xml | 24 ++++ src/test/resources/metrics_checks.xml | 217 ++++++++++++++++++++++++++++++++++ 2 files changed, 241 insertions(+) diff --git a/build.xml b/build.xml index 2e4f74d..7a52b17 100644 --- a/build.xml +++ b/build.xml @@ -3,7 +3,9 @@  <property name="sources" value="src/main/java"/> <property name="resources" value="res"/> + <property name="generated" value="generated/"/> <property name="classes" value="classes"/> + <property name="testresources" value="src/test/resources/"/> <property name="libs" value="lib"/> <property name="config" value="etc"/> <property name="webxmlfile" value="${config}/web.xml"/> @@ -21,11 +23,17 @@ <include name="commons-lang-2.6.jar"/> </fileset> </path> + <path id="checkstyle.classpath" > + <fileset dir="${libs}"> + <include name="checkstyle-6.17-all.jar" /> + </fileset> + </path> <target name="init"> <copy file="${contextxmltemplate}" tofile="${contextxml}"/> <copy file="config.template" tofile="config"/> <mkdir dir="${classes}"/> + <mkdir dir="${generated}"/> </target>  @@ -52,6 +60,22 @@ </java> </target> + <taskdef resource="com/puppycrawl/tools/checkstyle/ant/checkstyle-ant-task.properties"> + <classpath refid="checkstyle.classpath" /> + </taskdef> + <target name="checks" depends="compile"> + <checkstyle config="${testresources}/metrics_checks.xml"> + <fileset dir="${sources}" + includes="**/*.java" /> + <classpath> + <path refid="classpath" /> + <path refid="checkstyle.classpath" /> + </classpath> + <formatter type="plain" + toFile="${generated}/checkstyle_report.txt"/> + </checkstyle> + </target> +  <target name="war" depends="compile"> diff --git a/src/test/resources/metrics_checks.xml b/src/test/resources/metrics_checks.xml new file mode 100644 index 0000000..0777894 --- /dev/null +++ b/src/test/resources/metrics_checks.xml @@ -0,0 +1,217 @@ +<?xml version="1.0"?> +<!DOCTYPE module PUBLIC + "-//Puppy Crawl//DTD Check Configuration 1.3//EN" + "http://www.puppycrawl.com/dtds/configuration_1_3.dtd"> + + + +<module name = "Checker"> + <property name="charset" value="UTF-8"/> + + <property name="severity" value="warning"/> + + <property name="fileExtensions" value="java, properties, xml"/> +  +  + <module name="FileTabCharacter"> + <property name="eachLine" value="true"/> + </module> + + <module name="TreeWalker"> + <module name="OuterTypeFilename"/> + <module name="IllegalTokenText"> + <property name="tokens" value="STRING_LITERAL, CHAR_LITERAL"/> + <property name="format" value="\\u00(08|09|0(a|A)|0(c|C)|0(d|D)|22|27|5(C|c))|\\(0(10|11|12|14|15|42|47)|134)"/> + <property name="message" value="Avoid using corresponding octal or Unicode escape."/> + </module> + <module name="AvoidEscapedUnicodeCharacters"> + <property name="allowEscapesForControlCharacters" value="true"/> + <property name="allowByTailComment" value="true"/> + <property name="allowNonPrintableEscapes" value="true"/> + </module> + <module name="LineLength"> + <property name="max" value="100"/> + <property name="ignorePattern" value="^package.*|^import.*|a href|href|http://|https://|ftp://"/> + </module> + <module name="AvoidStarImport"/> + <module name="OneTopLevelClass"/> + <module name="NoLineWrap"/> + <module name="EmptyBlock"> + <property name="option" value="TEXT"/> + <property name="tokens" value="LITERAL_TRY, LITERAL_FINALLY, LITERAL_IF, LITERAL_ELSE, LITERAL_SWITCH"/> + </module> + <module name="NeedBraces"/> + <module name="LeftCurly"> + <property name="maxLineLength" value="100"/> + </module> + <module name="RightCurly"/> + <module name="RightCurly"> + <property name="option" value="alone"/> + <property name="tokens" value="CLASS_DEF, METHOD_DEF, CTOR_DEF, LITERAL_FOR, LITERAL_WHILE, LITERAL_DO, STATIC_INIT, INSTANCE_INIT"/> + </module> + <module name="WhitespaceAround"> + <property name="allowEmptyConstructors" value="true"/> + <property name="allowEmptyMethods" value="true"/> + <property name="allowEmptyTypes" value="true"/> + <property name="allowEmptyLoops" value="true"/> + <message key="ws.notFollowed" + value="WhitespaceAround: ''{0}'' is not followed by whitespace. Empty blocks may only be represented as '{}' when not part of a multi-block statement (4.1.3)"/> + <message key="ws.notPreceded" + value="WhitespaceAround: ''{0}'' is not preceded with whitespace."/> + </module> + <module name="OneStatementPerLine"/> + <module name="MultipleVariableDeclarations"/> + <module name="ArrayTypeStyle"/> + <module name="MissingSwitchDefault"/> + <module name="FallThrough"/> + <module name="UpperEll"/> + <module name="ModifierOrder"/> + <module name="EmptyLineSeparator"> + <property name="allowNoEmptyLineBetweenFields" value="true"/> + </module> + <module name="SeparatorWrap"> + <property name="tokens" value="DOT"/> + <property name="option" value="nl"/> + </module> + <module name="SeparatorWrap"> + <property name="tokens" value="COMMA"/> + <property name="option" value="EOL"/> + </module> + <module name="PackageName"> + <property name="format" value="^[a-z]+(\.[a-z][a-z0-9]*)*$"/> + <message key="name.invalidPattern" + value="Package name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="TypeName"> + <message key="name.invalidPattern" + value="Type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="MemberName"> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9]*$"/> + <message key="name.invalidPattern" + value="Member name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="ParameterName"> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9]*$"/> + <message key="name.invalidPattern" + value="Parameter name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="CatchParameterName"> + <property name="format" value="^[a-z][a-zA-Z0-9]*$"/> + <message key="name.invalidPattern" + value="Catch parameter name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="LocalVariableName"> + <property name="tokens" value="VARIABLE_DEF"/> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9]*$"/> + <property name="allowOneCharVarInForLoop" value="true"/> + <message key="name.invalidPattern" + value="Local variable name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="ClassTypeParameterName"> + <property name="format" value="(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)"/> + <message key="name.invalidPattern" + value="Class type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="MethodTypeParameterName"> + <property name="format" value="(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)"/> + <message key="name.invalidPattern" + value="Method type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="InterfaceTypeParameterName"> + <property name="format" value="(^[A-Z][0-9]?)$|([A-Z][a-zA-Z0-9]*[T]$)"/> + <message key="name.invalidPattern" + value="Interface type name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="NoFinalizer"/> + <module name="GenericWhitespace"> + <message key="ws.followed" + value="GenericWhitespace ''{0}'' is followed by whitespace."/> + <message key="ws.preceded" + value="GenericWhitespace ''{0}'' is preceded with whitespace."/> + <message key="ws.illegalFollow" + value="GenericWhitespace ''{0}'' should followed by whitespace."/> + <message key="ws.notPreceded" + value="GenericWhitespace ''{0}'' is not preceded with whitespace."/> + </module> + <module name="Indentation"> + <property name="basicOffset" value="2"/> + <property name="braceAdjustment" value="0"/> + <property name="caseIndent" value="2"/> + <property name="throwsIndent" value="4"/> + <property name="lineWrappingIndentation" value="4"/> + <property name="arrayInitIndent" value="2"/> + </module> + <module name="AbbreviationAsWordInName"> + <property name="ignoreFinal" value="false"/> + <property name="allowedAbbreviationLength" value="1"/> + </module> + <module name="OverloadMethodsDeclarationOrder"/> + <module name="VariableDeclarationUsageDistance"/> + <module name="CustomImportOrder"> + <property name="specialImportsRegExp" value="org.torproject"/> + <property name="sortImportsInGroupAlphabetically" value="true"/> + <property name="customImportOrderRules" value="STATIC###SPECIAL_IMPORTS###THIRD_PARTY_PACKAGE###STANDARD_JAVA_PACKAGE"/> + </module> + <module name="MethodParamPad"/> + <module name="OperatorWrap"> + <property name="option" value="NL"/> + <property name="tokens" value="BAND, BOR, BSR, BXOR, DIV, EQUAL, GE, GT, LAND, LE, LITERAL_INSTANCEOF, LOR, LT, MINUS, MOD, NOT_EQUAL, PLUS, QUESTION, SL, SR, STAR "/> + </module> + <module name="AnnotationLocation"> + <property name="tokens" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF"/> + </module> + <module name="AnnotationLocation"> + <property name="tokens" value="VARIABLE_DEF"/> + <property name="allowSamelineMultipleAnnotations" value="true"/> + </module> + <module name="NonEmptyAtclauseDescription"/> + <module name="JavadocTagContinuationIndentation"/> + <module name="SummaryJavadoc"> + <property name="forbiddenSummaryFragments" value="^@return the *|^This method returns |^A [{]@code [a-zA-Z0-9]+[}]( is a )"/> + </module> + <module name="JavadocParagraph"/> + <module name="AtclauseOrder"> + <property name="tagOrder" value="@param, @return, @throws, @deprecated"/> + <property name="target" value="CLASS_DEF, INTERFACE_DEF, ENUM_DEF, METHOD_DEF, CTOR_DEF, VARIABLE_DEF"/> + </module> + <module name="JavadocMethod"> + <property name="scope" value="public"/> + <property name="allowMissingParamTags" value="true"/> + <property name="allowMissingThrowsTags" value="true"/> + <property name="allowMissingReturnTag" value="true"/> + <property name="minLineCount" value="2"/> + <property name="allowedAnnotations" value="Override, Test"/> + <property name="allowThrowsTagsForSubclasses" value="true"/> + </module> + <module name="MethodName"> + <property name="format" value="^[a-z][a-z0-9][a-zA-Z0-9_]*$"/> + <message key="name.invalidPattern" + value="Method name ''{0}'' must match pattern ''{1}''."/> + </module> + <module name="SingleLineJavadoc"> + <property name="ignoreInlineTags" value="false"/> + </module> + <module name="EmptyCatchBlock"> + <property name="exceptionVariableName" value="expected"/> + </module> + <module name="CommentsIndentation"/> + </module> +</module>

1 0

[exonerator/master] Fix most of the style issues found by checkstyle.
by karsten＠torproject.org 07 Jul '16

07 Jul '16

commit 61171791dddc0245355bcc12b204a2c42674d295 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 6 11:18:01 2016 +0200 Fix most of the style issues found by checkstyle. --- .../exonerator/ExoneraTorDatabaseImporter.java | 78 ++++++++++-------- .../torproject/exonerator/ExoneraTorServlet.java | 92 +++++++++++----------- 2 files changed, 90 insertions(+), 80 deletions(-) diff --git a/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java b/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java index 41751ca..23c009b 100644 --- a/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java +++ b/src/main/java/org/torproject/exonerator/ExoneraTorDatabaseImporter.java @@ -1,7 +1,14 @@ -/* Copyright 2011, 2012 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ + package org.torproject.exonerator; +import org.torproject.descriptor.DescriptorCollector; +import org.torproject.descriptor.DescriptorSourceFactory; + +import org.apache.commons.codec.binary.Base64; +import org.apache.commons.codec.binary.Hex; + import java.io.BufferedInputStream; import java.io.BufferedReader; import java.io.BufferedWriter; @@ -12,7 +19,6 @@ import java.io.FileReader; import java.io.FileWriter; import java.io.IOException; import java.io.StringReader; -import java.io.UnsupportedEncodingException; import java.sql.CallableStatement; import java.sql.Connection; import java.sql.DriverManager; @@ -29,12 +35,6 @@ import java.util.Set; import java.util.Stack; import java.util.TimeZone; -import org.apache.commons.codec.binary.Base64; -import org.apache.commons.codec.binary.Hex; -import org.apache.commons.codec.digest.DigestUtils; -import org.torproject.descriptor.DescriptorCollector; -import org.torproject.descriptor.DescriptorSourceFactory; - /* Import Tor descriptors into the ExoneraTor database. */ public class ExoneraTorDatabaseImporter { @@ -101,6 +101,7 @@ public class ExoneraTorDatabaseImporter { /* Callable statements to import data into the database. */ private static CallableStatement insertStatusentryStatement; + private static CallableStatement insertExitlistentryStatement; /* Prepare statements for importing data into the database. */ @@ -159,7 +160,9 @@ public class ExoneraTorDatabaseImporter { /* Last and next parse histories containing paths of parsed files and * last modified times. */ private static Map<String, Long> - lastImportHistory = new HashMap<String, Long>(), + lastImportHistory = new HashMap<String, Long>(); + + private static Map<String, Long> nextImportHistory = new HashMap<String, Long>(); /* Read stats/exonerator-import-history file from disk and remember @@ -224,8 +227,8 @@ public class ExoneraTorDatabaseImporter { long lastModified = file.lastModified(); String filename = file.getName(); nextImportHistory.put(filename, lastModified); - if (!lastImportHistory.containsKey(filename) || - lastImportHistory.get(filename) < lastModified) { + if (!lastImportHistory.containsKey(filename) + || lastImportHistory.get(filename) < lastModified) { try { FileInputStream fis = new FileInputStream(file); BufferedInputStream bis = new BufferedInputStream(fis); @@ -264,8 +267,8 @@ public class ExoneraTorDatabaseImporter { String startToken = null; if (line.equals("network-status-version 3")) { startToken = "network-status-version 3"; - } else if (line.startsWith("Downloaded ") || - line.startsWith("ExitNode ")) { + } else if (line.startsWith("Downloaded ") + || line.startsWith("ExitNode ")) { startToken = "ExitNode "; } else { System.out.println("Unknown descriptor type in file '" + file @@ -273,7 +276,8 @@ public class ExoneraTorDatabaseImporter { return; } String splitToken = "\n" + startToken; - int length = bytes.length, start = asciiString.indexOf(startToken); + int length = bytes.length; + int start = asciiString.indexOf(startToken); while (start < length) { int end = asciiString.indexOf(splitToken, start); if (end < 0) { @@ -298,6 +302,7 @@ public class ExoneraTorDatabaseImporter { /* Date format to parse UTC timestamps. */ private static SimpleDateFormat parseFormat; + static { parseFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); parseFormat.setTimeZone(TimeZone.getTimeZone("UTC")); @@ -308,32 +313,35 @@ public class ExoneraTorDatabaseImporter { try { BufferedReader br = new BufferedReader(new StringReader(new String( bytes, "US-ASCII"))); - String line, fingerprint = null, descriptor = null; + String line; + String fingerprint = null; + String descriptor = null; Set<String> orAddresses = new HashSet<String>(); long validAfterMillis = -1L; StringBuilder rawStatusentryBuilder = null; boolean isRunning = false; while ((line = br.readLine()) != null) { - if (line.startsWith("vote-status ") && - !line.equals("vote-status consensus")) { - System.out.println("File '" + file + "' contains network status " - + "*votes*, not network status *consensuses*. Skipping."); + if (line.startsWith("vote-status ") + && !line.equals("vote-status consensus")) { + System.out.println("File '" + file + "' contains network " + + "status *votes*, not network status *consensuses*. " + + "Skipping."); return; } else if (line.startsWith("valid-after ")) { String validAfterTime = line.substring("valid-after ".length()); try { - validAfterMillis = parseFormat.parse(validAfterTime). - getTime(); + validAfterMillis = parseFormat.parse(validAfterTime) + .getTime(); } catch (ParseException e) { System.out.println("Could not parse valid-after timestamp in " + "'" + file + "'. Skipping."); return; } - } else if (line.startsWith("r ") || - line.equals("directory-footer")) { + } else if (line.startsWith("r ") + || line.equals("directory-footer")) { if (isRunning) { - byte[] rawStatusentry = rawStatusentryBuilder.toString(). - getBytes(); + byte[] rawStatusentry = rawStatusentryBuilder.toString() + .getBytes(); importStatusentry(validAfterMillis, fingerprint, descriptor, orAddresses, rawStatusentry); orAddresses = new HashSet<String>(); @@ -419,10 +427,10 @@ public class ExoneraTorDatabaseImporter { addressHexString = addressHexString.replaceFirst("x", String.format("%" + (33 - addressHexString.length()) + "s", "0")); - if (!addressHexString.contains("x") && - addressHexString.length() == 32) { - orAddress48 = addressHexString.replaceAll(" ", "0"). - toLowerCase().substring(0, 12); + if (!addressHexString.contains("x") + && addressHexString.length() == 32) { + orAddress48 = addressHexString.replaceAll(" ", "0") + .toLowerCase().substring(0, 12); } } if (orAddress48 != null) { @@ -455,8 +463,8 @@ public class ExoneraTorDatabaseImporter { StringBuilder rawExitlistentryBuilder = new StringBuilder(); while (true) { String line = br.readLine(); - if ((line == null || line.startsWith("ExitNode ")) && - fingerprint != null) { + if ((line == null || line.startsWith("ExitNode ")) + && fingerprint != null) { for (String exitAddressLine : exitAddressLines) { String[] parts = exitAddressLine.split(" "); String exitAddress = parts[1]; @@ -481,8 +489,8 @@ public class ExoneraTorDatabaseImporter { + "'" + file + "'. Skipping."); return; } - byte[] rawExitlistentry = rawExitlistentryBuilder.toString(). - getBytes(); + byte[] rawExitlistentry = rawExitlistentryBuilder.toString() + .getBytes(); importExitlistentry(fingerprint, exitAddress24, exitAddress, scannedMillis, rawExitlistentry); } @@ -494,8 +502,8 @@ public class ExoneraTorDatabaseImporter { } rawExitlistentryBuilder.append(line + "\n"); if (line.startsWith("ExitNode ")) { - fingerprint = line.substring("ExitNode ".length()). - toLowerCase(); + fingerprint = line.substring("ExitNode ".length()) + .toLowerCase(); } else if (line.startsWith("ExitAddress ")) { exitAddressLines.add(line); } diff --git a/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java b/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java index 68d79a3..f637150 100644 --- a/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java +++ b/src/main/java/org/torproject/exonerator/ExoneraTorServlet.java @@ -1,7 +1,11 @@ -/* Copyright 2011--2015 The Tor Project +/* Copyright 2011--2016 The Tor Project * See LICENSE for licensing information */ + package org.torproject.exonerator; +import org.apache.commons.codec.binary.Hex; +import org.apache.commons.lang.StringEscapeUtils; + import java.io.IOException; import java.io.PrintWriter; import java.sql.CallableStatement; @@ -32,9 +36,6 @@ import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import javax.sql.DataSource; -import org.apache.commons.codec.binary.Hex; -import org.apache.commons.lang.StringEscapeUtils; - public class ExoneraTorServlet extends HttpServlet { private static final long serialVersionUID = 1370088989739567509L; @@ -113,8 +114,8 @@ public class ExoneraTorServlet extends HttpServlet { if (timestampStr != null && timestampStr.length() > 0) { try { timestamp = dateFormat.parse(timestampParameter).getTime(); - if (timestamp < firstAndLastDates[0] || - timestamp > firstAndLastDates[1]) { + if (timestamp < firstAndLastDates[0] + || timestamp > firstAndLastDates[1]) { timestampOutOfRange = true; } } catch (ParseException e) { @@ -123,12 +124,12 @@ public class ExoneraTorServlet extends HttpServlet { } /* Write form. */ - this.writeForm(out, rb, relayIP, relayIPHasError || - ("".equals(relayIP) && !"".equals(timestampStr)), timestampStr, - !relayIPHasError && - !("".equals(relayIP) && !"".equals(timestampStr)) && - (timestampHasError || timestampOutOfRange || - (!"".equals(relayIP) && "".equals(timestampStr)))); + this.writeForm(out, rb, relayIP, relayIPHasError + || ("".equals(relayIP) && !"".equals(timestampStr)), timestampStr, + !relayIPHasError + && !("".equals(relayIP) && !"".equals(timestampStr)) + && (timestampHasError || timestampOutOfRange + || (!"".equals(relayIP) && "".equals(timestampStr)))); /* If both parameters are empty, don't print any summary and exit. * This is the start page. */ @@ -219,8 +220,8 @@ public class ExoneraTorServlet extends HttpServlet { this.writeSummaryPositive(out, rb, relayIP, timestampStr); this.writeTechnicalDetails(out, rb, relayIP, timestampStr, statusEntries); - } else if (addressesInSameNetwork != null && - !addressesInSameNetwork.isEmpty()) { + } else if (addressesInSameNetwork != null + && !addressesInSameNetwork.isEmpty()) { this.writeSummaryAddressesInSameNetwork(out, rb, relayIP, timestampStr, addressesInSameNetwork); } else { @@ -240,10 +241,10 @@ public class ExoneraTorServlet extends HttpServlet { if (passedIpParameter != null && passedIpParameter.length() > 0) { String ipParameter = passedIpParameter.trim(); Pattern ipv4AddressPattern = Pattern.compile( - "^([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + - "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + - "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + - "([01]?\\d\\d?|2[0-4]\\d|25[0-5])$"); + "^([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + + "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + + "([01]?\\d\\d?|2[0-4]\\d|25[0-5])\\." + + "([01]?\\d\\d?|2[0-4]\\d|25[0-5])$"); Pattern ipv6AddressPattern = Pattern.compile( "^\\[?[0-9a-fA-F:]{3,39}\\]?$"); if (ipv4AddressPattern.matcher(ipParameter).matches()) { @@ -278,8 +279,8 @@ public class ExoneraTorServlet extends HttpServlet { addressHexString = addressHexString.replaceFirst("x", String.format("%" + (33 - addressHexString.length()) + "s", "0")); - if (!addressHexString.contains("x") && - addressHexString.length() == 32) { + if (!addressHexString.contains("x") + && addressHexString.length() == 32) { relayIP = ipParameter.toLowerCase(); } } @@ -325,10 +326,10 @@ public class ExoneraTorServlet extends HttpServlet { addressHexString = addressHexString.replaceFirst("x", String.format("%" + (33 - addressHexString.length()) + "s", "0")); - if (!addressHexString.contains("x") && - addressHexString.length() == 32) { - address48 = addressHexString.replaceAll(" ", "0"). - toLowerCase(); + if (!addressHexString.contains("x") + && addressHexString.length() == 32) { + address48 = addressHexString.replaceAll(" ", "0") + .toLowerCase(); } } return address48; @@ -340,8 +341,8 @@ public class ExoneraTorServlet extends HttpServlet { SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd"); dateFormat.setTimeZone(TimeZone.getTimeZone("UTC")); dateFormat.setLenient(false); - if (passedTimestampParameter != null && - passedTimestampParameter.length() > 0) { + if (passedTimestampParameter != null + && passedTimestampParameter.length() > 0) { String timestampParameter = passedTimestampParameter.trim(); try { long timestamp = dateFormat.parse(timestampParameter).getTime(); @@ -436,8 +437,8 @@ public class ExoneraTorServlet extends HttpServlet { ResultSet rs = cs.executeQuery(); while (rs.next()) { byte[] rawstatusentry = rs.getBytes(1); - SortedSet<String> addresses = new TreeSet<String>(), - addressesHex = new TreeSet<String>(); + SortedSet<String> addresses = new TreeSet<String>(); + SortedSet<String> addressesHex = new TreeSet<String>(); long validafter = rs.getTimestamp(2, utcCalendar).getTime(); String validAfterString = validAfterTimeFormat.format(validafter); String fingerprint = rs.getString(3).toUpperCase(); @@ -627,8 +628,8 @@ public class ExoneraTorServlet extends HttpServlet { ipValue, timestampHasError ? " has-error" : "", rb.getString("form.timestamp.label"), - timestampStr != null && timestampStr.length() > 0 ? - " value=\"" + timestampStr + "\"" : "", + timestampStr != null && timestampStr.length() > 0 + ? " value=\"" + timestampStr + "\"" : "", rb.getString("form.search.label")); } @@ -685,8 +686,8 @@ public class ExoneraTorServlet extends HttpServlet { private void writeSummaryInvalidIp(PrintWriter out, ResourceBundle rb, String ipParameter) throws IOException { - String escapedIpParameter = ipParameter.length() > 40 ? - StringEscapeUtils.escapeHtml(ipParameter.substring(0, 40)) + String escapedIpParameter = ipParameter.length() > 40 + ? StringEscapeUtils.escapeHtml(ipParameter.substring(0, 40)) + "[...]" : StringEscapeUtils.escapeHtml(ipParameter); this.writeSummary(out, rb.getString("summary.heading"), "panel-danger", @@ -697,10 +698,10 @@ public class ExoneraTorServlet extends HttpServlet { private void writeSummaryInvalidTimestamp(PrintWriter out, ResourceBundle rb, String timestampParameter) throws IOException { - String escapedTimestampParameter = timestampParameter.length() > 20 ? - StringEscapeUtils.escapeHtml(timestampParameter. - substring(0, 20)) + "[...]" : - StringEscapeUtils.escapeHtml(timestampParameter); + String escapedTimestampParameter = timestampParameter.length() > 20 + ? StringEscapeUtils.escapeHtml(timestampParameter + .substring(0, 20)) + "[...]" + : StringEscapeUtils.escapeHtml(timestampParameter); this.writeSummary(out, rb.getString("summary.heading"), "panel-danger", rb.getString("summary.invalidparams.invalidtimestamp.title"), @@ -727,7 +728,8 @@ public class ExoneraTorServlet extends HttpServlet { Object[][] panelItems = new Object[addressesInSameNetwork.size()][]; for (int i = 0; i < addressesInSameNetwork.size(); i++) { String addressInSameNetwork = addressesInSameNetwork.get(i); - String link, address; + String link; + String address; if (addressInSameNetwork.contains(":")) { link = String.format("/?ip=[%s]&timestamp=%s", addressInSameNetwork.replaceAll(":", "%3A"), timestampStr); @@ -748,8 +750,8 @@ public class ExoneraTorServlet extends HttpServlet { private void writeSummaryPositive(PrintWriter out, ResourceBundle rb, String relayIP, String timestampStr) throws IOException { - String formattedRelayIP = relayIP.contains(":") ? - "[" + relayIP + "]" : relayIP; + String formattedRelayIP = relayIP.contains(":") + ? "[" + relayIP + "]" : relayIP; this.writeSummary(out, rb.getString("summary.heading"), "panel-success", rb.getString("summary.positive.title"), null, rb.getString("summary.positive.body"), formattedRelayIP, @@ -758,8 +760,8 @@ public class ExoneraTorServlet extends HttpServlet { private void writeSummaryNegative(PrintWriter out, ResourceBundle rb, String relayIP, String timestampStr) throws IOException { - String formattedRelayIP = relayIP.contains(":") ? - "[" + relayIP + "]" : relayIP; + String formattedRelayIP = relayIP.contains(":") + ? "[" + relayIP + "]" : relayIP; this.writeSummary(out, rb.getString("summary.heading"), "panel-warning", rb.getString("summary.negative.title"), null, rb.getString("summary.negative.body"), formattedRelayIP, @@ -797,8 +799,8 @@ public class ExoneraTorServlet extends HttpServlet { private void writeTechnicalDetails(PrintWriter out, ResourceBundle rb, String relayIP, String timestampStr, List<String[]> tableRows) throws IOException { - String formattedRelayIP = relayIP.contains(":") ? - "[" + relayIP + "]" : relayIP; + String formattedRelayIP = relayIP.contains(":") + ? "[" + relayIP + "]" : relayIP; out.printf(" <div class=\"row\">\n" + " <div class=\"col-xs-12\">\n" + " <h2>%s</h2>\n" @@ -853,8 +855,8 @@ public class ExoneraTorServlet extends HttpServlet { private void writePermanentLink(PrintWriter out, ResourceBundle rb, String relayIP, String timestampStr) throws IOException { - String encodedAddress = relayIP.contains(":") ? - "[" + relayIP.replaceAll(":", "%3A") + "]" : relayIP; + String encodedAddress = relayIP.contains(":") + ? "[" + relayIP.replaceAll(":", "%3A") + "]" : relayIP; out.printf(" <div class=\"row\">\n" + " <div class=\"col-xs-12\">\n" + " <h2>%s</h2>\n"

1 0

[webwml/master] tup isn't our only pseudonym anymore
by sebastian＠torproject.org 07 Jul '16

07 Jul '16

commit 34d753af205e14f542fd0c4ee354041812b72b8a Author: Sebastian Hahn <sebastian(a)torproject.org> Date: Thu Jul 7 12:43:17 2016 +0200 tup isn't our only pseudonym anymore --- about/en/contributors.wml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/about/en/contributors.wml b/about/en/contributors.wml index 31be450..e90052d 100644 --- a/about/en/contributors.wml +++ b/about/en/contributors.wml @@ -170,8 +170,8 @@ href="<page docs/torbutton/index>">Torbutton</a>.</dd> <dd>Worked on the artwork and design for various projects, annual reports, and brochures.</dd> -<dt>tup (a pseudonym)</dt><dd>Added new features for -making Tor easier to use as a <a +<dt>tup</dt> +<dd>Added new features for making Tor easier to use as a <a href="<wiki>TransparentProxy">transparent proxy</a>. Also wrote the <a href="http://p56soo2ibjkx23xo.onion/">TorDNSEL code</a>.</dd>

1 0