- tor-commits - lists.torproject.org

[metrics-web/master] Fix network-status page.
by karsten＠torproject.org 19 Jan '12

19 Jan '12

commit 752ce259302ce7522a0852eb289fd23ee979f0b5 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 19 20:34:33 2012 +0100 Fix network-status page. --- .../torproject/ernie/web/NetworkStatusServlet.java | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/org/torproject/ernie/web/NetworkStatusServlet.java b/src/org/torproject/ernie/web/NetworkStatusServlet.java index 7d4abd1..c206a19 100644 --- a/src/org/torproject/ernie/web/NetworkStatusServlet.java +++ b/src/org/torproject/ernie/web/NetworkStatusServlet.java @@ -100,7 +100,7 @@ public class NetworkStatusServlet extends HttpServlet { + "descriptor.platform FROM statusentry JOIN descriptor " + "ON descriptor.descriptor = statusentry.descriptor " + "WHERE statusentry.validafter = " - + "(SELECT MAX(validafter) FROM statusentry) " + + "(SELECT MAX(validafter) FROM consensus) " + "ORDER BY " + orderBy + " " + orderParameter.toUpperCase(); ResultSet rs = statement.executeQuery(query);

1 0

r25327: {website} now that the flood is over, get rid of the sopa-pipa page, w (website/trunk/en)
by Roger Dingledine 19 Jan '12

19 Jan '12

Author: arma Date: 2012-01-19 17:57:45 +0000 (Thu, 19 Jan 2012) New Revision: 25327 Modified: website/trunk/en/sopa-pipa.wml Log: now that the flood is over, get rid of the sopa-pipa page, while still letting it function for those who linked to it Modified: website/trunk/en/sopa-pipa.wml =================================================================== --- website/trunk/en/sopa-pipa.wml 2012-01-19 08:06:20 UTC (rev 25326) +++ website/trunk/en/sopa-pipa.wml 2012-01-19 17:57:45 UTC (rev 25327) @@ -1,68 +1,8 @@ ## translation metadata -# Revision: $Revision: 25296 $ -# Translation-Priority: 2-medium +# Revision: $Revision: 23689 $ +# Status: obsolete -#include "head.wmi" TITLE="Blackout Against Copyright Overreach: Stop SOPA and PIPA" CHARSET="UTF-8" +#include "head.wmi" TITLE="Redirecting" REDIRECT_GLOBAL="https://blog.torproject.org/blog/blackout-against-copyright-overreach-stop-…" -<div id="content" class="clearfix"> - <div id="maincol"> -  - <h1><a href="https://blog.torproject.org/blog/blackout-against-copyright-overreach-stop-…">Blackout Against Copyright Overreach: Stop SOPA and PIPA</a></h1> -<hr> -<p> -The Tor Project doesn't usually get involved with U.S. copyright -debates. But SOPA and PIPA (the House's "Stop Online Piracy Act" and the -Senate's "Protect-IP Act") go beyond enforcement of copyright. These -copyright bills would strain the infrastructure of the Internet, on -which many free communications -- anonymous or identified -- depend. -Originally, the bills proposed that so-called "rogue sites" should be -blocked through the Internet's Domain Name System (DNS). That would have -broken DNSSEC security and shared U.S. censorship tactics with those of -China's "great firewall." -</p> - -<p> -Now, while we hear that DNS-blocking is off the table, the bills remain -threatening to the network of intermediaries who carry online speech. Most -critically to Tor, SOPA contained a provision forbidding "circumvention" -of court-ordered blocking that was written broadly enough that it <a -href="http://wendy.seltzer.org/blog/archives/2011/12/15/stopping-sopas-anti-circu…">could -apply to Tor</a> -- which helps its users to "circumvent" -local-network censorship. Further, both bills broaden the reach of -intermediary liability, to hold conduits and search engines liable -for user-supplied infringement. The private rights of action and -"safe harbors" could force or encourage providers to censor well -beyond the current DMCA's "notice and takedown" provision (of which <a -href="https://www.chillingeffects.org/">Chilling Effects</a> documents -numerous burdens and abuses). -</p> - -<p> -On January 18, we're joining <a -href="https://blog.wikimedia.org/2012/01/16/wikipedias-community-calls-for-anti-s…">Wikipedia</a>, -<a -href="http://blog.reddit.com/2012/01/stopped-they-must-be-on-this-all.html">Reddit</a>, -the <a -href="http://blog.media.mit.edu/2012/01/media-lab-is-against-sopa-and-pipa.html">MIT -Media Lab</a>, and <a href="http://americancensorship.org/">hundreds of -others</a> in protest, turning a portion of the Tor site black to call -attention to copyright balance and remind the US Congress and voters of -the value of the open Internet. -</p> - -<p> -U.S. citizens, please call or write, to <a -href="http://americancensorship.org/">urge your representatives to stop -SOPA and PIPA</a>. Elsewhere in the world, keep an eye out for similar -legislation. and bring the fight there too. -</p> - - <hr> - <h1>The above is a static version of our January 18 <a href="https://blog.torproject.org/blog/blackout-against-copyright-overreach-stop-…">blog post</a>.</h1> - - </div> -  -</div> - #include <foot.wmi>

1 0

[metrics-lib/master] Use a separate JVM for running the unit tests.
by karsten＠torproject.org 19 Jan '12

19 Jan '12

commit 9ceea770ec1b6b76c59c76d6e6a9e91081f82ce4 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 19 17:47:03 2012 +0100 Use a separate JVM for running the unit tests. Otherwise we'll run into weird problems with Ant when overriding Java's URL stream handler to test the downloading code (not merged yet). --- build.xml | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/build.xml b/build.xml index 83cd991..0f2539a 100644 --- a/build.xml +++ b/build.xml @@ -45,7 +45,7 @@ <fileset dir="${libs}"/> </classpath> </javac> - <junit haltonfailure="true" printsummary="off"> + <junit fork="true" haltonfailure="true" printsummary="off"> <classpath refid="classpath"/> <formatter type="plain" usefile="false"/> <batchtest>

1 0

[metrics-lib/master] Make DownloadCoordinator interface to facilitate testing.
by karsten＠torproject.org 19 Jan '12

19 Jan '12

commit bc9925b34030c4bba62fd0bda248a720e7da2989 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 19 15:21:50 2012 +0100 Make DownloadCoordinator interface to facilitate testing. --- .../descriptor/impl/DownloadCoordinator.java | 263 +------------------- .../descriptor/impl/DownloadCoordinatorImpl.java | 267 ++++++++++++++++++++ .../impl/RelayDescriptorDownloaderImpl.java | 11 +- 3 files changed, 276 insertions(+), 265 deletions(-) diff --git a/src/org/torproject/descriptor/impl/DownloadCoordinator.java b/src/org/torproject/descriptor/impl/DownloadCoordinator.java index 362006d..6d837a5 100644 --- a/src/org/torproject/descriptor/impl/DownloadCoordinator.java +++ b/src/org/torproject/descriptor/impl/DownloadCoordinator.java @@ -2,266 +2,9 @@ * See LICENSE for licensing information */ package org.torproject.descriptor.impl; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.Iterator; -import java.util.List; -import java.util.Map; -import java.util.Set; -import java.util.SortedMap; +public interface DownloadCoordinator { -import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorRequest; -import org.torproject.descriptor.DirSourceEntry; -import org.torproject.descriptor.RelayNetworkStatusConsensus; -import org.torproject.descriptor.RelayNetworkStatusVote; + public DescriptorRequestImpl createRequest(String nickname); -/* TODO This whole download logic is a mess and needs a cleanup. */ -public class DownloadCoordinator { - - private BlockingIteratorImpl<DescriptorRequest> descriptorQueue = - new BlockingIteratorImpl<DescriptorRequest>(); - protected Iterator<DescriptorRequest> getDescriptorQueue() { - return this.descriptorQueue; - } - - private SortedMap<String, DirectoryDownloader> directoryAuthorities; - private SortedMap<String, DirectoryDownloader> directoryMirrors; - private boolean downloadConsensusFromAllAuthorities; - private boolean includeCurrentReferencedVotes; - private long requestTimeoutMillis; - private long globalTimeoutMillis; - - protected DownloadCoordinator( - SortedMap<String, DirectoryDownloader> directoryAuthorities, - SortedMap<String, DirectoryDownloader> directoryMirrors, - boolean downloadConsensus, - boolean downloadConsensusFromAllAuthorities, - Set<String> downloadVotes, boolean includeCurrentReferencedVotes, - long requestTimeoutMillis, long globalTimeoutMillis) { - this.directoryAuthorities = directoryAuthorities; - this.directoryMirrors = directoryMirrors; - this.missingConsensus = downloadConsensus; - this.downloadConsensusFromAllAuthorities = - downloadConsensusFromAllAuthorities; - this.missingVotes = downloadVotes; - this.includeCurrentReferencedVotes = includeCurrentReferencedVotes; - this.requestTimeoutMillis = requestTimeoutMillis; - this.globalTimeoutMillis = globalTimeoutMillis; - if (this.directoryMirrors.isEmpty() && - this.directoryAuthorities.isEmpty()) { - this.descriptorQueue.setOutOfDescriptors(); - /* TODO Should we say anything if we don't have any directories - * configured? */ - } else { - GlobalTimer globalTimer = new GlobalTimer(this.globalTimeoutMillis, - this); - this.globalTimerThread = new Thread(globalTimer); - this.globalTimerThread.start(); - for (DirectoryDownloader directoryMirror : - this.directoryMirrors.values()) { - directoryMirror.setDownloadCoordinator(this); - directoryMirror.setRequestTimeout(this.requestTimeoutMillis); - new Thread(directoryMirror).start(); - } - for (DirectoryDownloader directoryAuthority : - this.directoryAuthorities.values()) { - directoryAuthority.setDownloadCoordinator(this); - directoryAuthority.setRequestTimeout(this.requestTimeoutMillis); - new Thread(directoryAuthority).start(); - } - } - } - - /* Interrupt all downloads if the total download time exceeds a given - * time. */ - private Thread globalTimerThread; - private static class GlobalTimer implements Runnable { - private long timeoutMillis; - private DownloadCoordinator downloadCoordinator; - private GlobalTimer(long timeoutMillis, - DownloadCoordinator downloadCoordinator) { - this.timeoutMillis = timeoutMillis; - this.downloadCoordinator = downloadCoordinator; - } - public void run() { - long started = System.currentTimeMillis(), sleep; - while ((sleep = started + this.timeoutMillis - - System.currentTimeMillis()) > 0L) { - try { - Thread.sleep(sleep); - } catch (InterruptedException e) { - return; - } - } - this.downloadCoordinator.interruptAllDownloads(); - } - } - - /* Are we missing the consensus, and should the next directory that - * hasn't tried downloading it before attempt to download it? */ - private boolean missingConsensus = false; - - /* Which directories are currently attempting to download the - * consensus? */ - private Set<String> requestingConsensuses = new HashSet<String>(); - - /* Which directories have attempted to download the consensus so far, - * including those directories that are currently attempting it? */ - private Set<String> requestedConsensuses = new HashSet<String>(); - - /* Which votes are we currently missing? */ - private Set<String> missingVotes = new HashSet<String>(); - - /* Which vote (map value) is a given directory (map key) currently - * attempting to download? */ - private Map<String, String> requestingVotes = - new HashMap<String, String>(); - - /* Which votes (map value) has a given directory (map key) attempted or - * is currently attempting to download? */ - private Map<String, Set<String>> requestedVotes = - new HashMap<String, Set<String>>(); - - private boolean hasFinishedDownloading = false; - - /* Look up what request a directory should make next. If there is - * nothing to do right now, but maybe later, block the caller. If - * we're done downloading, return null to notify the caller. */ - protected synchronized DescriptorRequestImpl createRequest( - String nickname) { - while (!this.hasFinishedDownloading) { - DescriptorRequestImpl request = new DescriptorRequestImpl(); - request.setDirectoryNickname(nickname); - if ((this.missingConsensus || - (this.downloadConsensusFromAllAuthorities && - this.directoryAuthorities.containsKey(nickname))) && - !this.requestedConsensuses.contains(nickname)) { - if (!this.downloadConsensusFromAllAuthorities) { - this.missingConsensus = false; - } - this.requestingConsensuses.add(nickname); - this.requestedConsensuses.add(nickname); - request.setRequestedResource( - "/tor/status-vote/current/consensus.z"); - request.setDescriptorType("consensus"); - return request; - } - if (!this.missingVotes.isEmpty() && - this.directoryAuthorities.containsKey(nickname)) { - String requestingVote = null; - for (String missingVote : this.missingVotes) { - if (!this.requestedVotes.containsKey(nickname) || - !this.requestedVotes.get(nickname).contains(missingVote)) { - requestingVote = missingVote; - } - } - if (requestingVote != null) { - this.requestingVotes.put(nickname, requestingVote); - if (!this.requestedVotes.containsKey(nickname)) { - this.requestedVotes.put(nickname, new HashSet<String>()); - } - this.requestedVotes.get(nickname).add(requestingVote); - this.missingVotes.remove(requestingVote); - request.setRequestedResource("/tor/status-vote/current/" - + requestingVote + ".z"); - request.setDescriptorType("vote"); - return request; - } - } - /* TODO Add server descriptors and extra-info descriptors later. */ - try { - this.wait(); - } catch (InterruptedException e) { - /* TODO What shall we do? */ - } - } - return null; - } - - /* Deliver a response which may either contain one or more descriptors - * or a failure response code. Update the lists of missing descriptors, - * decide if there are more descriptors to download, and wake up any - * waiting downloader threads. */ - protected synchronized void deliverResponse( - DescriptorRequestImpl response) { - String nickname = response.getDirectoryNickname(); - if (response.getDescriptorType().equals("consensus")) { - this.requestingConsensuses.remove(nickname); - if (response.getResponseCode() == 200) { - List<RelayNetworkStatusConsensus> parsedConsensuses = - RelayNetworkStatusConsensusImpl.parseConsensuses( - response.getResponseBytes()); - List<Descriptor> parsedDescriptors = - new ArrayList<Descriptor>(parsedConsensuses); - response.setDescriptors(parsedDescriptors); - if (this.includeCurrentReferencedVotes) { - /* TODO Only add votes if the consensus is not older than one - * hour. Or does that make no sense? */ - for (RelayNetworkStatusConsensus parsedConsensus : - parsedConsensuses) { - for (DirSourceEntry dirSource : - parsedConsensus.getDirSourceEntries().values()) { - String identity = dirSource.getIdentity(); - if (!this.missingVotes.contains(identity)) { - boolean alreadyRequested = false; - for (Set<String> requestedBefore : - this.requestedVotes.values()) { - if (requestedBefore.contains(identity)) { - alreadyRequested = true; - break; - } - } - if (!alreadyRequested) { - this.missingVotes.add(identity); - } - } - } - } - /* TODO Later, add referenced server descriptors. */ - } - } else { - this.missingConsensus = true; - } - } else if (response.getDescriptorType().equals("vote")) { - String requestedVote = requestingVotes.remove(nickname); - if (response.getResponseCode() == 200) { - List<RelayNetworkStatusVote> parsedVotes = - RelayNetworkStatusVoteImpl.parseVotes( - response.getResponseBytes()); - List<Descriptor> parsedDescriptors = - new ArrayList<Descriptor>(parsedVotes); - response.setDescriptors(parsedDescriptors); - } else { - this.missingVotes.add(requestedVote); - } - } - if (response.getRequestEnd() != 0L) { - this.descriptorQueue.add(response); - } - if ((!this.missingConsensus || - (this.downloadConsensusFromAllAuthorities && - this.requestedConsensuses.containsAll( - this.directoryAuthorities.keySet()) && - this.requestingConsensuses.isEmpty())) && - this.missingVotes.isEmpty() && - this.requestingVotes.isEmpty()) { - /* TODO This logic may be somewhat broken. We don't wait for all - * consensus requests to complete or fail, which results in adding - * (failed) requests to the queue when we think we're done. */ - this.hasFinishedDownloading = true; - this.globalTimerThread.interrupt(); - this.descriptorQueue.setOutOfDescriptors(); - } - /* Wake up all waiting downloader threads. Maybe they can now - * download something, or they'll realize we're done downloading. */ - this.notifyAll(); - } - - private synchronized void interruptAllDownloads() { - this.hasFinishedDownloading = true; - this.notifyAll(); - } + public void deliverResponse(DescriptorRequestImpl request); } - diff --git a/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java b/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java new file mode 100644 index 0000000..5d4ca21 --- /dev/null +++ b/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java @@ -0,0 +1,267 @@ +/* Copyright 2011, 2012 The Tor Project + * See LICENSE for licensing information */ +package org.torproject.descriptor.impl; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.SortedMap; + +import org.torproject.descriptor.Descriptor; +import org.torproject.descriptor.DescriptorRequest; +import org.torproject.descriptor.DirSourceEntry; +import org.torproject.descriptor.RelayNetworkStatusConsensus; +import org.torproject.descriptor.RelayNetworkStatusVote; + +/* TODO This whole download logic is a mess and needs a cleanup. */ +public class DownloadCoordinatorImpl implements DownloadCoordinator { + + private BlockingIteratorImpl<DescriptorRequest> descriptorQueue = + new BlockingIteratorImpl<DescriptorRequest>(); + protected Iterator<DescriptorRequest> getDescriptorQueue() { + return this.descriptorQueue; + } + + private SortedMap<String, DirectoryDownloader> directoryAuthorities; + private SortedMap<String, DirectoryDownloader> directoryMirrors; + private boolean downloadConsensusFromAllAuthorities; + private boolean includeCurrentReferencedVotes; + private long requestTimeoutMillis; + private long globalTimeoutMillis; + + protected DownloadCoordinatorImpl( + SortedMap<String, DirectoryDownloader> directoryAuthorities, + SortedMap<String, DirectoryDownloader> directoryMirrors, + boolean downloadConsensus, + boolean downloadConsensusFromAllAuthorities, + Set<String> downloadVotes, boolean includeCurrentReferencedVotes, + long requestTimeoutMillis, long globalTimeoutMillis) { + this.directoryAuthorities = directoryAuthorities; + this.directoryMirrors = directoryMirrors; + this.missingConsensus = downloadConsensus; + this.downloadConsensusFromAllAuthorities = + downloadConsensusFromAllAuthorities; + this.missingVotes = downloadVotes; + this.includeCurrentReferencedVotes = includeCurrentReferencedVotes; + this.requestTimeoutMillis = requestTimeoutMillis; + this.globalTimeoutMillis = globalTimeoutMillis; + if (this.directoryMirrors.isEmpty() && + this.directoryAuthorities.isEmpty()) { + this.descriptorQueue.setOutOfDescriptors(); + /* TODO Should we say anything if we don't have any directories + * configured? */ + } else { + GlobalTimer globalTimer = new GlobalTimer(this.globalTimeoutMillis, + this); + this.globalTimerThread = new Thread(globalTimer); + this.globalTimerThread.start(); + for (DirectoryDownloader directoryMirror : + this.directoryMirrors.values()) { + directoryMirror.setDownloadCoordinator(this); + directoryMirror.setRequestTimeout(this.requestTimeoutMillis); + new Thread(directoryMirror).start(); + } + for (DirectoryDownloader directoryAuthority : + this.directoryAuthorities.values()) { + directoryAuthority.setDownloadCoordinator(this); + directoryAuthority.setRequestTimeout(this.requestTimeoutMillis); + new Thread(directoryAuthority).start(); + } + } + } + + /* Interrupt all downloads if the total download time exceeds a given + * time. */ + private Thread globalTimerThread; + private static class GlobalTimer implements Runnable { + private long timeoutMillis; + private DownloadCoordinatorImpl downloadCoordinator; + private GlobalTimer(long timeoutMillis, + DownloadCoordinatorImpl downloadCoordinator) { + this.timeoutMillis = timeoutMillis; + this.downloadCoordinator = downloadCoordinator; + } + public void run() { + long started = System.currentTimeMillis(), sleep; + while ((sleep = started + this.timeoutMillis + - System.currentTimeMillis()) > 0L) { + try { + Thread.sleep(sleep); + } catch (InterruptedException e) { + return; + } + } + this.downloadCoordinator.interruptAllDownloads(); + } + } + + /* Are we missing the consensus, and should the next directory that + * hasn't tried downloading it before attempt to download it? */ + private boolean missingConsensus = false; + + /* Which directories are currently attempting to download the + * consensus? */ + private Set<String> requestingConsensuses = new HashSet<String>(); + + /* Which directories have attempted to download the consensus so far, + * including those directories that are currently attempting it? */ + private Set<String> requestedConsensuses = new HashSet<String>(); + + /* Which votes are we currently missing? */ + private Set<String> missingVotes = new HashSet<String>(); + + /* Which vote (map value) is a given directory (map key) currently + * attempting to download? */ + private Map<String, String> requestingVotes = + new HashMap<String, String>(); + + /* Which votes (map value) has a given directory (map key) attempted or + * is currently attempting to download? */ + private Map<String, Set<String>> requestedVotes = + new HashMap<String, Set<String>>(); + + private boolean hasFinishedDownloading = false; + + /* Look up what request a directory should make next. If there is + * nothing to do right now, but maybe later, block the caller. If + * we're done downloading, return null to notify the caller. */ + public synchronized DescriptorRequestImpl createRequest( + String nickname) { + while (!this.hasFinishedDownloading) { + DescriptorRequestImpl request = new DescriptorRequestImpl(); + request.setDirectoryNickname(nickname); + if ((this.missingConsensus || + (this.downloadConsensusFromAllAuthorities && + this.directoryAuthorities.containsKey(nickname))) && + !this.requestedConsensuses.contains(nickname)) { + if (!this.downloadConsensusFromAllAuthorities) { + this.missingConsensus = false; + } + this.requestingConsensuses.add(nickname); + this.requestedConsensuses.add(nickname); + request.setRequestedResource( + "/tor/status-vote/current/consensus.z"); + request.setDescriptorType("consensus"); + return request; + } + if (!this.missingVotes.isEmpty() && + this.directoryAuthorities.containsKey(nickname)) { + String requestingVote = null; + for (String missingVote : this.missingVotes) { + if (!this.requestedVotes.containsKey(nickname) || + !this.requestedVotes.get(nickname).contains(missingVote)) { + requestingVote = missingVote; + } + } + if (requestingVote != null) { + this.requestingVotes.put(nickname, requestingVote); + if (!this.requestedVotes.containsKey(nickname)) { + this.requestedVotes.put(nickname, new HashSet<String>()); + } + this.requestedVotes.get(nickname).add(requestingVote); + this.missingVotes.remove(requestingVote); + request.setRequestedResource("/tor/status-vote/current/" + + requestingVote + ".z"); + request.setDescriptorType("vote"); + return request; + } + } + /* TODO Add server descriptors and extra-info descriptors later. */ + try { + this.wait(); + } catch (InterruptedException e) { + /* TODO What shall we do? */ + } + } + return null; + } + + /* Deliver a response which may either contain one or more descriptors + * or a failure response code. Update the lists of missing descriptors, + * decide if there are more descriptors to download, and wake up any + * waiting downloader threads. */ + public synchronized void deliverResponse( + DescriptorRequestImpl response) { + String nickname = response.getDirectoryNickname(); + if (response.getDescriptorType().equals("consensus")) { + this.requestingConsensuses.remove(nickname); + if (response.getResponseCode() == 200) { + List<RelayNetworkStatusConsensus> parsedConsensuses = + RelayNetworkStatusConsensusImpl.parseConsensuses( + response.getResponseBytes()); + List<Descriptor> parsedDescriptors = + new ArrayList<Descriptor>(parsedConsensuses); + response.setDescriptors(parsedDescriptors); + if (this.includeCurrentReferencedVotes) { + /* TODO Only add votes if the consensus is not older than one + * hour. Or does that make no sense? */ + for (RelayNetworkStatusConsensus parsedConsensus : + parsedConsensuses) { + for (DirSourceEntry dirSource : + parsedConsensus.getDirSourceEntries().values()) { + String identity = dirSource.getIdentity(); + if (!this.missingVotes.contains(identity)) { + boolean alreadyRequested = false; + for (Set<String> requestedBefore : + this.requestedVotes.values()) { + if (requestedBefore.contains(identity)) { + alreadyRequested = true; + break; + } + } + if (!alreadyRequested) { + this.missingVotes.add(identity); + } + } + } + } + /* TODO Later, add referenced server descriptors. */ + } + } else { + this.missingConsensus = true; + } + } else if (response.getDescriptorType().equals("vote")) { + String requestedVote = requestingVotes.remove(nickname); + if (response.getResponseCode() == 200) { + List<RelayNetworkStatusVote> parsedVotes = + RelayNetworkStatusVoteImpl.parseVotes( + response.getResponseBytes()); + List<Descriptor> parsedDescriptors = + new ArrayList<Descriptor>(parsedVotes); + response.setDescriptors(parsedDescriptors); + } else { + this.missingVotes.add(requestedVote); + } + } + if (response.getRequestEnd() != 0L) { + this.descriptorQueue.add(response); + } + if ((!this.missingConsensus || + (this.downloadConsensusFromAllAuthorities && + this.requestedConsensuses.containsAll( + this.directoryAuthorities.keySet()) && + this.requestingConsensuses.isEmpty())) && + this.missingVotes.isEmpty() && + this.requestingVotes.isEmpty()) { + /* TODO This logic may be somewhat broken. We don't wait for all + * consensus requests to complete or fail, which results in adding + * (failed) requests to the queue when we think we're done. */ + this.hasFinishedDownloading = true; + this.globalTimerThread.interrupt(); + this.descriptorQueue.setOutOfDescriptors(); + } + /* Wake up all waiting downloader threads. Maybe they can now + * download something, or they'll realize we're done downloading. */ + this.notifyAll(); + } + + private synchronized void interruptAllDownloads() { + this.hasFinishedDownloading = true; + this.notifyAll(); + } +} + diff --git a/src/org/torproject/descriptor/impl/RelayDescriptorDownloaderImpl.java b/src/org/torproject/descriptor/impl/RelayDescriptorDownloaderImpl.java index 6ea81e1..7cdcc0c 100644 --- a/src/org/torproject/descriptor/impl/RelayDescriptorDownloaderImpl.java +++ b/src/org/torproject/descriptor/impl/RelayDescriptorDownloaderImpl.java @@ -227,11 +227,12 @@ public class RelayDescriptorDownloaderImpl + "permitted once."); } this.hasStartedDownloading = true; - DownloadCoordinator downloadCoordinator = new DownloadCoordinator( - this.directoryAuthorities, this.directoryMirrors, - this.downloadConsensus, this.downloadConsensusFromAllAuthorities, - this.downloadVotes, this.includeCurrentReferencedVotes, - this.requestTimeoutMillis, this.globalTimeoutMillis); + DownloadCoordinatorImpl downloadCoordinator = + new DownloadCoordinatorImpl(this.directoryAuthorities, + this.directoryMirrors, this.downloadConsensus, + this.downloadConsensusFromAllAuthorities, this.downloadVotes, + this.includeCurrentReferencedVotes, this.requestTimeoutMillis, + this.globalTimeoutMillis); Iterator<DescriptorRequest> descriptorQueue = downloadCoordinator. getDescriptorQueue(); return descriptorQueue;

1 0

[metrics-lib/master] Fix a few warnings found by Eclipse.
by karsten＠torproject.org 19 Jan '12

19 Jan '12

commit bc5aa5ab13225601277086f645ca9da2986dee32 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 19 13:30:02 2012 +0100 Fix a few warnings found by Eclipse. --- .../impl/RelayNetworkStatusConsensusImplTest.java | 6 ++++ .../descriptor/impl/ServerDescriptorImplTest.java | 25 ++++++++----------- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java b/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java index 5296512..5c55285 100644 --- a/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java +++ b/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java @@ -915,6 +915,12 @@ public class RelayNetworkStatusConsensusImplTest { } @Test(expected = DescriptorParseException.class) + public void testDirSourceNicknameTooLong() + throws DescriptorParseException { + DirSourceBuilder.createWithNickname("gabelmooisfinebutthisistoolong"); + } + + @Test(expected = DescriptorParseException.class) public void testDirSourceIdentityTooShort() throws DescriptorParseException { DirSourceBuilder.createWithIdentity("ED03BB616EB2F60BEC8015111"); diff --git a/test/org/torproject/descriptor/impl/ServerDescriptorImplTest.java b/test/org/torproject/descriptor/impl/ServerDescriptorImplTest.java index aac01a8..eec5594 100644 --- a/test/org/torproject/descriptor/impl/ServerDescriptorImplTest.java +++ b/test/org/torproject/descriptor/impl/ServerDescriptorImplTest.java @@ -516,6 +516,7 @@ public class ServerDescriptorImplTest { public void testUptimeOpt() throws DescriptorParseException { ServerDescriptor descriptor = DescriptorBuilder. createWithUptimeLine("opt uptime 48"); + assertEquals(48, (int) descriptor.getUptime()); } @Test(expected = DescriptorParseException.class) @@ -615,8 +616,7 @@ public class ServerDescriptorImplTest { @Test() public void testOnionKeyOpt() throws DescriptorParseException { - ServerDescriptor descriptor = DescriptorBuilder. - createWithOnionKeyLines("opt onion-key\n" + DescriptorBuilder.createWithOnionKeyLines("opt onion-key\n" + "-----BEGIN RSA PUBLIC KEY-----\n" + "MIGJAoGBAKM+iiHhO6eHsvd6Xjws9z9EQB1V/Bpuy5ciGJ1U4V9SeiKooSo5Bp" + "PL\no3XT+6PIgzl3R6uycjS3Ejk47vLEJdcVTm/VG6E0ppu3olIynCI4QryfCE" @@ -627,8 +627,7 @@ public class ServerDescriptorImplTest { @Test() public void testSigningKeyOpt() throws DescriptorParseException { - ServerDescriptor descriptor = DescriptorBuilder. - createWithSigningKeyLines("opt signing-key\n" + DescriptorBuilder.createWithSigningKeyLines("opt signing-key\n" + "-----BEGIN RSA PUBLIC KEY-----\n" + "MIGJAoGBALMm3r3QDh482Ewe6Ub9wvRIfmEkoNX6q5cEAtQRNHSDcNx41gjELb" + "cl\nEniVMParBYACKfOxkS+mTTnIRDKVNEJTsDOwryNrc4X9JnPc/nn6ymYPiN" @@ -946,9 +945,8 @@ public class ServerDescriptorImplTest { @Test(expected = DescriptorParseException.class) public void testWriteHistoryNoS() throws DescriptorParseException { - ServerDescriptor descriptor = DescriptorBuilder. - createWithWriteHistoryLine("write-history 2012-01-01 03:51:44 " - + "(900 "); + DescriptorBuilder.createWithWriteHistoryLine( + "write-history 2012-01-01 03:51:44 (900 "); } @Test(expected = DescriptorParseException.class) @@ -989,16 +987,15 @@ public class ServerDescriptorImplTest { assertTrue(bandwidthValues.isEmpty()); } - /* TODO There are some old server descriptors with " read-history" + /* TODO There are some old server descriptors with "read-history " * lines. Find out if these were spec-compliant and if other lines may * start with leading spaces, too. */ @Test(expected = DescriptorParseException.class) public void testReadHistoryLeadingSpace() throws DescriptorParseException { - String readHistoryLine = " read-history 2012-01-01 03:51:44 (900 s) " + String readHistoryLine = "read-history 2012-01-01 03:51:44 (900 s) " + "4268032,139264,7797760,1415168"; - ServerDescriptor descriptor = DescriptorBuilder. - createWithReadHistoryLine(readHistoryLine); + DescriptorBuilder.createWithReadHistoryLine(readHistoryLine); } @Test() @@ -1057,7 +1054,7 @@ public class ServerDescriptorImplTest { public void testAllowSingleHopExitsOpt() throws DescriptorParseException { ServerDescriptor descriptor = DescriptorBuilder. - createWithCachesExtraInfoLine("opt allow-single-hop-exits"); + createWithAllowSingleHopExitsLine("opt allow-single-hop-exits"); assertTrue(descriptor.getAllowSingleHopExits()); } @@ -1065,14 +1062,14 @@ public class ServerDescriptorImplTest { public void testAllowSingleHopExitsNoSpace() throws DescriptorParseException { ServerDescriptor descriptor = DescriptorBuilder. - createWithCachesExtraInfoLine("allow-single-hop-exits"); + createWithAllowSingleHopExitsLine("allow-single-hop-exits"); assertTrue(descriptor.getAllowSingleHopExits()); } @Test(expected = DescriptorParseException.class) public void testAllowSingleHopExitsTrue() throws DescriptorParseException { - DescriptorBuilder.createWithCachesExtraInfoLine( + DescriptorBuilder.createWithAllowSingleHopExitsLine( "allow-single-hop-exits true"); } }

1 0

[metrics-lib/master] Make ConsensusBuilder an own class for better testing.
by karsten＠torproject.org 19 Jan '12

19 Jan '12

commit befc2e3d2fa4e2c21e89e7b4a69ba64035463271 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 19 16:36:46 2012 +0100 Make ConsensusBuilder an own class for better testing. We'll want to create sample consensuses for testing the descriptor downloader and reader, not just for testing the parsing code. --- .../descriptor/impl/ConsensusBuilder.java | 232 ++++++++++++++++++++ .../impl/RelayNetworkStatusConsensusImplTest.java | 223 +------------------- 2 files changed, 233 insertions(+), 222 deletions(-) diff --git a/test/org/torproject/descriptor/impl/ConsensusBuilder.java b/test/org/torproject/descriptor/impl/ConsensusBuilder.java new file mode 100644 index 0000000..51fff2e --- /dev/null +++ b/test/org/torproject/descriptor/impl/ConsensusBuilder.java @@ -0,0 +1,232 @@ +/* Copyright 2012 The Tor Project + * See LICENSE for licensing information */ +package org.torproject.descriptor.impl; + +import java.util.ArrayList; +import java.util.List; + +import org.torproject.descriptor.RelayNetworkStatusConsensus; + +/* Helper class to build a consensus based on default data and + * modifications requested by test methods. */ +public class ConsensusBuilder { + String networkStatusVersionLine = "network-status-version 3"; + protected static RelayNetworkStatusConsensus + createWithNetworkStatusVersionLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.networkStatusVersionLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String voteStatusLine = "vote-status consensus"; + protected static RelayNetworkStatusConsensus + createWithVoteStatusLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.voteStatusLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String consensusMethodLine = "consensus-method 11"; + protected static RelayNetworkStatusConsensus + createWithConsensusMethodLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.consensusMethodLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String validAfterLine = "valid-after 2011-11-30 09:00:00"; + protected static RelayNetworkStatusConsensus + createWithValidAfterLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.validAfterLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String freshUntilLine = "fresh-until 2011-11-30 10:00:00"; + protected static RelayNetworkStatusConsensus + createWithFreshUntilLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.freshUntilLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String validUntilLine = "valid-until 2011-11-30 12:00:00"; + protected static RelayNetworkStatusConsensus + createWithValidUntilLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.validUntilLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String votingDelayLine = "voting-delay 300 300"; + protected static RelayNetworkStatusConsensus + createWithVotingDelayLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.votingDelayLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + String clientVersionsLine = "client-versions 0.2.1.31," + + "0.2.2.34,0.2.3.6-alpha,0.2.3.7-alpha,0.2.3.8-alpha"; + protected static RelayNetworkStatusConsensus + createWithClientVersionsLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.clientVersionsLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + String serverVersionsLine = "server-versions 0.2.1.31," + + "0.2.2.34,0.2.3.6-alpha,0.2.3.7-alpha,0.2.3.8-alpha"; + protected static RelayNetworkStatusConsensus + createWithServerVersionsLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.serverVersionsLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String knownFlagsLine = "known-flags Authority BadExit Exit " + + "Fast Guard HSDir Named Running Stable Unnamed V2Dir Valid"; + protected static RelayNetworkStatusConsensus + createWithKnownFlagsLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.knownFlagsLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String paramsLine = "params " + + "CircuitPriorityHalflifeMsec=30000 bwauthbestratio=1 " + + "bwauthcircs=1 bwauthdescbw=0 bwauthkp=10000 bwauthpid=1 " + + "bwauthtd=5000 bwauthti=50000 bwauthtidecay=5000 cbtnummodes=3 " + + "cbtquantile=80 circwindow=1000 refuseunknownexits=1"; + protected static RelayNetworkStatusConsensus + createWithParamsLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.paramsLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + List<String> dirSources = new ArrayList<String>(); + List<String> statusEntries = new ArrayList<String>(); + private String directoryFooterLine = "directory-footer"; + protected static RelayNetworkStatusConsensus + createWithDirectoryFooterLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.directoryFooterLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private String bandwidthWeightsLine = "bandwidth-weights Wbd=285 " + + "Wbe=0 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=1021 Wee=10000 " + + "Weg=1021 Wem=10000 Wgb=10000 Wgd=8694 Wgg=10000 Wgm=10000 " + + "Wmb=10000 Wmd=285 Wme=0 Wmg=0 Wmm=10000"; + protected static RelayNetworkStatusConsensus + createWithBandwidthWeightsLine(String line) + throws DescriptorParseException { + ConsensusBuilder cb = new ConsensusBuilder(); + cb.bandwidthWeightsLine = line; + return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); + } + private List<String> directorySignatures = new ArrayList<String>(); + protected void addDirectorySignature(String directorySignatureString) { + this.directorySignatures.add(directorySignatureString); + } + + protected ConsensusBuilder() { + this.dirSources.add("dir-source tor26 " + + "14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 86.59.21.38 " + + "86.59.21.38 80 443\ncontact Peter Palfrader\nvote-digest " + + "0333880AA67ED7E07C11108656D0C8D6DD1C7E5D"); + this.dirSources.add("dir-source ides " + + "27B6B5996C426270A5C95488AA5BCEB6BCC86956 216.224.124.114 " + + "216.224.124.114 9030 9090\ncontact Mike Perry " + + "<mikeperryTAfsckedTODorg>\nvote-digest " + + "1A8827ECD53184F7A771EFA9B3D30DC473FE8670"); + this.statusEntries.add("r ANONIONROUTER " + + "AHhuQ8zFQJdT8l42Axxc6m6kNwI yEMZ5B/JQixNZgC1+2rLe0pR9rU " + + "2011-11-30 02:52:58 93.128.66.111 24051 24052\ns Exit Fast " + + "Named Running V2Dir Valid\nv Tor 0.2.2.34\nw " + + "Bandwidth=1100\np reject 25,119,135-139,6881-6999"); + this.statusEntries.add("r Magellan AHlabo2RwnD8I7MPOIpJVVPgGJQ " + + "rB/7uzI4mU38bZ9cSXEy+Z/4Cuk 2011-11-30 05:37:35 " + + "188.177.149.216 9001 9030\ns Fast Named Running V2Dir " + + "Valid\nv Tor 0.2.2.34\nw Bandwidth=367\np reject 1-65535"); + this.directorySignatures.add("directory-signature " + + "14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 " + + "3509BA5A624403A905C74DA5C8A0CEC9E0D3AF86\n" + + "-----BEGIN SIGNATURE-----\n" + + "NYRcTWAMRiYYiGW0hIbzeZKU6sefg98AwwXrQUCudO8wfA1cfgttTDoscB9I" + + "TbOY\nr+c30jV/qQCMamTAEDGgJTw8KghI32vytupKallI1EjCOF8UvL1UnA" + + "LgpaR7sZ3W\n7WQZVVrWDtnYaULOEKfwnGnRC7WwE+YRSysbzwwCVs0=\n" + + "-----END SIGNATURE-----"); + this.directorySignatures.add("directory-signature " + + "27B6B5996C426270A5C95488AA5BCEB6BCC86956 " + + "D5C30C15BB3F1DA27669C2D88439939E8F418FCF\n" + + "-----BEGIN SIGNATURE-----\n" + + "DzFPj3vyYrCv0W3r8qDPJPlmeLnadY+drjWkdOqO66Ih/hAWBb9KcBJAX1sX" + + "aDA7\n/iSaDhduBXuJdcu8lbmMP8d6uYBdRjHXqWDXySUZAkSfPB4JJPNGvf" + + "oQA/qeby7E\n5374pPPL6WwCLJHkKtk21S9oHDmFBdlZq7JWQelWlVM=\n" + + "-----END SIGNATURE-----"); + } + protected byte[] buildConsensus() { + StringBuilder sb = new StringBuilder(); + this.appendHeader(sb); + this.appendBody(sb); + this.appendFooter(sb); + return sb.toString().getBytes(); + } + private void appendHeader(StringBuilder sb) { + if (this.networkStatusVersionLine != null) { + sb.append(this.networkStatusVersionLine + "\n"); + } + if (this.voteStatusLine != null) { + sb.append(this.voteStatusLine + "\n"); + } + if (this.consensusMethodLine != null) { + sb.append(this.consensusMethodLine + "\n"); + } + if (this.validAfterLine != null) { + sb.append(this.validAfterLine + "\n"); + } + if (this.freshUntilLine != null) { + sb.append(this.freshUntilLine + "\n"); + } + if (this.validUntilLine != null) { + sb.append(this.validUntilLine + "\n"); + } + if (this.votingDelayLine != null) { + sb.append(this.votingDelayLine + "\n"); + } + if (this.clientVersionsLine != null) { + sb.append(this.clientVersionsLine + "\n"); + } + if (this.serverVersionsLine != null) { + sb.append(this.serverVersionsLine + "\n"); + } + if (this.knownFlagsLine != null) { + sb.append(this.knownFlagsLine + "\n"); + } + if (this.paramsLine != null) { + sb.append(this.paramsLine + "\n"); + } + for (String dirSource : this.dirSources) { + sb.append(dirSource + "\n"); + } + } + private void appendBody(StringBuilder sb) { + for (String statusEntry : this.statusEntries) { + sb.append(statusEntry + "\n"); + } + } + private void appendFooter(StringBuilder sb) { + if (this.directoryFooterLine != null) { + sb.append(this.directoryFooterLine + "\n"); + } + if (this.bandwidthWeightsLine != null) { + sb.append(this.bandwidthWeightsLine + "\n"); + } + for (String directorySignature : this.directorySignatures) { + sb.append(directorySignature + "\n"); + } + } +} + diff --git a/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java b/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java index 5c55285..954a3d3 100644 --- a/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java +++ b/test/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java @@ -8,8 +8,6 @@ import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; -import java.util.ArrayList; -import java.util.List; import org.junit.Test; import org.torproject.descriptor.RelayNetworkStatusConsensus; @@ -22,225 +20,6 @@ import org.torproject.descriptor.RelayNetworkStatusConsensus; * a parsed consensus is fully compatible to dir-spec.txt. */ public class RelayNetworkStatusConsensusImplTest { - /* Helper class to build a consensus based on default data and - * modifications requested by test methods. */ - private static class ConsensusBuilder { - private String networkStatusVersionLine = "network-status-version 3"; - private static RelayNetworkStatusConsensus - createWithNetworkStatusVersionLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.networkStatusVersionLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String voteStatusLine = "vote-status consensus"; - private static RelayNetworkStatusConsensus - createWithVoteStatusLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.voteStatusLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String consensusMethodLine = "consensus-method 11"; - private static RelayNetworkStatusConsensus - createWithConsensusMethodLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.consensusMethodLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String validAfterLine = "valid-after 2011-11-30 09:00:00"; - private static RelayNetworkStatusConsensus - createWithValidAfterLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.validAfterLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String freshUntilLine = "fresh-until 2011-11-30 10:00:00"; - private static RelayNetworkStatusConsensus - createWithFreshUntilLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.freshUntilLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String validUntilLine = "valid-until 2011-11-30 12:00:00"; - private static RelayNetworkStatusConsensus - createWithValidUntilLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.validUntilLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String votingDelayLine = "voting-delay 300 300"; - private static RelayNetworkStatusConsensus - createWithVotingDelayLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.votingDelayLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String clientVersionsLine = "client-versions 0.2.1.31," - + "0.2.2.34,0.2.3.6-alpha,0.2.3.7-alpha,0.2.3.8-alpha"; - private static RelayNetworkStatusConsensus - createWithClientVersionsLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.clientVersionsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String serverVersionsLine = "server-versions 0.2.1.31," - + "0.2.2.34,0.2.3.6-alpha,0.2.3.7-alpha,0.2.3.8-alpha"; - private static RelayNetworkStatusConsensus - createWithServerVersionsLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.serverVersionsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String knownFlagsLine = "known-flags Authority BadExit Exit " - + "Fast Guard HSDir Named Running Stable Unnamed V2Dir Valid"; - private static RelayNetworkStatusConsensus - createWithKnownFlagsLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.knownFlagsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String paramsLine = "params " - + "CircuitPriorityHalflifeMsec=30000 bwauthbestratio=1 " - + "bwauthcircs=1 bwauthdescbw=0 bwauthkp=10000 bwauthpid=1 " - + "bwauthtd=5000 bwauthti=50000 bwauthtidecay=5000 cbtnummodes=3 " - + "cbtquantile=80 circwindow=1000 refuseunknownexits=1"; - private static RelayNetworkStatusConsensus - createWithParamsLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.paramsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private List<String> dirSources = new ArrayList<String>(); - private List<String> statusEntries = new ArrayList<String>(); - private String directoryFooterLine = "directory-footer"; - private static RelayNetworkStatusConsensus - createWithDirectoryFooterLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.directoryFooterLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private String bandwidthWeightsLine = "bandwidth-weights Wbd=285 " - + "Wbe=0 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=1021 Wee=10000 " - + "Weg=1021 Wem=10000 Wgb=10000 Wgd=8694 Wgg=10000 Wgm=10000 " - + "Wmb=10000 Wmd=285 Wme=0 Wmg=0 Wmm=10000"; - private static RelayNetworkStatusConsensus - createWithBandwidthWeightsLine(String line) - throws DescriptorParseException { - ConsensusBuilder cb = new ConsensusBuilder(); - cb.bandwidthWeightsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); - } - private List<String> directorySignatures = new ArrayList<String>(); - private ConsensusBuilder() { - this.dirSources.add("dir-source tor26 " - + "14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 86.59.21.38 " - + "86.59.21.38 80 443\ncontact Peter Palfrader\nvote-digest " - + "0333880AA67ED7E07C11108656D0C8D6DD1C7E5D"); - this.dirSources.add("dir-source ides " - + "27B6B5996C426270A5C95488AA5BCEB6BCC86956 216.224.124.114 " - + "216.224.124.114 9030 9090\ncontact Mike Perry " - + "<mikeperryTAfsckedTODorg>\nvote-digest " - + "1A8827ECD53184F7A771EFA9B3D30DC473FE8670"); - this.statusEntries.add("r ANONIONROUTER " - + "AHhuQ8zFQJdT8l42Axxc6m6kNwI yEMZ5B/JQixNZgC1+2rLe0pR9rU " - + "2011-11-30 02:52:58 93.128.66.111 24051 24052\ns Exit Fast " - + "Named Running V2Dir Valid\nv Tor 0.2.2.34\nw " - + "Bandwidth=1100\np reject 25,119,135-139,6881-6999"); - this.statusEntries.add("r Magellan AHlabo2RwnD8I7MPOIpJVVPgGJQ " - + "rB/7uzI4mU38bZ9cSXEy+Z/4Cuk 2011-11-30 05:37:35 " - + "188.177.149.216 9001 9030\ns Fast Named Running V2Dir " - + "Valid\nv Tor 0.2.2.34\nw Bandwidth=367\np reject 1-65535"); - this.directorySignatures.add("directory-signature " - + "14C131DFC5C6F93646BE72FA1401C02A8DF2E8B4 " - + "3509BA5A624403A905C74DA5C8A0CEC9E0D3AF86\n" - + "-----BEGIN SIGNATURE-----\n" - + "NYRcTWAMRiYYiGW0hIbzeZKU6sefg98AwwXrQUCudO8wfA1cfgttTDoscB9I" - + "TbOY\nr+c30jV/qQCMamTAEDGgJTw8KghI32vytupKallI1EjCOF8UvL1UnA" - + "LgpaR7sZ3W\n7WQZVVrWDtnYaULOEKfwnGnRC7WwE+YRSysbzwwCVs0=\n" - + "-----END SIGNATURE-----"); - this.directorySignatures.add("directory-signature " - + "27B6B5996C426270A5C95488AA5BCEB6BCC86956 " - + "D5C30C15BB3F1DA27669C2D88439939E8F418FCF\n" - + "-----BEGIN SIGNATURE-----\n" - + "DzFPj3vyYrCv0W3r8qDPJPlmeLnadY+drjWkdOqO66Ih/hAWBb9KcBJAX1sX" - + "aDA7\n/iSaDhduBXuJdcu8lbmMP8d6uYBdRjHXqWDXySUZAkSfPB4JJPNGvf" - + "oQA/qeby7E\n5374pPPL6WwCLJHkKtk21S9oHDmFBdlZq7JWQelWlVM=\n" - + "-----END SIGNATURE-----"); - } - private byte[] buildConsensus() { - StringBuilder sb = new StringBuilder(); - this.appendHeader(sb); - this.appendBody(sb); - this.appendFooter(sb); - return sb.toString().getBytes(); - } - private void appendHeader(StringBuilder sb) { - if (this.networkStatusVersionLine != null) { - sb.append(this.networkStatusVersionLine + "\n"); - } - if (this.voteStatusLine != null) { - sb.append(this.voteStatusLine + "\n"); - } - if (this.consensusMethodLine != null) { - sb.append(this.consensusMethodLine + "\n"); - } - if (this.validAfterLine != null) { - sb.append(this.validAfterLine + "\n"); - } - if (this.freshUntilLine != null) { - sb.append(this.freshUntilLine + "\n"); - } - if (this.validUntilLine != null) { - sb.append(this.validUntilLine + "\n"); - } - if (this.votingDelayLine != null) { - sb.append(this.votingDelayLine + "\n"); - } - if (this.clientVersionsLine != null) { - sb.append(this.clientVersionsLine + "\n"); - } - if (this.serverVersionsLine != null) { - sb.append(this.serverVersionsLine + "\n"); - } - if (this.knownFlagsLine != null) { - sb.append(this.knownFlagsLine + "\n"); - } - if (this.paramsLine != null) { - sb.append(this.paramsLine + "\n"); - } - for (String dirSource : this.dirSources) { - sb.append(dirSource + "\n"); - } - } - private void appendBody(StringBuilder sb) { - for (String statusEntry : this.statusEntries) { - sb.append(statusEntry + "\n"); - } - } - private void appendFooter(StringBuilder sb) { - if (this.directoryFooterLine != null) { - sb.append(this.directoryFooterLine + "\n"); - } - if (this.bandwidthWeightsLine != null) { - sb.append(this.bandwidthWeightsLine + "\n"); - } - for (String directorySignature : this.directorySignatures) { - sb.append(directorySignature + "\n"); - } - } - } - /* Helper class to build a directory source based on default data and * modifications requested by test methods. */ private static class DirSourceBuilder { @@ -459,7 +238,7 @@ public class RelayNetworkStatusConsensusImplTest { createWithDirectorySignature(String directorySignatureString) throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); - cb.directorySignatures.add(directorySignatureString); + cb.addDirectorySignature(directorySignatureString); return new RelayNetworkStatusConsensusImpl(cb.buildConsensus()); } private String identity = "ED03BB616EB2F60BEC80151114BB25CEF515B226";

1 0

[metrics-lib/master] Improve downloader a bit.
by karsten＠torproject.org 19 Jan '12

19 Jan '12

commit 1bf8a7f914458f5b3968c453abc91dd24808aef6 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 19 17:36:13 2012 +0100 Improve downloader a bit. --- .../torproject/descriptor/DescriptorRequest.java | 4 +++ .../torproject/descriptor/impl/DescriptorImpl.java | 3 ++ .../descriptor/impl/DescriptorRequestImpl.java | 8 ++++++ .../descriptor/impl/DirectoryDownloader.java | 8 +++++- .../descriptor/impl/DownloadCoordinatorImpl.java | 24 ++++++------------- 5 files changed, 30 insertions(+), 17 deletions(-) diff --git a/src/org/torproject/descriptor/DescriptorRequest.java b/src/org/torproject/descriptor/DescriptorRequest.java index b0c2c02..f68332f 100644 --- a/src/org/torproject/descriptor/DescriptorRequest.java +++ b/src/org/torproject/descriptor/DescriptorRequest.java @@ -15,6 +15,10 @@ public interface DescriptorRequest { * previously configured. */ public String getDirectoryNickname(); + /* Return the first exception that was thrown when making this request + * or parsing the response, or null if no exception was thrown. */ + public Exception getException(); + /* Return the response code that the directory mirror or authority * returned. */ public int getResponseCode(); diff --git a/src/org/torproject/descriptor/impl/DescriptorImpl.java b/src/org/torproject/descriptor/impl/DescriptorImpl.java index 7e7abee..2d20fa7 100644 --- a/src/org/torproject/descriptor/impl/DescriptorImpl.java +++ b/src/org/torproject/descriptor/impl/DescriptorImpl.java @@ -19,6 +19,9 @@ public abstract class DescriptorImpl implements Descriptor { byte[] rawDescriptorBytes, String fileName) throws DescriptorParseException { List<Descriptor> parsedDescriptors = new ArrayList<Descriptor>(); + if (rawDescriptorBytes == null) { + return parsedDescriptors; + } byte[] first100Chars = new byte[Math.min(100, rawDescriptorBytes.length)]; System.arraycopy(rawDescriptorBytes, 0, first100Chars, 0, diff --git a/src/org/torproject/descriptor/impl/DescriptorRequestImpl.java b/src/org/torproject/descriptor/impl/DescriptorRequestImpl.java index 6926e59..6c44a79 100644 --- a/src/org/torproject/descriptor/impl/DescriptorRequestImpl.java +++ b/src/org/torproject/descriptor/impl/DescriptorRequestImpl.java @@ -87,5 +87,13 @@ public class DescriptorRequestImpl implements DescriptorRequest { public List<Descriptor> getDescriptors() { return this.descriptors; } + + private Exception exception; + protected void setException(Exception exception) { + this.exception = exception; + } + public Exception getException() { + return this.exception; + } } diff --git a/src/org/torproject/descriptor/impl/DirectoryDownloader.java b/src/org/torproject/descriptor/impl/DirectoryDownloader.java index d07c5c9..b6d9b05 100644 --- a/src/org/torproject/descriptor/impl/DirectoryDownloader.java +++ b/src/org/torproject/descriptor/impl/DirectoryDownloader.java @@ -66,8 +66,11 @@ public class DirectoryDownloader implements Runnable { byte[] responseBytes = baos.toByteArray(); request.setResponseBytes(responseBytes); request.setRequestEnd(System.currentTimeMillis()); + request.setDescriptors(DescriptorImpl. + parseRelayOrBridgeDescriptors(responseBytes, null)); } - } catch (IOException e) { + } catch (Exception e) { + request.setException(e); /* Stop downloading from this directory if there are any * problems, e.g., refused connections. */ keepRunning = false; @@ -83,6 +86,9 @@ public class DirectoryDownloader implements Runnable { } /* Interrupt a download request if it takes longer than a given time. */ + /* TODO Also look at URLConnection.setConnectTimeout() and + * URLConnection.setReadTimeout() instead of implementing this + * ourselves. */ private static class RequestTimeout implements Runnable { private long timeoutMillis; private Thread downloaderThread; diff --git a/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java b/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java index 5d4ca21..0c0ebc3 100644 --- a/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java +++ b/src/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java @@ -190,17 +190,16 @@ public class DownloadCoordinatorImpl implements DownloadCoordinator { if (response.getDescriptorType().equals("consensus")) { this.requestingConsensuses.remove(nickname); if (response.getResponseCode() == 200) { - List<RelayNetworkStatusConsensus> parsedConsensuses = - RelayNetworkStatusConsensusImpl.parseConsensuses( - response.getResponseBytes()); - List<Descriptor> parsedDescriptors = - new ArrayList<Descriptor>(parsedConsensuses); - response.setDescriptors(parsedDescriptors); if (this.includeCurrentReferencedVotes) { /* TODO Only add votes if the consensus is not older than one * hour. Or does that make no sense? */ - for (RelayNetworkStatusConsensus parsedConsensus : - parsedConsensuses) { + for (Descriptor parsedDescriptor : response.getDescriptors()) { + if (!(parsedDescriptor instanceof + RelayNetworkStatusConsensus)) { + continue; + } + RelayNetworkStatusConsensus parsedConsensus = + (RelayNetworkStatusConsensus) parsedDescriptor; for (DirSourceEntry dirSource : parsedConsensus.getDirSourceEntries().values()) { String identity = dirSource.getIdentity(); @@ -226,14 +225,7 @@ public class DownloadCoordinatorImpl implements DownloadCoordinator { } } else if (response.getDescriptorType().equals("vote")) { String requestedVote = requestingVotes.remove(nickname); - if (response.getResponseCode() == 200) { - List<RelayNetworkStatusVote> parsedVotes = - RelayNetworkStatusVoteImpl.parseVotes( - response.getResponseBytes()); - List<Descriptor> parsedDescriptors = - new ArrayList<Descriptor>(parsedVotes); - response.setDescriptors(parsedDescriptors); - } else { + if (response.getResponseCode() != 200) { this.missingVotes.add(requestedVote); } }

1 0

[webstats/master] Add logimport.sh and logarchive.sh to git
by runa＠torproject.org 19 Jan '12

19 Jan '12

commit acfa536cf24f253dfdfc1321e0033c878553110a Author: Runa A. Sandvik <runa.sandvik(a)gmail.com> Date: Thu Jan 19 08:49:04 2012 +0000 Add logimport.sh and logarchive.sh to git --- logarchive.sh | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ logimport.sh | 47 ++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+), 0 deletions(-) diff --git a/logarchive.sh b/logarchive.sh new file mode 100644 index 0000000..bab17b5 --- /dev/null +++ b/logarchive.sh @@ -0,0 +1,74 @@ +#!/bin/bash +# +# Define a few variables +logs="/srv/webstats.torproject.org/home/webstats/out" +archive="/srv/webstats.torproject.org/archive" +log_archive="/srv/webstats.torproject.org/archive/weblogs" +data_archive="/srv/webstats.torproject.org/archive/data" +data_publish="/srv/webstats.torproject.org/htdocs/data" +record="/srv/webstats.torproject.org/logarchive.log" + +# Define the hosts we have logs for +hosts=( + metrics.torproject.org-access.log +) + +# See if we actually have logs to process +if [ ! "$(ls -A $logs)" ]; +then + echo "`date` - No logs to process" >> $record + exit 1 +fi + +# rsync ALL the logs so that we can delete them from the out/ directory +# later +rsync -ar "$logs/" "$log_archive/" + +for host in "${hosts[@]}" +do + year=`find "$log_archive/" -name $host | cut -d / -f 6 | sort | uniq` + + for y in $year; do + month=`find "$log_archive/$y/" -name $host | cut -d / -f 7 | sort | uniq` + cd "$archive/" + + for m in $month; do + all_the_logs=`find "weblogs/$y/$m" -name $host` + for i in $all_the_logs; do + # If we already have a tarball, check to see if the file is already a + # part of it + if [ -e "$data_archive/$host-$y-$m.tar" ] + then + # See if the file is already a part of the tarball + tar --list --file="$data_archive/$host-$y-$m.tar" $i + check_exists=`echo $?` + if [[ $check_exists -eq "1" ]] + then + tar --append --file="$host-$y-$m.tar" $i + fi + else + tar --append --file="$host-$y-$m.tar" $i + fi + done + + # If we did create a new tarball, move it + if [ -e "$host-$y-$m.tar" ] + then + mv "$host-$y-$m.tar" "$data_archive" + fi + + # Pack up the tarball and make it available online + cd "$data_archive" + bzip2 -kf1 "$host-$y-$m.tar" + mv "$host-$y-$m.tar.bz2" "$data_publish" + + done + + # And remove logs from the out/ directory + rm -rf "$logs/$y/" + + done + + # And report back + echo "`date` - Logs processed for $host" >> $record +done diff --git a/logimport.sh b/logimport.sh new file mode 100644 index 0000000..438e3dd --- /dev/null +++ b/logimport.sh @@ -0,0 +1,47 @@ +#!/bin/bash +# +# The sanitized logs are in /srv/webstats.torproject.org/home/webstats/out/, +# with the following format: year/month/day/$virtualhost-access.log +# +# Need to concat the logs in chronological order and import into +# awstats and webalizer. + +# Define a few variables +logtmp="/srv/webstats.torproject.org/logtmp" +logs="/srv/webstats.torproject.org/home/webstats/out" +record="/srv/webstats.torproject.org/logimport.log" + +# Define the hosts we have logs for +hosts=( + metrics +) + +# See if we actually have logs to process +if [ ! "$(ls -A $logs)" ]; +then + echo "`date` - No logs to process" >> $record + exit 1 +fi + +for host in "${hosts[@]}" +do + # Concat the logs in chronological order + mkdir "$logtmp" + cd "$logtmp" + find "$logs/" -name "$host.torproject.org-access.log" | sort | xargs -I {} cat {} > "$host.torproject.org-access.log" + + # Time to run the web log analysis tools + # + # Running awstats first + /srv/webstats.torproject.org/awstats/awstats.pl -config=$host.torproject.org -update + + # And then webalizer + webalizer -c "/srv/webstats.torproject.org/configs/webalizer.$host.torproject.org.conf" + + # Cleanup + cd "/srv/webstats.torproject.org" + rm -rf "$logtmp" + + # And report back + echo "`date` - Logs processed for $host" >> $record +done

1 0

[webstats/master] Move files to out/ when they're at least 4 days old.
by runa＠torproject.org 19 Jan '12

19 Jan '12

commit f2789a8671cbbb97c85be0ecfec4108ae66e93b8 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jan 18 15:22:25 2012 +0100 Move files to out/ when they're at least 4 days old. We're not moving files to the out/ directory for further processing if they're less than 4 days old. The idea is that these files can still change when new logs arrive, and once they're in out/, they shouldn't change anymore. Instead, we're moving them to state/full/ until they're at least 4 days old. However, we forgot that last step. Implement it now. --- src/org/torproject/webstats/Main.java | 50 ++++++++++++++++++++++++++++++-- 1 files changed, 46 insertions(+), 4 deletions(-) diff --git a/src/org/torproject/webstats/Main.java b/src/org/torproject/webstats/Main.java index 1812567..7f1afe1 100644 --- a/src/org/torproject/webstats/Main.java +++ b/src/org/torproject/webstats/Main.java @@ -73,10 +73,13 @@ import org.apache.commons.compress.compressors.gzip.*; * days old, move the file to state/full/. * c. If b. does not apply, append a line to out-history.new and move * the file to out/. - * 7. Rename state/out-history to state/out-history.old and rename + * 7. For each file in state/full/, check whether the sanitized log is at + * least four (4) days old and not contained in state/out-history. If + * so, append a line to out-history.new and move the file to out/. + * 8. Rename state/out-history to state/out-history.old and rename * state/out-history.new to state/out-history. Delete * state/out-history.old. - * 8. Delete state/lock and exit. + * 9. Delete state/lock and exit. * * If the program is interrupted and leaves a lock file in state/lock, it * requires an operator to fix the state/ directory and make it work @@ -130,8 +133,9 @@ public class Main { for (String outputFileName : updatedOutputFiles) { /* Step 6 */ moveOutputFile(outputFileName); } - overwriteOutHistoryFile(); /* Step 7 */ - deleteLockFile(); /* Step 8 */ + moveFullFilesToOut(); /* Step 7 */ + overwriteOutHistoryFile(); /* Step 8 */ + deleteLockFile(); /* Step 9 */ } /* Define file and directory names. */ @@ -150,6 +154,7 @@ public class Main { private static File stateOutHistoryOldFile = new File("state/out-history.old"); private static File stateDiffDirectory = new File("state/diff"); + private static String stateFullDirectoryString = "state/full"; private static File stateFullDirectory = new File("state/full"); private static File stateTempDirectory = new File("state/temp"); @@ -489,6 +494,43 @@ public class Main { stateTempFile.renameTo(outFile); } } + private static void moveFullFilesToOut() { + Stack<String> fileNames = new Stack<String>(); + fileNames.add(stateFullDirectoryString); + while (!fileNames.isEmpty()) { + String fileName = fileNames.pop(); + File fileOrDirectory = new File(fileName); + if (fileOrDirectory.isDirectory()) { + for (File file : fileOrDirectory.listFiles()) { + fileNames.add(fileName + "/" + file.getName()); + } + } else { + String outputFileName = fileName.substring( + (stateFullDirectoryString + "/").length()); + File outFile = new File(outDirectory, outputFileName); + File stateFullFile = new File(stateFullDirectory, outputFileName); + long ageInDays = -1L; + try { + ageInDays = (now + - outputFileFormat.parse(outputFileName.substring(0, + outputFileName.lastIndexOf("/") + 1)).getTime()) + / (24L * 60L * 60L * 1000L); + } catch (ParseException e) { + e.printStackTrace(); + System.err.println("Could not parse timestamp from '" + + outputFileName + "'. Exiting."); + System.exit(1); + } + if (!outHistoryFiles.contains(outFile.getAbsolutePath()) && + ageInDays >= 4L) { + outFile.getParentFile().mkdirs(); + String line = outFile.getAbsolutePath(); + appendToHistoryFile(stateOutHistoryNewFile, line); + stateFullFile.renameTo(outFile); + } + } + } + } private static void overwriteOutHistoryFile() { stateOutHistoryFile.renameTo(stateOutHistoryOldFile); stateOutHistoryNewFile.renameTo(stateOutHistoryFile);

1 0

[webstats/master] Merge branch 'notmaster' of git://git.torproject.org/user/karsten/webstats
by runa＠torproject.org 19 Jan '12

19 Jan '12

commit b4c16a00f4daf62ba85ee9fa4fc7648f52f6a821 Merge: 82d9510 f2789a8 Author: Runa A. Sandvik <runa.sandvik(a)gmail.com> Date: Thu Jan 19 08:42:05 2012 +0000 Merge branch 'notmaster' of git://git.torproject.org/user/karsten/webstats src/org/torproject/webstats/Main.java | 50 ++++++++++++++++++++++++++++++-- 1 files changed, 46 insertions(+), 4 deletions(-)

1 0