October 2011 - tor-commits - lists.torproject.org

[stem/master] Accidently suppressing integ asserts
by atagar＠torproject.org 20 Oct '11

20 Oct '11

commit 0a083c8b73a663d38f473e0180b0009d28e1b5d9 Author: Damian Johnson <atagar(a)torproject.org> Date: Thu Oct 20 10:09:36 2011 -0700 Accidently suppressing integ asserts A previous commit to suppress errors from the close() method in python 2.7 also suppressed the following asserts. These asserts still work with 2.7 so removing them from the try block. --- test/integ/message.py | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/… [View More]

1 0

[stem/master] Message integ tests compatability issues
by atagar＠torproject.org 20 Oct '11

20 Oct '11

commit 26374bafd5957d3039cf10596de0f7a021f1afb9 Author: Damian Johnson <atagar(a)torproject.org> Date: Thu Oct 20 09:49:45 2011 -0700 Message integ tests compatability issues The message integration tests had compatability issues with... - Newer python versions (2.6 -> 2.7) due to an unexpected failure when calling close() on the socket's file object. We're operating with a closed socket at that time so all bets are off about if this should/shouldn't … [View More]raise so the difference in behavior just warranted a testing workaround. - Older tor versions don't support 'GETINFO config-text'. I've left this as a TODO note for now and will add a version check for that test later. --- test/integ/message.py | 24 ++++++++++++++++-------- 1 files changed, 16 insertions(+), 8 deletions(-) diff --git a/test/integ/message.py b/test/integ/message.py index 46f20d2..053e9df 100644 --- a/test/integ/message.py +++ b/test/integ/message.py @@ -60,15 +60,20 @@ class TestMessageFunctions(unittest.TestCase): self.assertRaises(stem.types.ControlSocketClosed, stem.types.read_message, control_socket_file) # Closing the file handler, however, will cause a different type of error. + # This seems to depend on the python version, in 2.6 we get an + # AttributeError and in 2.7 the close() call raises... + # error: [Errno 32] Broken pipe - control_socket_file.close() - control_socket_file.write("GETINFO version\r\n") - - # receives: AttributeError: 'NoneType' object has no attribute 'sendall' - self.assertRaises(AttributeError, control_socket_file.flush) - - # receives: stem.types.ControlSocketClosed: socket file has been closed - self.assertRaises(stem.types.ControlSocketClosed, stem.types.read_message, control_socket_file) + try: + control_socket_file.close() + control_socket_file.write("GETINFO version\r\n") + + # receives: AttributeError: 'NoneType' object has no attribute 'sendall' + self.assertRaises(AttributeError, control_socket_file.flush) + + # receives: stem.types.ControlSocketClosed: socket file has been closed + self.assertRaises(stem.types.ControlSocketClosed, stem.types.read_message, control_socket_file) + except: pass def test_invalid_command(self): """ @@ -135,6 +140,9 @@ class TestMessageFunctions(unittest.TestCase): Parses the 'GETINFO config-text' response. """ + # TODO: add a 0.2.2.7-alpha version check for this test (that was when + # 'GETINFO config-text' was added) + # We can't be certain of the order, and there may be extra config-text # entries as per... # https://trac.torproject.org/projects/tor/ticket/2362 [View Less]

1 0

[metrics-web/master] Stop archiving consensuses and votes after checking them.
by karsten＠torproject.org 20 Oct '11

20 Oct '11

commit 29dbe7fbb197c7c22f85d3fcd610daa8613a2de2 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Oct 20 14:30:47 2011 +0200 Stop archiving consensuses and votes after checking them. The idea of archiving consensus and votes was that we could compare them over time to see if, e.g., a consensus parameter gets added or removed. This works in theory, but in practice, this approach makes the code more complex than it needs to be. We can as well change … [View More]valid consensus parameters in the code. --- src/org/torproject/chc/Archiver.java | 120 ---------------------- src/org/torproject/chc/Downloader.java | 15 +--- src/org/torproject/chc/Main.java | 34 +------ src/org/torproject/chc/MetricsWebsiteReport.java | 5 - src/org/torproject/chc/NagiosReport.java | 19 ---- src/org/torproject/chc/Parser.java | 11 -- src/org/torproject/chc/Report.java | 4 - src/org/torproject/chc/Status.java | 9 -- src/org/torproject/chc/StdOutReport.java | 19 ---- 9 files changed, 5 insertions(+), 231 deletions(-) diff --git a/src/org/torproject/chc/Archiver.java b/src/org/torproject/chc/Archiver.java deleted file mode 100644 index 6271348..0000000 --- a/src/org/torproject/chc/Archiver.java +++ /dev/null @@ -1,120 +0,0 @@ -/* Copyright 2011 The Tor Project - * See LICENSE for licensing information */ -package org.torproject.chc; - -import java.io.*; -import java.util.*; - -/* Load the last network status consensus and corresponding votes from - * disk and save newly downloaded ones to disk. */ -public class Archiver { - - /* Local directory containing cached consensuses and votes. */ - private File cachedFilesDirectory = new File("chc-cache"); - - /* Copy of the most recent cached consensus and votes. */ - private String loadedConsensus; - private List<String> loadedVotes = new ArrayList<String>(); - - /* Load the most recent cached consensus and votes to memory. */ - public void loadLastFromDisk() { - if (!this.cachedFilesDirectory.exists()) { - return; - } - String lastValidAfter = this.findLastConsensusPrefix(); - if (lastValidAfter != null) { - for (File file : this.cachedFilesDirectory.listFiles()) { - if (file.isDirectory() || - !file.getName().startsWith(lastValidAfter)) { - continue; - } - String content = null; - try { - FileInputStream fis = new FileInputStream(file); - BufferedInputStream bis = new BufferedInputStream(fis); - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - int len; - byte[] data = new byte[1024]; - while ((len = bis.read(data, 0, 1024)) >= 0) { - baos.write(data, 0, len); - } - bis.close(); - byte[] allData = baos.toByteArray(); - content = new String(allData); - } catch (IOException e) { - System.err.println("Could not read cached status from file '" - + file.getAbsolutePath() + "'. Skipping."); - continue; - } - if (file.getName().contains("-consensus")) { - this.loadedConsensus = content; - } else if (file.getName().contains("-vote-")) { - this.loadedVotes.add(content); - } - } - } - } - - /* Delete all cached consensuses and votes but the last consensus and - * corresponding votes. */ - public void deleteAllButLast() { - if (!this.cachedFilesDirectory.exists()) { - return; - } - String lastValidAfter = this.findLastConsensusPrefix(); - if (lastValidAfter != null) { - for (File file : this.cachedFilesDirectory.listFiles()) { - if (!file.getName().startsWith(lastValidAfter)) { - file.delete(); - } - } - } - } - - /* Save a status to disk under the given file name. */ - public void saveStatusStringToDisk(String content, String fileName) { - try { - this.cachedFilesDirectory.mkdirs(); - BufferedWriter bw = new BufferedWriter(new FileWriter(new File( - this.cachedFilesDirectory, fileName))); - bw.write(content); - bw.close(); - } catch (IOException e) { - System.err.println("Could not save status to file '" - + this.cachedFilesDirectory.getAbsolutePath() + "/" + fileName - + "'. Ignoring."); - } - } - - /* Find and return the timestamp prefix of the last published - * consensus. */ - private String findLastConsensusPrefix() { - String lastValidAfter = null; - for (File file : this.cachedFilesDirectory.listFiles()) { - if (file.isDirectory() || - file.getName().length() != - "yyyy-MM-dd-HH-mm-ss-consensus".length() || - !file.getName().endsWith("-consensus")) { - continue; - } - String prefix = file.getName().substring(0, - "yyyy-MM-dd-HH-mm-ss".length()); - if (lastValidAfter == null || - prefix.compareTo(lastValidAfter) > 0) { - lastValidAfter = prefix; - } - } - return lastValidAfter; - } - - /* Return the previously loaded (unparsed) consensus string. */ - public String getConsensusString() { - return this.loadedConsensus; - } - - /* Return the previously loaded (unparsed) vote strings. */ - public List<String> getVoteStrings() { - return this.loadedVotes; - } -} - diff --git a/src/org/torproject/chc/Downloader.java b/src/org/torproject/chc/Downloader.java index bdd23fe..cf451ee 100644 --- a/src/org/torproject/chc/Downloader.java +++ b/src/org/torproject/chc/Downloader.java @@ -20,21 +20,8 @@ public class Downloader { + "208.83.223.34:443,128.31.0.34:9131,194.109.206.212"). split(","))); - /* Set the last known valid-after time to avoid downloading a new - * consensus if there cannot be a new one yet. */ - private long lastKnownValidAfterMillis; - public void setLastKnownValidAfterMillis( - long lastKnownValidAfterMillis) { - this.lastKnownValidAfterMillis = lastKnownValidAfterMillis; - } - - /* Download a new consensus and corresponding votes if we expect them to - * be newer than the ones we have. */ + /* Download a new consensus and corresponding votes. */ public void downloadFromAuthorities() { - if (System.currentTimeMillis() - lastKnownValidAfterMillis < - 60L * 60L * 1000L) { - return; - } this.downloadConsensus(); if (this.downloadedConsensus != null) { this.parseConsensusToFindReferencedVotes(); diff --git a/src/org/torproject/chc/Main.java b/src/org/torproject/chc/Main.java index 5aadc6e..d42ee44 100644 --- a/src/org/torproject/chc/Main.java +++ b/src/org/torproject/chc/Main.java @@ -16,26 +16,13 @@ public class Main { reports.add(new NagiosReport("stats/consensus-health")); reports.add(new StdOutReport()); - /* Load last-known consensus and votes from disk and parse them. */ - Archiver archiver = new Archiver(); - archiver.loadLastFromDisk(); - Parser parser = new Parser(); - Status parsedCachedConsensus = parser.parse( - archiver.getConsensusString(), archiver.getVoteStrings()); - if (parsedCachedConsensus != null) { - for (Report report : reports) { - report.processCachedConsensus(parsedCachedConsensus); - } - } - /* Download consensus and corresponding votes from the directory - * authorities and parse them, too. */ + * authorities. */ Downloader downloader = new Downloader(); - if (parsedCachedConsensus != null) { - downloader.setLastKnownValidAfterMillis( - parsedCachedConsensus.getValidAfterMillis()); - } downloader.downloadFromAuthorities(); + + /* Parse consensus and votes and pass them to the reports. */ + Parser parser = new Parser(); Status parsedDownloadedConsensus = parser.parse( downloader.getConsensusString(), downloader.getVoteStrings()); if (parsedDownloadedConsensus != null) { @@ -44,19 +31,6 @@ public class Main { } } - /* Save the new consensus and corresponding votes to disk and delete - * all previous ones. */ - if (parsedDownloadedConsensus != null) { - archiver.saveStatusStringToDisk( - parsedDownloadedConsensus.getUnparsedString(), - parsedDownloadedConsensus.getFileName()); - for (Status vote : parsedDownloadedConsensus.getVotes()) { - archiver.saveStatusStringToDisk(vote.getUnparsedString(), - vote.getFileName()); - } - archiver.deleteAllButLast(); - } - /* Finish writing reports. */ for (Report report : reports) { report.writeReport(); diff --git a/src/org/torproject/chc/MetricsWebsiteReport.java b/src/org/torproject/chc/MetricsWebsiteReport.java index 2a54927..f77b89f 100644 --- a/src/org/torproject/chc/MetricsWebsiteReport.java +++ b/src/org/torproject/chc/MetricsWebsiteReport.java @@ -25,11 +25,6 @@ public class MetricsWebsiteReport implements Report { this.htmlOutputFile = new File(htmlOutputFilename); } - /* Process the last but one consensus (ignored in this report). */ - public void processCachedConsensus(Status cachedConsensus) { - /* Do nothing. */ - } - /* Store the downloaded consensus and corresponding votes for later * processing. */ private Status downloadedConsensus; diff --git a/src/org/torproject/chc/NagiosReport.java b/src/org/torproject/chc/NagiosReport.java index 11bdaa3..2b5a6d3 100644 --- a/src/org/torproject/chc/NagiosReport.java +++ b/src/org/torproject/chc/NagiosReport.java @@ -18,12 +18,6 @@ public class NagiosReport implements Report { this.nagiosOutputFile = new File(nagiosOutputFilename); } - /* Store the cached consensus for processing. */ - private Status cachedConsensus; - public void processCachedConsensus(Status cachedConsensus) { - this.cachedConsensus = cachedConsensus; - } - /* Store the current consensus and corresponding votes for * processing. */ private Status downloadedConsensus; @@ -58,8 +52,6 @@ public class NagiosReport implements Report { this.checkMissingVotes(); this.checkBandwidthScanners(); } - } else if (this.cachedConsensus != null) { - this.checkConsensusValid(this.cachedConsensus); } else { this.nagiosUnknowns.add("No consensus known"); } @@ -204,17 +196,6 @@ public class NagiosReport implements Report { } } - /* Check that the most recent consensus is not more than 3 hours - * old. */ - public void checkConsensusValid(Status consensus) { - if (consensus.getValidAfterMillis() < - System.currentTimeMillis() - 3L * 60L * 60L * 1000L) { - this.nagiosCriticals.add("The last known consensus published at " - + dateTimeFormat.format(consensus.getValidAfterMillis()) - + " is more than 3 hours old and therefore not valid anymore"); - } - } - /* Write all output to the Nagios status file. The most severe status * goes in the first line of the output file and the same status and all * log messages in the second line. */ diff --git a/src/org/torproject/chc/Parser.java b/src/org/torproject/chc/Parser.java index f7e9b57..47609a8 100644 --- a/src/org/torproject/chc/Parser.java +++ b/src/org/torproject/chc/Parser.java @@ -28,12 +28,9 @@ public class Parser { /* Date-time formats to parse and format timestamps. */ private static SimpleDateFormat dateTimeFormat; - private static SimpleDateFormat fileNameFormat; static { dateTimeFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); dateTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); - fileNameFormat = new SimpleDateFormat("yyyy-MM-dd-HH-mm-ss"); - fileNameFormat.setTimeZone(TimeZone.getTimeZone("UTC")); } /* Parse a consensus or vote string into a Status instance. */ @@ -143,14 +140,6 @@ public class Parser { br.close(); status.setRunningRelays(runningRelays); status.setBandwidthWeights(bandwidthWeights); - if (isConsensus) { - status.setFileName(fileNameFormat.format( - status.getValidAfterMillis()) + "-consensus"); - } else { - status.setFileName(fileNameFormat.format( - status.getValidAfterMillis()) + "-vote-" - + status.getFingerprint()); - } } catch (IOException e) { System.err.println("Caught an IOException while parsing a " + (isConsensus ? "consensus" : "vote") + " string. Skipping."); diff --git a/src/org/torproject/chc/Report.java b/src/org/torproject/chc/Report.java index 28483cc..c3937e0 100644 --- a/src/org/torproject/chc/Report.java +++ b/src/org/torproject/chc/Report.java @@ -8,10 +8,6 @@ import java.util.*; * some form. */ public interface Report { - /* Process the cached consensus and corresponding votes to compare them - * to the downloaded ones. */ - public abstract void processCachedConsensus(Status cachedConsensus); - /* Process the downloaded current consensus and corresponding votes to * find irregularities between them. */ public abstract void processDownloadedConsensus( diff --git a/src/org/torproject/chc/Status.java b/src/org/torproject/chc/Status.java index 51a9052..cbf7395 100644 --- a/src/org/torproject/chc/Status.java +++ b/src/org/torproject/chc/Status.java @@ -29,15 +29,6 @@ public class Status implements Comparable<Status> { return this.unparsedString; } - /* File name when writing this status to disk. */ - private String fileName; - public void setFileName(String fileName) { - this.fileName = fileName; - } - public String getFileName() { - return this.fileName; - } - /* Votes published at the same time as this consensus; votes don't * reference any statuses. */ private SortedSet<Status> votes = new TreeSet<Status>(); diff --git a/src/org/torproject/chc/StdOutReport.java b/src/org/torproject/chc/StdOutReport.java index f99c2eb..15a9148 100644 --- a/src/org/torproject/chc/StdOutReport.java +++ b/src/org/torproject/chc/StdOutReport.java @@ -21,12 +21,6 @@ public class StdOutReport implements Report { dateTimeFormat.setTimeZone(TimeZone.getTimeZone("UTC")); } - /* Cached consensus for later processing. */ - private Status cachedConsensus; - public void processCachedConsensus(Status cachedConsensus) { - this.cachedConsensus = cachedConsensus; - } - /* Downloaded consensus and corresponding votes for later * processing. */ private Status downloadedConsensus; @@ -49,8 +43,6 @@ public class StdOutReport implements Report { this.checkMissingVotes(); this.checkBandwidthScanners(); } - } else if (this.cachedConsensus != null) { - this.checkConsensusValid(this.cachedConsensus); } else { this.warnings.put("No consensus known", 0L); } @@ -242,17 +234,6 @@ public class StdOutReport implements Report { } } - /* Check if the most recent consensus is older than 3 hours. */ - private void checkConsensusValid(Status consensus) { - if (consensus.getValidAfterMillis() < - System.currentTimeMillis() - 3L * 60L * 60L * 1000L) { - this.warnings.put("The last known consensus published at " - + dateTimeFormat.format(consensus.getValidAfterMillis()) - + " is more than 3 hours old and therefore not valid anymore", - 0L); - } - } - /* Prepare a report to be written to stdout. */ private String preparedReport = null; private void prepareReport() { [View Less]

1 0

[metrics-web/master] Fix a String encoding problem with OpenJDK.
by karsten＠torproject.org 20 Oct '11

20 Oct '11

commit 8d02e8529ffc757be17dea188d239c333a62eee1 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Oct 20 13:22:07 2011 +0200 Fix a String encoding problem with OpenJDK. --- .../cron/RelayDescriptorDatabaseImporter.java | 6 +++++- 1 files changed, 5 insertions(+), 1 deletions(-) diff --git a/src/org/torproject/ernie/cron/RelayDescriptorDatabaseImporter.java b/src/org/torproject/ernie/cron/RelayDescriptorDatabaseImporter.java index f8f1e9a..4a9f97c 100644 --- a/… [View More]

1 0

[metrics-web/master] Download consensuses from different authorities in parallel.
by karsten＠torproject.org 20 Oct '11

20 Oct '11

commit f5eceb840d660595dea66b25510e9b934b0e908f Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Oct 20 18:33:45 2011 +0200 Download consensuses from different authorities in parallel. --- src/org/torproject/chc/Downloader.java | 84 +++++++++++++++++++++++-------- 1 files changed, 62 insertions(+), 22 deletions(-) diff --git a/src/org/torproject/chc/Downloader.java b/src/org/torproject/chc/Downloader.java index 17ae71c..e56d20d 100644 --- a/src/org/torproject/chc/… [View More]Downloader.java +++ b/src/org/torproject/chc/Downloader.java @@ -40,20 +40,28 @@ public class Downloader { private SortedMap<String, String> downloadedConsensuses = new TreeMap<String, String>(); private void downloadConsensus() { + Map<String, String> urls = new HashMap<String, String>(); for (Map.Entry<String, String> e : this.authorities.entrySet()) { String nickname = e.getKey(); String address = e.getValue(); String resource = "/tor/status-vote/current/consensus.z"; String fullUrl = "http://" + address + resource; - String response = this.downloadFromAuthority(fullUrl); - if (response != null) { + urls.put(nickname, fullUrl); + } + Map<String, String> responses = + this.downloadFromAuthority(new HashSet<String>(urls.values())); + for (Map.Entry<String, String> e : urls.entrySet()) { + String nickname = e.getKey(); + String url = e.getValue(); + if (responses.containsKey(url)) { + String response = responses.get(url); this.downloadedConsensuses.put(nickname, response); } else { System.err.println("Could not download consensus from directory " + "authority " + nickname + ". Ignoring."); } } - if (this.downloadedConsensuses.isEmpty()) { + if (responses.isEmpty()) { System.err.println("Could not download consensus from any of the " + "directory authorities. Ignoring."); } @@ -65,14 +73,14 @@ public class Downloader { Thread mainThread; String url; String response; - boolean interrupted = false; + boolean finished = false; public DownloadRunnable(String url) { this.mainThread = Thread.currentThread(); this.url = url; } public void run() { try { - URL u = new URL(url); + URL u = new URL(this.url); HttpURLConnection huc = (HttpURLConnection) u.openConnection(); huc.setRequestMethod("GET"); huc.connect(); @@ -83,37 +91,65 @@ public class Downloader { ByteArrayOutputStream baos = new ByteArrayOutputStream(); int len; byte[] data = new byte[1024]; - while (!this.interrupted && + while (!this.finished && (len = in.read(data, 0, 1024)) >= 0) { baos.write(data, 0, len); } - if (this.interrupted) { + if (this.finished) { return; } in.close(); byte[] allData = baos.toByteArray(); this.response = new String(allData); + this.finished = true; this.mainThread.interrupt(); } } catch (IOException e) { /* Can't do much except leaving this.response at null. */ } + this.finished = true; } } - /* Download a consensus or vote from a directory authority using a - * timeout of 60 seconds. */ - private String downloadFromAuthority(final String url) { - DownloadRunnable downloadRunnable = new DownloadRunnable(url); - new Thread(downloadRunnable).start(); - try { - Thread.sleep(60L * 1000L); - } catch (InterruptedException e) { - /* Do nothing. */ + /* Download one or more consensuses or votes from one or more directory + * authorities using a timeout of 60 seconds. */ + private Map<String, String> downloadFromAuthority(Set<String> urls) { + Set<DownloadRunnable> downloadRunnables = + new HashSet<DownloadRunnable>(); + for (String url : urls) { + DownloadRunnable downloadRunnable = new DownloadRunnable(url); + downloadRunnables.add(downloadRunnable); + new Thread(downloadRunnable).start(); + } + long started = System.currentTimeMillis(), sleep; + while ((sleep = started + 60L * 1000L - System.currentTimeMillis()) + > 0L) { + try { + Thread.sleep(sleep); + } catch (InterruptedException e) { + /* Do nothing. */ + } + boolean unfinished = false; + for (DownloadRunnable downloadRunnable : downloadRunnables) { + if (!downloadRunnable.finished) { + unfinished = true; + break; + } + } + if (!unfinished) { + break; + } } - String response = downloadRunnable.response; - downloadRunnable.interrupted = true; - return response; + Map<String, String> responses = new HashMap<String, String>(); + for (DownloadRunnable downloadRunnable : downloadRunnables) { + String url = downloadRunnable.url; + String response = downloadRunnable.response; + if (response != null) { + responses.put(url, response); + } + downloadRunnable.finished = true; + } + return responses; } /* Date-time formats to parse and format timestamps. */ @@ -125,7 +161,7 @@ public class Downloader { /* Parse the downloaded consensus to find fingerprints of directory * authorities publishing the corresponding votes. */ - private List<String> fingerprints = new ArrayList<String>(); + private SortedSet<String> fingerprints = new TreeSet<String>(); private void parseConsensusToFindReferencedVotes() { for (String downloadedConsensus : this.downloadedConsensuses.values()) { @@ -189,8 +225,12 @@ public class Downloader { String resource = "/tor/status-vote/current/" + fingerprint + ".z"; String fullUrl = "http://" + authority + resource; - downloadedVote = this.downloadFromAuthority(fullUrl); - if (downloadedVote != null) { + Set<String> urls = new HashSet<String>(); + urls.add(fullUrl); + Map<String, String> downloadedVotes = + this.downloadFromAuthority(urls); + if (downloadedVotes.containsKey(fullUrl)) { + downloadedVote = downloadedVotes.get(fullUrl); this.downloadedVotes.add(downloadedVote); } } [View Less]

1 0

[metrics-web/master] Check the consensus from each authority.
by karsten＠torproject.org 20 Oct '11

20 Oct '11

commit 7ebdf3fc4f4eac3fc9b557e42b5ce75fffe0c571 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Oct 20 16:22:28 2011 +0200 Check the consensus from each authority. So far, we downloaded the consensus from the authorities in random order and stopped after receiving the first valid response. The problem is that some authorities may reply with a non-fresh consensus whereas others reply with a fresh one. We now download the consensus from each … [View More]authority and warn if one or more of them isn't fresh anymore. We also warn if an authority doesn't return a consensus within a timeout of 60 seconds at all. Implements #4250. --- src/org/torproject/chc/Downloader.java | 56 ++++++++++-------- src/org/torproject/chc/Main.java | 10 +-- src/org/torproject/chc/MetricsWebsiteReport.java | 17 ++++- src/org/torproject/chc/NagiosReport.java | 69 +++++++++++++++++++-- src/org/torproject/chc/Parser.java | 34 ++++++++--- src/org/torproject/chc/Report.java | 4 +- src/org/torproject/chc/StdOutReport.java | 70 +++++++++++++++++++-- 7 files changed, 203 insertions(+), 57 deletions(-) diff --git a/src/org/torproject/chc/Downloader.java b/src/org/torproject/chc/Downloader.java index cf451ee..17ae71c 100644 --- a/src/org/torproject/chc/Downloader.java +++ b/src/org/torproject/chc/Downloader.java @@ -14,41 +14,46 @@ public class Downloader { /* List of directory authorities to download consensuses and votes * from. */ - private static final List<String> AUTHORITIES = - new ArrayList<String>(Arrays.asList(("212.112.245.170,86.59.21.38," - + "216.224.124.114:9030,213.115.239.118:443,193.23.244.244," - + "208.83.223.34:443,128.31.0.34:9131,194.109.206.212"). - split(","))); + private SortedMap<String, String> authorities = + new TreeMap<String, String>(); + public Downloader() { + this.authorities.put("gabelmoo", "212.112.245.170"); + this.authorities.put("tor26", "86.59.21.38"); + this.authorities.put("ides", "216.224.124.114:9030"); + this.authorities.put("maatuska", "213.115.239.118:443"); + this.authorities.put("dannenberg", "193.23.244.244"); + this.authorities.put("urras", "208.83.223.34:443"); + this.authorities.put("moria1", "128.31.0.34:9131"); + this.authorities.put("dizum", "194.109.206.212"); + } /* Download a new consensus and corresponding votes. */ public void downloadFromAuthorities() { this.downloadConsensus(); - if (this.downloadedConsensus != null) { + if (!this.downloadedConsensuses.isEmpty()) { this.parseConsensusToFindReferencedVotes(); this.downloadReferencedVotes(); } } - /* Download the most recent consensus. */ - private String downloadedConsensus; + /* Download the most recent consensus from all authorities. */ + private SortedMap<String, String> downloadedConsensuses = + new TreeMap<String, String>(); private void downloadConsensus() { - List<String> authorities = new ArrayList<String>(AUTHORITIES); - Collections.shuffle(authorities); - for (String authority : authorities) { - if (this.downloadedConsensus != null) { - break; - } + for (Map.Entry<String, String> e : this.authorities.entrySet()) { + String nickname = e.getKey(); + String address = e.getValue(); String resource = "/tor/status-vote/current/consensus.z"; - String fullUrl = "http://" + authority + resource; + String fullUrl = "http://" + address + resource; String response = this.downloadFromAuthority(fullUrl); if (response != null) { - this.downloadedConsensus = response; + this.downloadedConsensuses.put(nickname, response); } else { System.err.println("Could not download consensus from directory " - + "authority " + authority + ". Ignoring."); + + "authority " + nickname + ". Ignoring."); } } - if (this.downloadedConsensus == null) { + if (this.downloadedConsensuses.isEmpty()) { System.err.println("Could not download consensus from any of the " + "directory authorities. Ignoring."); } @@ -122,10 +127,11 @@ public class Downloader { * authorities publishing the corresponding votes. */ private List<String> fingerprints = new ArrayList<String>(); private void parseConsensusToFindReferencedVotes() { - if (this.downloadedConsensus != null) { + for (String downloadedConsensus : + this.downloadedConsensuses.values()) { try { BufferedReader br = new BufferedReader(new StringReader( - this.downloadedConsensus)); + downloadedConsensus)); String line; while ((line = br.readLine()) != null) { if (line.startsWith("valid-after ")) { @@ -173,7 +179,8 @@ public class Downloader { private void downloadReferencedVotes() { for (String fingerprint : this.fingerprints) { String downloadedVote = null; - List<String> authorities = new ArrayList<String>(AUTHORITIES); + List<String> authorities = new ArrayList<String>( + this.authorities.values()); Collections.shuffle(authorities); for (String authority : authorities) { if (downloadedVote != null) { @@ -190,9 +197,10 @@ public class Downloader { } } - /* Return the previously downloaded (unparsed) consensus string. */ - public String getConsensusString() { - return this.downloadedConsensus; + /* Return the previously downloaded (unparsed) consensus string by + * authority nickname. */ + public SortedMap<String, String> getConsensusStrings() { + return this.downloadedConsensuses; } /* Return the previously downloaded (unparsed) vote strings. */ diff --git a/src/org/torproject/chc/Main.java b/src/org/torproject/chc/Main.java index d42ee44..aa25e57 100644 --- a/src/org/torproject/chc/Main.java +++ b/src/org/torproject/chc/Main.java @@ -23,12 +23,10 @@ public class Main { /* Parse consensus and votes and pass them to the reports. */ Parser parser = new Parser(); - Status parsedDownloadedConsensus = parser.parse( - downloader.getConsensusString(), downloader.getVoteStrings()); - if (parsedDownloadedConsensus != null) { - for (Report report : reports) { - report.processDownloadedConsensus(parsedDownloadedConsensus); - } + SortedMap<String, Status> parsedDownloadedConsensuses = parser.parse( + downloader.getConsensusStrings(), downloader.getVoteStrings()); + for (Report report : reports) { + report.processDownloadedConsensuses(parsedDownloadedConsensuses); } /* Finish writing reports. */ diff --git a/src/org/torproject/chc/MetricsWebsiteReport.java b/src/org/torproject/chc/MetricsWebsiteReport.java index f77b89f..01d126c 100644 --- a/src/org/torproject/chc/MetricsWebsiteReport.java +++ b/src/org/torproject/chc/MetricsWebsiteReport.java @@ -29,9 +29,20 @@ public class MetricsWebsiteReport implements Report { * processing. */ private Status downloadedConsensus; private SortedSet<Status> downloadedVotes; - public void processDownloadedConsensus(Status downloadedConsensus) { - this.downloadedConsensus = downloadedConsensus; - this.downloadedVotes = downloadedConsensus.getVotes(); + public void processDownloadedConsensuses( + SortedMap<String, Status> downloadedConsensuses) { + long mostRecentValidAfterMillis = -1L; + for (Status downloadedConsensus : downloadedConsensuses.values()) { + if (downloadedConsensus.getValidAfterMillis() > + mostRecentValidAfterMillis) { + this.downloadedConsensus = downloadedConsensus; + mostRecentValidAfterMillis = + downloadedConsensus.getValidAfterMillis(); + } + } + if (this.downloadedConsensus != null) { + this.downloadedVotes = this.downloadedConsensus.getVotes(); + } } /* Writer to write all HTML output to. */ diff --git a/src/org/torproject/chc/NagiosReport.java b/src/org/torproject/chc/NagiosReport.java index 2b5a6d3..a9f27af 100644 --- a/src/org/torproject/chc/NagiosReport.java +++ b/src/org/torproject/chc/NagiosReport.java @@ -20,11 +20,12 @@ public class NagiosReport implements Report { /* Store the current consensus and corresponding votes for * processing. */ + private SortedMap<String, Status> downloadedConsensuses; private Status downloadedConsensus; private SortedSet<Status> downloadedVotes; - public void processDownloadedConsensus(Status downloadedConsensus) { - this.downloadedConsensus = downloadedConsensus; - this.downloadedVotes = downloadedConsensus.getVotes(); + public void processDownloadedConsensuses( + SortedMap<String, Status> downloadedConsensuses) { + this.downloadedConsensuses = downloadedConsensuses; } /* Lists of output messages sorted by warnings, criticals, and @@ -43,6 +44,9 @@ public class NagiosReport implements Report { /* Check consensus and votes and write any findings to the output * file. */ public void writeReport() { + this.findMostRecentConsensus(); + this.checkMissingConsensuses(); + this.checkAllConsensusesFresh(); if (this.downloadedConsensus != null) { if (this.isConsensusFresh(this.downloadedConsensus)) { this.checkConsensusMethods(); @@ -58,13 +62,66 @@ public class NagiosReport implements Report { this.writeNagiosStatusFile(); } + /* Find most recent consensus and corresponding votes. */ + private void findMostRecentConsensus() { + long mostRecentValidAfterMillis = -1L; + for (Status downloadedConsensus : downloadedConsensuses.values()) { + if (downloadedConsensus.getValidAfterMillis() > + mostRecentValidAfterMillis) { + this.downloadedConsensus = downloadedConsensus; + mostRecentValidAfterMillis = + downloadedConsensus.getValidAfterMillis(); + } + } + if (this.downloadedConsensus != null) { + this.downloadedVotes = this.downloadedConsensus.getVotes(); + } + } + + /* Check if any directory authority didn't tell us a consensus. */ + private void checkMissingConsensuses() { + SortedSet<String> missingConsensuses = new TreeSet<String>( + Arrays.asList(("gabelmoo,tor26,ides,maatuska,dannenberg,urras," + + "moria1,dizum").split(","))); + missingConsensuses.removeAll(this.downloadedConsensuses.keySet()); + if (!missingConsensuses.isEmpty()) { + StringBuilder sb = new StringBuilder(); + for (String nickname : missingConsensuses) { + sb.append(", " + nickname); + } + this.nagiosCriticals.add("The following directory authorities did " + + "not return a consensus within a timeout of 60 seconds: " + + sb.toString().substring(2)); + } + } + + /* Check if all consensuses are fresh. */ + private void checkAllConsensusesFresh() { + long fresh = System.currentTimeMillis() - 60L * 60L * 1000L; + SortedSet<String> nonFresh = new TreeSet<String>(); + for (Map.Entry<String, Status> e : downloadedConsensuses.entrySet()) { + String nickname = e.getKey(); + Status downloadedConsensus = e.getValue(); + if (downloadedConsensus.getValidAfterMillis() < fresh) { + nonFresh.add(nickname); + } + } + if (!nonFresh.isEmpty()) { + StringBuilder sb = new StringBuilder(); + for (String nickname : nonFresh) { + sb.append(", " + nickname); + } + this.nagiosCriticals.add("The consensuses published by the " + + "following directory authorities are more than 1 hour old " + + "and therefore not fresh anymore: " + + sb.toString().substring(2)); + } + } + /* Check if the most recent consensus is older than 1 hour. */ private boolean isConsensusFresh(Status consensus) { if (consensus.getValidAfterMillis() < System.currentTimeMillis() - 60L * 60L * 1000L) { - this.nagiosCriticals.add("The last known consensus published at " - + dateTimeFormat.format(consensus.getValidAfterMillis()) - + " is more than 1 hour old and therefore not fresh anymore"); return false; } else { return true; diff --git a/src/org/torproject/chc/Parser.java b/src/org/torproject/chc/Parser.java index 47609a8..053c7b4 100644 --- a/src/org/torproject/chc/Parser.java +++ b/src/org/torproject/chc/Parser.java @@ -12,18 +12,34 @@ public class Parser { /* Parse and return a consensus and corresponding votes, or null if * something goes wrong. */ - public Status parse(String consensusString, List<String> voteStrings) { - Status consensus = this.parseConsensusOrVote(consensusString, true); - if (consensus != null) { - for (String voteString : voteStrings) { - Status vote = this.parseConsensusOrVote(voteString, false); - if (consensus.getValidAfterMillis() == - vote.getValidAfterMillis()) { - consensus.addVote(vote); + public SortedMap<String, Status> parse( + SortedMap<String, String> consensusStrings, + List<String> voteStrings) { + SortedSet<Status> parsedVotes = new TreeSet<Status>(); + for (String voteString : voteStrings) { + Status parsedVote = this.parseConsensusOrVote(voteString, false); + if (parsedVote != null) { + parsedVotes.add(parsedVote); + } + } + SortedMap<String, Status> parsedConsensuses = + new TreeMap<String, Status>(); + for (Map.Entry<String, String> e : consensusStrings.entrySet()) { + String nickname = e.getKey(); + String consensusString = e.getValue(); + Status parsedConsensus = this.parseConsensusOrVote(consensusString, + true); + if (parsedConsensus != null) { + for (Status parsedVote : parsedVotes) { + if (parsedConsensus.getValidAfterMillis() == + parsedVote.getValidAfterMillis()) { + parsedConsensus.addVote(parsedVote); + } } + parsedConsensuses.put(nickname, parsedConsensus); } } - return consensus; + return parsedConsensuses; } /* Date-time formats to parse and format timestamps. */ diff --git a/src/org/torproject/chc/Report.java b/src/org/torproject/chc/Report.java index c3937e0..0f1a306 100644 --- a/src/org/torproject/chc/Report.java +++ b/src/org/torproject/chc/Report.java @@ -10,8 +10,8 @@ public interface Report { /* Process the downloaded current consensus and corresponding votes to * find irregularities between them. */ - public abstract void processDownloadedConsensus( - Status downloadedConsensus); + public abstract void processDownloadedConsensuses( + SortedMap<String, Status> downloadedConsensuses); /* Finish writing report. */ public abstract void writeReport(); diff --git a/src/org/torproject/chc/StdOutReport.java b/src/org/torproject/chc/StdOutReport.java index 15a9148..f683ae6 100644 --- a/src/org/torproject/chc/StdOutReport.java +++ b/src/org/torproject/chc/StdOutReport.java @@ -23,17 +23,21 @@ public class StdOutReport implements Report { /* Downloaded consensus and corresponding votes for later * processing. */ + private SortedMap<String, Status> downloadedConsensuses; private Status downloadedConsensus; private SortedSet<Status> downloadedVotes; - public void processDownloadedConsensus(Status downloadedConsensus) { - this.downloadedConsensus = downloadedConsensus; - this.downloadedVotes = downloadedConsensus.getVotes(); + public void processDownloadedConsensuses( + SortedMap<String, Status> downloadedConsensuses) { + this.downloadedConsensuses = downloadedConsensuses; } /* Check consensuses and votes for irregularities and write output to * stdout. */ public void writeReport() { this.readLastWarned(); + this.findMostRecentConsensus(); + this.checkMissingConsensuses(); + this.checkAllConsensusesFresh(); if (this.downloadedConsensus != null) { if (this.isConsensusFresh(this.downloadedConsensus)) { this.checkConsensusMethods(); @@ -87,14 +91,66 @@ public class StdOutReport implements Report { } } + /* Find most recent consensus and corresponding votes. */ + private void findMostRecentConsensus() { + long mostRecentValidAfterMillis = -1L; + for (Status downloadedConsensus : downloadedConsensuses.values()) { + if (downloadedConsensus.getValidAfterMillis() > + mostRecentValidAfterMillis) { + this.downloadedConsensus = downloadedConsensus; + mostRecentValidAfterMillis = + downloadedConsensus.getValidAfterMillis(); + } + } + if (this.downloadedConsensus != null) { + this.downloadedVotes = this.downloadedConsensus.getVotes(); + } + } + + /* Check if any directory authority didn't tell us a consensus. */ + private void checkMissingConsensuses() { + SortedSet<String> missingConsensuses = new TreeSet<String>( + Arrays.asList(("gabelmoo,tor26,ides,maatuska,dannenberg,urras," + + "moria1,dizum").split(","))); + missingConsensuses.removeAll(this.downloadedConsensuses.keySet()); + if (!missingConsensuses.isEmpty()) { + StringBuilder sb = new StringBuilder(); + for (String nickname : missingConsensuses) { + sb.append(", " + nickname); + } + this.warnings.put("The following directory authorities did not " + + "return a consensus within a timeout of 60 seconds: " + + sb.toString().substring(2), 150L * 60L * 1000L); + } + } + + /* Check if all consensuses are fresh. */ + private void checkAllConsensusesFresh() { + long fresh = System.currentTimeMillis() - 60L * 60L * 1000L; + SortedSet<String> nonFresh = new TreeSet<String>(); + for (Map.Entry<String, Status> e : downloadedConsensuses.entrySet()) { + String nickname = e.getKey(); + Status downloadedConsensus = e.getValue(); + if (downloadedConsensus.getValidAfterMillis() < fresh) { + nonFresh.add(nickname); + } + } + if (!nonFresh.isEmpty()) { + StringBuilder sb = new StringBuilder(); + for (String nickname : nonFresh) { + sb.append(", " + nickname); + } + this.warnings.put("The consensuses published by the following " + + "directory authorities are more than 1 hour old and " + + "therefore not fresh anymore: " + + sb.toString().substring(2), 150L * 60L * 1000L); + } + } + /* Check if the most recent consensus is older than 1 hour. */ private boolean isConsensusFresh(Status consensus) { if (consensus.getValidAfterMillis() < System.currentTimeMillis() - 60L * 60L * 1000L) { - this.warnings.put("The last known consensus published at " - + dateTimeFormat.format(consensus.getValidAfterMillis()) - + " is more than 1 hour old and therefore not fresh anymore", - 0L); return false; } else { return true; [View Less]

1 0

r25177: {website} swap torstatus and torbuttton positions in the project list. (website/trunk/en)
by Andrew Lewman 20 Oct '11

20 Oct '11

Author: phobos Date: 2011-10-20 14:18:49 +0000 (Thu, 20 Oct 2011) New Revision: 25177 Modified: website/trunk/en/index.wml Log: swap torstatus and torbuttton positions in the project list. Modified: website/trunk/en/index.wml =================================================================== --- website/trunk/en/index.wml 2011-10-19 05:40:15 UTC (rev 25176) +++ website/trunk/en/index.wml 2011-10-20 14:18:49 UTC (rev 25177) @@ -75,24 +75,21 @@ </td> </… [View More]tr> <tr> -  <td> <div class="project"> - <a href="http://torstatus.blutmagie.de/"><img - src="$(IMGROOT)/icon-TorStatus.jpg" alt="TorStatus Icon"></a> - <h3><a href="http://torstatus.blutmagie.de/">TorStatus</a></h3> - <p>Site providing an overview of the Tor network.</p> + <a href="<page projects/torbrowser>"><img + src="$(IMGROOT)/icon-TorBrowser.jpg" alt="TorBrowser Icon"></a> + <h3><a href="<page projects/torbrowser>">Tor + Browser</a></h3> + <p>Tor Browser contains everything you need to safely + browse the Internet.</p> </div> </td> -  + --> <td> <div class="project"> <a href="http://www.atagar.com/arm/"><img @@ -103,14 +100,17 @@ </td> </tr> <tr> - <td> - <div class="project"> - <a href="<page projects/torbrowser>"><img - src="$(IMGROOT)/icon-TorBrowser.jpg" alt="TorBrowser Icon"></a> - <h3><a href="<page projects/torbrowser>">Tor - Browser</a></h3> - <p>Tor Browser contains everything you need to safely - browse the Internet.</p> +  + <td> + <div class="project"> + <a href="http://torstatus.blutmagie.de/"><img + src="$(IMGROOT)/icon-TorStatus.jpg" alt="TorStatus Icon"></a> + <h3><a href="http://torstatus.blutmagie.de/">TorStatus</a></h3> + <p>Site providing an overview of the Tor network.</p> </div> </td> <td> @@ -125,7 +125,7 @@ </tr> <tr> <td> - <div class="project"> + <div class="project"> <a href="<page projects/vidalia>"><img src="$(IMGROOT)/icon-Vidalia.jpg" alt="Vidalia Icon"></a> <h3><a href="<page projects/vidalia>">Vidalia</a></h3> [View Less]

1 0

[translation/gettor] Update translations for gettor
by translation＠torproject.org 20 Oct '11

20 Oct '11

commit 5304d1eeb1b1acab12f7d2d6f816af3235582224 Author: Translation commit bot <translation(a)torproject.org> Date: Thu Oct 20 06:15:22 2011 +0000 Update translations for gettor --- es/gettor.po | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/es/gettor.po b/es/gettor.po index 7fb5996..83d75df 100644 --- a/es/gettor.po +++ b/es/gettor.po @@ -10,7 +10,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.… [View More]

1 0

[translation/vidalia_help] Update translations for vidalia_help
by translation＠torproject.org 20 Oct '11

20 Oct '11

commit 5bfcd7aba4fe29dde48a1b441b17f935c127ee80 Author: Translation commit bot <translation(a)torproject.org> Date: Thu Oct 20 06:04:16 2011 +0000 Update translations for vidalia_help --- es/troubleshooting.po | 13 +++++++++---- 1 files changed, 9 insertions(+), 4 deletions(-) diff --git a/es/troubleshooting.po b/es/troubleshooting.po index 7ff51b7..742a971 100644 --- a/es/troubleshooting.po +++ b/es/troubleshooting.po @@ -1,12 +1,13 @@ # # Translators: +# <lenazun(a)… [View More]

1 0

[translation/tsum] Update translations for tsum
by translation＠torproject.org 20 Oct '11

20 Oct '11

commit 82cc6a8865e9c9ee5ef3845bae364601f8d0d4f4 Author: Translation commit bot <translation(a)torproject.org> Date: Thu Oct 20 05:45:40 2011 +0000 Update translations for tsum --- ar/tsum.po | 12 +++++++++++- 1 files changed, 11 insertions(+), 1 deletions(-) diff --git a/ar/tsum.po b/ar/tsum.po index 6e28e9f..121fe9e 100644 --- a/ar/tsum.po +++ b/ar/tsum.po @@ -11,7 +11,7 @@ msgstr "" "Project-Id-Version: The Tor Project\n" "Report-Msgid-Bugs-To: https://trac.torproject.org/… [View More]projects/tor\n" "POT-Creation-Date: 2011-10-17 13:23+0200\n" -"PO-Revision-Date: 2011-10-20 05:14+0000\n" +"PO-Revision-Date: 2011-10-20 05:25+0000\n" "Last-Translator: sharafaldin <sudanhacker(a)hotmail.com>\n" "Language-Team: LANGUAGE <LL(a)li.org>\n" "MIME-Version: 1.0\n" @@ -33,6 +33,9 @@ msgid "" "find the answer to your question in this document, email community-" "support(a)lists.torproject.org." msgstr "" +"دليل المستخدم يحتوي على معلومات عن كيفية تحميل تور وكيف تقوم بإستخدامه وماذا" +" تفعل في حال عدم مقدرة تور على الإتصال بالشبكة . إذا لم تجد إجابة على سؤالك " +"في هذا المستند قم بإرسال رسالة إلي community-support(a)lists.torproject.org." #. type: Plain text #: tsum.text:9 @@ -542,6 +545,8 @@ msgid "" "\".exe\" file in the same directory. Run this file and wait for the Tor " "Browser Bundle to start." msgstr "" +"مرة واحدة في عملية التفريغ انتهت ، يجب أن تشاهد المنشأة حديثا \". إكس\" ملف " +"في نفس الدليل. تشغيل هذا الملف وانتظر حزمة متصفح تور للبدء." #. type: Plain text #: tsum.text:245 @@ -637,6 +642,9 @@ msgid "" "technically possible to use Tor with other browsers, but by doing so you " "open yourself up to potential attacks." msgstr "" +"### ارغب في استخدام متصفح آخر لأسباب أمنية ، نحن ننصحك فقط بالتصفح عن طريق " +"تور وذلك بإستخدام حزمة متصفح تور . تقنياً يمكنك استخدام تور مع متصفحات أخرى " +"، ولكن عندما تفعل ذلك تصبح عرضة لعديد من الهجمات" #. type: Plain text #: tsum.text:289 @@ -645,5 +653,7 @@ msgid "" "Internet connection. After all, your traffic is sent through many different" " countries, sometimes across oceans around the world!" msgstr "" +"### لماذا تور بطئ \n" +"تور في بعض الأحيان يكون ابطأ من الإنترنت العادي وذلك لأن بياناتك تعبر عدد من الدول المختلفة بل في بعض الأحيان تعبر محيطات حول العالم" [View Less]

1 0