June 2017 - tor-commits - lists.torproject.org

[tor/release-0.2.9] Merge branch 'maint-0.2.5' into maint-0.2.6
by nickm＠torproject.org 27 Jun '17

27 Jun '17

commit 3de27618e65affe3688a5d92bce96e1e4ed5a82a Merge: 3913f95 ccae991 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jun 27 11:04:44 2017 -0400 Merge branch 'maint-0.2.5' into maint-0.2.6 changes/bug22737 | 12 ++++++++++++ src/or/connection_or.c | 4 +++- 2 files changed, 15 insertions(+), 1 deletion(-)

1 0

[tor/release-0.2.4] Merge branch 'bug22737_024' into maint-0.2.4
by nickm＠torproject.org 27 Jun '17

27 Jun '17

commit d56f6993990ab75b6ab8a80027ee60e610489a2c Merge: 104e8fa 8d2978b Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jun 27 11:04:41 2017 -0400 Merge branch 'bug22737_024' into maint-0.2.4 changes/bug22737 | 12 ++++++++++++ src/or/connection_or.c | 4 +++- 2 files changed, 15 insertions(+), 1 deletion(-)

1 0

[metrics-lib/master] Remove all deprecation warnings. Test might not compile yet.
by karsten＠torproject.org 27 Jun '17

27 Jun '17

commit a56c96d66c880e60ddf818697b006502f3177eac Author: iwakeh <iwakeh(a)torproject.org> Date: Fri Jun 23 08:37:42 2017 +0000 Remove all deprecation warnings. Test might not compile yet. Part of task-22154. --- .../descriptor/DescriptorDownloader.java | 201 ------------- .../org/torproject/descriptor/DescriptorFile.java | 84 ------ .../descriptor/DescriptorParseException.java | 4 +- .../torproject/descriptor/DescriptorParser.java | 17 -- .../torproject/descriptor/DescriptorReader.java | 84 ------ .../torproject/descriptor/DescriptorRequest.java | 104 ------- .../descriptor/DescriptorSourceFactory.java | 40 +-- .../org/torproject/descriptor/DirSourceEntry.java | 10 - .../descriptor/DirectoryKeyCertificate.java | 10 - .../torproject/descriptor/DirectorySignature.java | 11 - .../java/org/torproject/descriptor/ExitList.java | 11 - .../org/torproject/descriptor/ExitListEntry.java | 55 ---- .../torproject/descriptor/ExtraInfoDescriptor.java | 47 --- .../org/torproject/descriptor/Microdescriptor.java | 11 - .../torproject/descriptor/NetworkStatusEntry.java | 11 - .../org/torproject/descriptor/RelayDirectory.java | 11 - .../torproject/descriptor/RelayNetworkStatus.java | 11 - .../descriptor/RelayNetworkStatusConsensus.java | 24 -- .../descriptor/RelayNetworkStatusVote.java | 29 -- .../torproject/descriptor/ServerDescriptor.java | 48 +-- .../impl/BridgeExtraInfoDescriptorImpl.java | 7 +- .../descriptor/impl/BridgeNetworkStatusImpl.java | 16 +- .../descriptor/impl/BridgePoolAssignmentImpl.java | 3 +- .../impl/BridgeServerDescriptorImpl.java | 7 +- .../descriptor/impl/DescriptorCollectorImpl.java | 264 ----------------- .../descriptor/impl/DescriptorDownloaderImpl.java | 294 ------------------- .../descriptor/impl/DescriptorFileImpl.java | 93 ------ .../torproject/descriptor/impl/DescriptorImpl.java | 13 +- .../descriptor/impl/DescriptorParseException.java | 23 -- .../descriptor/impl/DescriptorParserImpl.java | 55 ++-- .../descriptor/impl/DescriptorReaderImpl.java | 145 +-------- .../descriptor/impl/DescriptorRequestImpl.java | 137 --------- .../descriptor/impl/DirSourceEntryImpl.java | 23 +- .../descriptor/impl/DirectoryDownloader.java | 111 ------- .../impl/DirectoryKeyCertificateImpl.java | 18 +- .../descriptor/impl/DirectorySignatureImpl.java | 23 +- .../descriptor/impl/DownloadCoordinator.java | 11 - .../descriptor/impl/DownloadCoordinatorImpl.java | 323 --------------------- .../descriptor/impl/ExitListEntryImpl.java | 69 +---- .../torproject/descriptor/impl/ExitListImpl.java | 27 +- .../descriptor/impl/ExtraInfoDescriptorImpl.java | 50 +--- .../descriptor/impl/MicrodescriptorImpl.java | 21 +- .../descriptor/impl/NetworkStatusEntryImpl.java | 24 +- .../descriptor/impl/NetworkStatusImpl.java | 13 +- .../descriptor/impl/RelayDirectoryImpl.java | 27 +- .../impl/RelayExtraInfoDescriptorImpl.java | 7 +- .../impl/RelayNetworkStatusConsensusImpl.java | 36 +-- .../descriptor/impl/RelayNetworkStatusImpl.java | 14 +- .../impl/RelayNetworkStatusVoteImpl.java | 43 +-- .../descriptor/impl/RelayServerDescriptorImpl.java | 7 +- .../descriptor/impl/ServerDescriptorImpl.java | 48 +-- .../descriptor/impl/TorperfResultImpl.java | 14 +- .../descriptor/impl/UnparseableDescriptorImpl.java | 4 +- .../descriptor/DummyDownloaderImplementation.java | 15 - .../impl/DescriptorCollectorImplTest.java | 121 -------- 55 files changed, 136 insertions(+), 2793 deletions(-) diff --git a/src/main/java/org/torproject/descriptor/DescriptorDownloader.java b/src/main/java/org/torproject/descriptor/DescriptorDownloader.java deleted file mode 100644 index 52b25bf..0000000 --- a/src/main/java/org/torproject/descriptor/DescriptorDownloader.java +++ /dev/null @@ -1,201 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor; - -import java.util.Iterator; -import java.util.Set; - -/** - * Descriptor source that downloads relay descriptors from directory - * authorities or mirrors. - * - * Downloading descriptors is done in a batch which starts after - * setting any configuration options and initiating the download - * process. - * - * @deprecated Removed as descriptor source in favor of the much more widely - * used {@link DescriptorCollector}. - * - * @since 1.0.0 - */ -public interface DescriptorDownloader { - - /** - * Add a directory authority to download descriptors from, which is - * only required for downloading network status votes and will be used - * when no directory mirrors are available. - * - * @since 1.0.0 - */ - public void addDirectoryAuthority(String nickname, String ip, - int dirPort); - - /** - * Add a directory mirror to download descriptors from, which is - * preferred for downloading descriptors, except for network status - * votes which are only available on directory authorities. - * - * @since 1.0.0 - */ - public void addDirectoryMirror(String nickname, String ip, int dirPort); - - /** - * Include the current network status consensus in the downloads. - * - * @since 1.0.0 - */ - public void setIncludeCurrentConsensus(); - - /** - * Include the current network status consensus in the downloads, and - * attempt to download it from all directory authorities. - * - * The primary purpose of doing this is to compare different - * consensuses and download characteristics to each other. Typically, - * downloading from a single directory mirror or authority is - * sufficient. - * - * @since 1.0.0 - */ - public void setIncludeCurrentConsensusFromAllDirectoryAuthorities(); - - /** - * Include the current network status votes referenced from a - * previously downloaded consensus in the downloads, which requires - * downloading the current consensus from at least one directory mirror - * or authority. - * - * @since 1.0.0 - */ - public void setIncludeCurrentReferencedVotes(); - - /** - * Include the current network status vote published by the given - * directory authority in the downloads, which requires downloading from - * at least one directory authority. - * - * @since 1.0.0 - */ - public void setIncludeCurrentVote(String fingerprint); - - /** - * Include the current network status votes published by the given - * directory authorities in the downloads, which requires downloading - * from at least one directory authority. - * - * @since 1.0.0 - */ - public void setIncludeCurrentVotes(Set<String> fingerprints); - - /** - * Include all server descriptors referenced from a previously - * downloaded network status consensus in the downloads. - * - * @since 1.0.0 - */ - public void setIncludeReferencedServerDescriptors(); - - /** - * Exclude the server descriptor with the given identifier from the - * downloads even if it's referenced from a consensus and we're supposed - * to download all referenced server descriptors. - * - * @since 1.0.0 - */ - public void setExcludeServerDescriptor(String identifier); - - /** - * Exclude the server descriptors with the given identifiers from the - * downloads even if they are referenced from a consensus and we're - * supposed to download all referenced server descriptors. - * - * @since 1.0.0 - */ - public void setExcludeServerDescriptors(Set<String> identifier); - - /** - * Include all extra-info descriptors referenced from previously - * downloaded server descriptors in the downloads. - * - * @since 1.0.0 - */ - public void setIncludeReferencedExtraInfoDescriptors(); - - /** - * Exclude the extra-info descriptor with the given identifier from the - * downloads even if it's referenced from a server descriptor and we're - * supposed to download all referenced extra-info descriptors. - * - * @since 1.0.0 - */ - public void setExcludeExtraInfoDescriptor(String identifier); - - /** - * Exclude the extra-info descriptors with the given identifiers from - * the downloads even if they are referenced from server descriptors - * and we're supposed to download all referenced extra-info - * descriptors. - * - * @since 1.0.0 - */ - public void setExcludeExtraInfoDescriptors(Set<String> identifiers); - - /** - * Define a connect timeout for a single request. - * - * If a timeout expires, no further requests will be sent to the - * directory authority or mirror. Setting this value to 0 disables the - * connect timeout. Default value is 1 minute (60 * 1000). - * - * @since 1.0.0 - */ - public void setConnectTimeout(long connectTimeoutMillis); - - /** - * Define a read timeout for a single request. - * - * If a timeout expires, no further requests will be sent to the - * directory authority or mirror. Setting this value to 0 disables the - * read timeout. Default value is 1 minute (60 * 1000). - * - * @since 1.0.0 - */ - public void setReadTimeout(long readTimeoutMillis); - - /** - * Define a global timeout for all requests. - * - * Once this timeout expires, all running requests are aborted and no - * further requests are made. Setting this value to 0 disables the - * global timeout. Default is 1 hour (60 * 60 * 1000). - * - * @since 1.0.0 - */ - public void setGlobalTimeout(long globalTimeoutMillis); - - /** - * Fail descriptor parsing when encountering an unrecognized line. - * - * This option is not set by default, because the Tor specifications - * allow for new lines to be added that shall be ignored by older Tor - * versions. But some applications may want to handle unrecognized - * descriptor lines explicitly. - * - * @since 1.0.0 - */ - public void setFailUnrecognizedDescriptorLines(); - - /** - * Download the previously configured relay descriptors and make them - * available via the returned blocking iterator. - * - * Whenever the downloader runs out of descriptors and expects to - * provide more shortly after, it blocks the caller. This method can - * only be run once. - * - * @since 1.0.0 - */ - public Iterator<DescriptorRequest> downloadDescriptors(); -} - diff --git a/src/main/java/org/torproject/descriptor/DescriptorFile.java b/src/main/java/org/torproject/descriptor/DescriptorFile.java deleted file mode 100644 index e8c0d21..0000000 --- a/src/main/java/org/torproject/descriptor/DescriptorFile.java +++ /dev/null @@ -1,84 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor; - -import java.io.File; -import java.util.List; - -/** - * Container for descriptors read from a file. - * - * When the {@link DescriptorReader} reads descriptors from local files - * it provides an iterator over these containers which in turn contain - * references to classes implementing the {@link Descriptor} interface. - * This container also stores potentially useful meta-data about the - * descriptor file. - * - * @deprecated Irrelevant after introducing - * {@link DescriptorReader#readDescriptors(File...)}, which returns - * {@link Descriptor}s rather than {@link DescriptorFile}s, and including - * the descriptor file reference in {@link Descriptor#getDescriptorFile()} - * and the first thrown {@link DescriptorParseException} in - * {@link UnparseableDescriptor#getDescriptorParseException()}. - * - * @since 1.0.0 - */ -public interface DescriptorFile { - - /** - * Return the directory where this descriptor file was contained, or - * null if the file was contained in a tarball. - * - * @since 1.0.0 - */ - public File getDirectory(); - - /** - * Return the tarball where this descriptor file was contained, or null - * if the file was not contained in a tarball. - * - * @since 1.0.0 - */ - public File getTarball(); - - /** - * Return the descriptor file itself, or null if the descriptor file - * was contained in a tarball. - * - * @since 1.0.0 - */ - public File getFile(); - - /** - * Return the descriptor file name, which is either the absolute path - * of the file on disk, or the tar file entry name. - * - * @since 1.0.0 - */ - public String getFileName(); - - /** - * Return the time in milliseconds since the epoch when the descriptor - * file on disk was last modified. - * - * @since 1.0.0 - */ - public long getLastModified(); - - /** - * Return the descriptors contained in the descriptor file. - * - * @since 1.0.0 - */ - public List<Descriptor> getDescriptors(); - - /** - * Return the first exception that was thrown when reading this file or - * parsing its content, or null if no exception was thrown. - * - * @since 1.0.0 - */ - public Exception getException(); -} - diff --git a/src/main/java/org/torproject/descriptor/DescriptorParseException.java b/src/main/java/org/torproject/descriptor/DescriptorParseException.java index 5f46077..c9e7d93 100644 --- a/src/main/java/org/torproject/descriptor/DescriptorParseException.java +++ b/src/main/java/org/torproject/descriptor/DescriptorParseException.java @@ -9,9 +9,7 @@ package org.torproject.descriptor; * * @since 1.0.0 */ -@SuppressWarnings("deprecation") -public class DescriptorParseException - extends org.torproject.descriptor.impl.DescriptorParseException { +public class DescriptorParseException extends Exception { private static final long serialVersionUID = 100L; diff --git a/src/main/java/org/torproject/descriptor/DescriptorParser.java b/src/main/java/org/torproject/descriptor/DescriptorParser.java index 61f69f4..0082b5f 100644 --- a/src/main/java/org/torproject/descriptor/DescriptorParser.java +++ b/src/main/java/org/torproject/descriptor/DescriptorParser.java @@ -24,23 +24,6 @@ import java.util.List; public interface DescriptorParser { /** - * Fail descriptor parsing when encountering an unrecognized line. - * - * This option is not set by default, because the Tor specifications - * allow for new lines to be added that shall be ignored by older Tor - * versions. But some applications may want to handle unrecognized - * descriptor lines explicitly. - * - * @deprecated Removed in an attempt to simplify the interface. Applications - * that must fail descriptors with unrecognized lines can instead check - * whether {@link Descriptor#getUnrecognizedLines()} returns any lines. - * - * @since 1.0.0 - */ - public void setFailUnrecognizedDescriptorLines( - boolean failUnrecognizedDescriptorLines); - - /** * Parse descriptors in the given byte array, possibly parsing the * publication time from the file name, depending on the descriptor * type. diff --git a/src/main/java/org/torproject/descriptor/DescriptorReader.java b/src/main/java/org/torproject/descriptor/DescriptorReader.java index e15e5d9..4adff39 100644 --- a/src/main/java/org/torproject/descriptor/DescriptorReader.java +++ b/src/main/java/org/torproject/descriptor/DescriptorReader.java @@ -4,7 +4,6 @@ package org.torproject.descriptor; import java.io.File; -import java.util.Iterator; import java.util.SortedMap; /** @@ -38,46 +37,6 @@ import java.util.SortedMap; public interface DescriptorReader { /** - * Add a local directory to read descriptors from, which may contain - * descriptor files or tarballs containing descriptor files. - * - * @deprecated Replaced with a parameter in {@link #readDescriptors(File...)}, - * which ignores any directories added via this deprecated method. - * - * @since 1.0.0 - */ - public void addDirectory(File directory); - - /** - * Add a tarball to read descriptors from, which may be uncompressed, - * bz2-compressed, or xz-compressed. - * - * @deprecated Replaced with a parameter in {@link #readDescriptors(File...)}, - * which ignores any tarballs added via this deprecated method. - * - * @since 1.0.0 - */ - public void addTarball(File tarball); - - /** - * Exclude files that are listed in the given history file and that - * haven't changed since they have last been read. - * - * Add a new line for each descriptor that is read in this execution - * and remove lines for files that don't exist anymore. - * - * Lines in the history file contain the last modified time in - * milliseconds since the epoch and the absolute path of a file. - * - * @deprecated Replaced by {@link #setHistoryFile(File)} and - * {@link #saveHistoryFile(File)} which let the application explicitly - * tell us when it's done processing read descriptors. - * - * @since 1.0.0 - */ - public void setExcludeFiles(File historyFile); - - /** * Set a history file to load before reading descriptors and exclude * descriptor files that haven't changed since they have last been read. * @@ -136,35 +95,6 @@ public interface DescriptorReader { public SortedMap<String, Long> getParsedFiles(); /** - * Fail descriptor parsing when encountering an unrecognized line. - * - * This option is not set by default, because the Tor specifications - * allow for new lines to be added that shall be ignored by older Tor - * versions. But some applications may want to handle unrecognized - * descriptor lines explicitly. - * - * @deprecated Removed in an attempt to simplify the interface. Applications - * that must fail descriptors with unrecognized lines can instead check - * whether {@link Descriptor#getUnrecognizedLines()} returns any lines. - * - * @since 1.0.0 - */ - public void setFailUnrecognizedDescriptorLines(); - - /** - * Don't keep more than this number of parsed descriptor files in the - * queue. - * - * The default is 100, but if descriptor files contain hundreds or - * even thousands of descriptors, that default may be too high. - * - * @deprecated Replaced with {@link #setMaxDescriptorsInQueue(int)}. - * - * @since 1.0.0 - */ - public void setMaxDescriptorFilesInQueue(int max); - - /** * Don't keep more than this number of descriptors in the queue (default: * 100). * @@ -175,20 +105,6 @@ public interface DescriptorReader { public void setMaxDescriptorsInQueue(int maxDescriptorsInQueue); /** - * Read the previously configured descriptors and make them available - * via the returned blocking iterator. - * - * Whenever the reader runs out of descriptors and expects to provide - * more shortly after, it blocks the caller. This method can only be - * run once. - * - * @deprecated Replaced with {@link #readDescriptors(File...)}. - * - * @since 1.0.0 - */ - public Iterator<DescriptorFile> readDescriptors(); - - /** * Read descriptors from the given descriptor file(s) and return the parsed * descriptors. * diff --git a/src/main/java/org/torproject/descriptor/DescriptorRequest.java b/src/main/java/org/torproject/descriptor/DescriptorRequest.java deleted file mode 100644 index 9009ca8..0000000 --- a/src/main/java/org/torproject/descriptor/DescriptorRequest.java +++ /dev/null @@ -1,104 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor; - -import java.util.List; - -/** - * Container for descriptors downloaded from a directory authority or - * mirror. - * - * When the {@link DescriptorDownloader} downloads descriptors from - * directory authorities or mirrors it provides an iterator over these - * containers which in turn contain references to classes implementing the - * {@link Descriptor} interface. This container also stores potentially - * useful meta-data about the descriptor request. - * - * @deprecated Removed as descriptor container together with - * {@link DescriptorDownloader} in favor of the much more widely used - * {@link DescriptorCollector}. - * - * @since 1.0.0 - */ -public interface DescriptorRequest { - - /** - * Return the request URL that was used in this request. - * - * @since 1.0.0 - */ - public String getRequestUrl(); - - /** - * Return the nickname of the directory mirror or authority as - * previously configured. - * - * @since 1.0.0 - */ - public String getDirectoryNickname(); - - /** - * Return the first exception that was thrown when making this request - * or parsing the response, or null if no exception was thrown. - * - * @since 1.0.0 - */ - public Exception getException(); - - /** - * Return the response code that the directory mirror or authority - * returned. - * - * @since 1.0.0 - */ - public int getResponseCode(); - - /** - * Return the time in milliseconds since the epoch when this request - * was started. - * - * @since 1.0.0 - */ - public long getRequestStart(); - - /** - * Return the time in milliseconds since the epoch when this request - * ended. - * - * @since 1.0.0 - */ - public long getRequestEnd(); - - /** - * Return whether this request ended, because the connect timeout has - * expired. - * - * @since 1.0.0 - */ - public boolean connectTimeoutHasExpired(); - - /** - * Return whether this request ended, because the read timeout has - * expired. - * - * @since 1.0.0 - */ - public boolean readTimeoutHasExpired(); - - /** - * Return whether this request ended, because the global timeout for - * all requests has expired. - * - * @since 1.0.0 - */ - public boolean globalTimeoutHasExpired(); - - /** - * Return the descriptors contained in the reply. - * - * @since 1.0.0 - */ - public List<Descriptor> getDescriptors(); -} - diff --git a/src/main/java/org/torproject/descriptor/DescriptorSourceFactory.java b/src/main/java/org/torproject/descriptor/DescriptorSourceFactory.java index 7125ace..f2c1a43 100644 --- a/src/main/java/org/torproject/descriptor/DescriptorSourceFactory.java +++ b/src/main/java/org/torproject/descriptor/DescriptorSourceFactory.java @@ -11,10 +11,9 @@ import org.slf4j.LoggerFactory; * * Descriptor sources are the only producers of classes implementing * the {@link Descriptor} superinterface. There exist descriptor sources - * for obtaining remote descriptor data ({@link DescriptorDownloader} and - * {@link DescriptorCollector}) and descriptor sources for processing - * local descriptor data ({@link DescriptorReader} and - * {@link DescriptorParser}). + * for obtaining remote descriptor data ({@link DescriptorCollector}) and + * descriptor sources for processing local descriptor data + * ({@link DescriptorReader} and {@link DescriptorParser}). * * By default, this factory returns implementations from the library's * own impl package. This may be overridden by setting Java properties, @@ -23,7 +22,6 @@ import org.slf4j.LoggerFactory; * These properties can be used for setting the implementation: * <ul> * <li>{@code descriptor.collector}</li> - * <li>{@code descriptor.downloader}</li> * <li>{@code descriptor.parser}</li> * <li>{@code descriptor.reader}</li> * </ul> @@ -46,15 +44,6 @@ public final class DescriptorSourceFactory { DescriptorSourceFactory.class); /** - * Default implementation of the {@link DescriptorDownloader} - * descriptor source. - * - * @since 1.0.0 - */ - public static final String DOWNLOADER_DEFAULT = - "org.torproject.descriptor.impl.DescriptorDownloaderImpl"; - - /** * Default implementation of the {@link DescriptorParser} descriptor * source. * @@ -101,16 +90,6 @@ public final class DescriptorSourceFactory { /** * Property name for overriding the implementation of the - * {@link DescriptorDownloader} descriptor source, which is by default - * set to the class in {@link #DOWNLOADER_DEFAULT}. - * - * @since 1.0.0 - */ - public static final String DOWNLOADER_PROPERTY = - "descriptor.downloader"; - - /** - * Property name for overriding the implementation of the * {@link DescriptorCollector} descriptor source, which is by default * set to the class in {@link #COLLECTOR_DEFAULT}. * @@ -139,16 +118,6 @@ public final class DescriptorSourceFactory { } /** - * Create a new {@link DescriptorDownloader} by instantiating the class - * in {@link #DOWNLOADER_PROPERTY}. - * - * @since 1.0.0 - */ - public static final DescriptorDownloader createDescriptorDownloader() { - return (DescriptorDownloader) retrieve(DOWNLOADER_PROPERTY); - } - - /** * Create a new {@link DescriptorCollector} by instantiating the class * in {@link #COLLECTOR_PROPERTY}. * @@ -166,9 +135,6 @@ public final class DescriptorSourceFactory { case PARSER_PROPERTY: clazzName = System.getProperty(type, PARSER_DEFAULT); break; - case DOWNLOADER_PROPERTY: - clazzName = System.getProperty(type, DOWNLOADER_DEFAULT); - break; case READER_PROPERTY: clazzName = System.getProperty(type, READER_DEFAULT); break; diff --git a/src/main/java/org/torproject/descriptor/DirSourceEntry.java b/src/main/java/org/torproject/descriptor/DirSourceEntry.java index 9996ff2..69e8525 100644 --- a/src/main/java/org/torproject/descriptor/DirSourceEntry.java +++ b/src/main/java/org/torproject/descriptor/DirSourceEntry.java @@ -89,16 +89,6 @@ public interface DirSourceEntry { * Return the SHA-1 vote digest, encoded as 40 lower-case hexadecimal * characters. * - * @deprecated Renamed to {@link #getVoteDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getVoteDigest(); - - /** - * Return the SHA-1 vote digest, encoded as 40 lower-case hexadecimal - * characters. - * * @since 1.7.0 */ public String getVoteDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/DirectoryKeyCertificate.java b/src/main/java/org/torproject/descriptor/DirectoryKeyCertificate.java index f2d37ad..2819fc3 100644 --- a/src/main/java/org/torproject/descriptor/DirectoryKeyCertificate.java +++ b/src/main/java/org/torproject/descriptor/DirectoryKeyCertificate.java @@ -102,16 +102,6 @@ public interface DirectoryKeyCertificate extends Descriptor { * Return the SHA-1 certificate digest, encoded as 40 lower-case * hexadecimal characters. * - * @deprecated Renamed to {@link #getDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getCertificateDigest(); - - /** - * Return the SHA-1 certificate digest, encoded as 40 lower-case - * hexadecimal characters. - * * @since 1.7.0 */ public String getDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/DirectorySignature.java b/src/main/java/org/torproject/descriptor/DirectorySignature.java index bb3d5c4..9ce44df 100644 --- a/src/main/java/org/torproject/descriptor/DirectorySignature.java +++ b/src/main/java/org/torproject/descriptor/DirectorySignature.java @@ -37,17 +37,6 @@ public interface DirectorySignature { * in the version 3 directory protocol, encoded as 40 upper-case * hexadecimal characters. * - * @deprecated Renamed to {@link #getSigningKeyDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getSigningKeyDigest(); - - /** - * Return the SHA-1 digest of the authority's medium-term signing key - * in the version 3 directory protocol, encoded as 40 upper-case - * hexadecimal characters. - * * @since 1.7.0 */ public String getSigningKeyDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/ExitList.java b/src/main/java/org/torproject/descriptor/ExitList.java index 052362a..2623ee5 100644 --- a/src/main/java/org/torproject/descriptor/ExitList.java +++ b/src/main/java/org/torproject/descriptor/ExitList.java @@ -74,17 +74,6 @@ public interface ExitList extends Descriptor { /** * Return the unordered set of exit scan results. * - * @since 1.0.0 - * @deprecated The {@link ExitListEntry} type has been deprecated and - * superseded by {@link ExitList.Entry} which is returned by - * {@link #getEntries()}. - */ - @Deprecated - public Set<ExitListEntry> getExitListEntries(); - - /** - * Return the unordered set of exit scan results. - * * @since 1.1.0 */ public Set<ExitList.Entry> getEntries(); diff --git a/src/main/java/org/torproject/descriptor/ExitListEntry.java b/src/main/java/org/torproject/descriptor/ExitListEntry.java deleted file mode 100644 index 94d5970..0000000 --- a/src/main/java/org/torproject/descriptor/ExitListEntry.java +++ /dev/null @@ -1,55 +0,0 @@ -/* Copyright 2012--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor; - -/** - * Exit list entry containing results from a single exit scan. - * - * @since 1.0.0 - * @deprecated Superseded by {@link ExitList.Entry}. - */ -@Deprecated -public interface ExitListEntry extends ExitList.Entry { - - /** - * Return the scanned relay's fingerprint, which is a SHA-1 digest of - * the relays's public identity key, encoded as 40 upper-case - * hexadecimal characters. - * - * @since 1.0.0 - */ - public String getFingerprint(); - - /** - * Return the time in milliseconds since the epoch when the scanned - * relay's last known descriptor was published. - * - * @since 1.0.0 - */ - public long getPublishedMillis(); - - /** - * Return the time in milliseconds since the epoch when the network - * status that this scan was based on was published. - * - * @since 1.0.0 - */ - public long getLastStatusMillis(); - - /** - * Return the IPv4 address in dotted-quad format that was determined in - * the scan. - * - * @since 1.0.0 - */ - public String getExitAddress(); - - /** - * Return the scan time in milliseconds since the epoch. - * - * @since 1.0.0 - */ - public long getScanMillis(); -} - diff --git a/src/main/java/org/torproject/descriptor/ExtraInfoDescriptor.java b/src/main/java/org/torproject/descriptor/ExtraInfoDescriptor.java index 04a46d4..333c853 100644 --- a/src/main/java/org/torproject/descriptor/ExtraInfoDescriptor.java +++ b/src/main/java/org/torproject/descriptor/ExtraInfoDescriptor.java @@ -48,18 +48,6 @@ public interface ExtraInfoDescriptor extends Descriptor { * characters, that is used to reference this descriptor from a server * descriptor. * - * @deprecated Renamed to {@link #getDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getExtraInfoDigest(); - - /** - * Return the SHA-1 descriptor digest, encoded as 40 lower-case (relay - * descriptors) or upper-case (bridge descriptors) hexadecimal - * characters, that is used to reference this descriptor from a server - * descriptor. - * * @since 1.7.0 */ public String getDigestSha1Hex(); @@ -69,17 +57,6 @@ public interface ExtraInfoDescriptor extends Descriptor { * characters without padding characters, that may be used to reference * this descriptor from a server descriptor. * - * @deprecated Renamed to {@link #getDigestSha256Base64()}. - * - * @since 1.1.0 - */ - public String getExtraInfoDigestSha256(); - - /** - * Return the SHA-256 descriptor digest, encoded as 43 base64 - * characters without padding characters, that may be used to reference - * this descriptor from a server descriptor. - * * @since 1.7.0 */ public String getDigestSha256Base64(); @@ -135,18 +112,6 @@ public interface ExtraInfoDescriptor extends Descriptor { * upper-case hexadecimal characters, or null if no GeoIP database * digest is included. * - * @deprecated Renamed to {@link #getGeoipDbDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getGeoipDbDigest(); - - /** - * Return a SHA-1 digest of the GeoIP database file used by this server - * to resolve client IP addresses to country codes, encoded as 40 - * upper-case hexadecimal characters, or null if no GeoIP database - * digest is included. - * * @since 1.7.0 */ public String getGeoipDbDigestSha1Hex(); @@ -157,18 +122,6 @@ public interface ExtraInfoDescriptor extends Descriptor { * upper-case hexadecimal characters, or null if no GeoIPv6 database * digest is included. * - * @deprecated Renamed to {@link #getGeoip6DbDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getGeoip6DbDigest(); - - /** - * Return a SHA-1 digest of the GeoIPv6 database file used by this - * server to resolve client IP addresses to country codes, encoded as 40 - * upper-case hexadecimal characters, or null if no GeoIPv6 database - * digest is included. - * * @since 1.7.0 */ public String getGeoip6DbDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/Microdescriptor.java b/src/main/java/org/torproject/descriptor/Microdescriptor.java index b49b8bb..ef69417 100644 --- a/src/main/java/org/torproject/descriptor/Microdescriptor.java +++ b/src/main/java/org/torproject/descriptor/Microdescriptor.java @@ -28,17 +28,6 @@ public interface Microdescriptor extends Descriptor { * characters without padding characters, that is used to reference this * descriptor from a vote or microdescriptor consensus. * - * @deprecated Renamed to {@link #getDigestSha256Base64()}. - * - * @since 1.0.0 - */ - public String getMicrodescriptorDigest(); - - /** - * Return the SHA-256 descriptor digest, encoded as 43 base64 - * characters without padding characters, that is used to reference this - * descriptor from a vote or microdescriptor consensus. - * * @since 1.7.0 */ public String getDigestSha256Base64(); diff --git a/src/main/java/org/torproject/descriptor/NetworkStatusEntry.java b/src/main/java/org/torproject/descriptor/NetworkStatusEntry.java index 9c5dae5..bbc0173 100644 --- a/src/main/java/org/torproject/descriptor/NetworkStatusEntry.java +++ b/src/main/java/org/torproject/descriptor/NetworkStatusEntry.java @@ -93,17 +93,6 @@ public interface NetworkStatusEntry { * base64 characters without padding characters, if the containing network * status is a vote or microdesc consensus, or null otherwise. * - * @deprecated Renamed to {@link #getMicrodescriptorDigestsSha256Base64()}. - * - * @since 1.0.0 - */ - public Set<String> getMicrodescriptorDigests(); - - /** - * Return the (possibly empty) set of microdescriptor digests, encoded as 43 - * base64 characters without padding characters, if the containing network - * status is a vote or microdesc consensus, or null otherwise. - * * @since 1.7.0 */ public Set<String> getMicrodescriptorDigestsSha256Base64(); diff --git a/src/main/java/org/torproject/descriptor/RelayDirectory.java b/src/main/java/org/torproject/descriptor/RelayDirectory.java index dd696cf..f05f65d 100644 --- a/src/main/java/org/torproject/descriptor/RelayDirectory.java +++ b/src/main/java/org/torproject/descriptor/RelayDirectory.java @@ -97,17 +97,6 @@ public interface RelayDirectory extends Descriptor { * hexadecimal characters, that the directory authority used to sign the * directory. * - * @deprecated Renamed to {@link #getDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getDirectoryDigest(); - - /** - * Return the SHA-1 directory digest, encoded as 40 lower-case - * hexadecimal characters, that the directory authority used to sign the - * directory. - * * @since 1.7.0 */ public String getDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/RelayNetworkStatus.java b/src/main/java/org/torproject/descriptor/RelayNetworkStatus.java index b14d0c3..1197c2e 100644 --- a/src/main/java/org/torproject/descriptor/RelayNetworkStatus.java +++ b/src/main/java/org/torproject/descriptor/RelayNetworkStatus.java @@ -169,17 +169,6 @@ public interface RelayNetworkStatus extends Descriptor { * characters, that the directory authority used to sign the network * status. * - * @deprecated Renamed to {@link #getDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getStatusDigest(); - - /** - * Return the SHA-1 status digest, encoded as 40 lower-case hexadecimal - * characters, that the directory authority used to sign the network - * status. - * * @since 1.7.0 */ public String getDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/RelayNetworkStatusConsensus.java b/src/main/java/org/torproject/descriptor/RelayNetworkStatusConsensus.java index fe66007..2b433dc 100644 --- a/src/main/java/org/torproject/descriptor/RelayNetworkStatusConsensus.java +++ b/src/main/java/org/torproject/descriptor/RelayNetworkStatusConsensus.java @@ -252,19 +252,6 @@ public interface RelayNetworkStatusConsensus extends Descriptor { public NetworkStatusEntry getStatusEntry(String fingerprint); /** - * Return directory signatures of this consensus, with map keys being - * SHA-1 digests of the authorities' identity keys in the version 3 - * directory protocol, encoded as 40 upper-case hexadecimal characters. - * - * @deprecated Replaced by {@link #getSignatures()} which permits an - * arbitrary number of signatures made by an authority using the - * same identity key digest and different algorithms. - * - * @since 1.0.0 - */ - public SortedMap<String, DirectorySignature> getDirectorySignatures(); - - /** * Return the list of signatures contained in this consensus. * * @since 1.3.0 @@ -286,17 +273,6 @@ public interface RelayNetworkStatusConsensus extends Descriptor { * hexadecimal characters that directory authorities use to sign the * consensus. * - * @deprecated Renamed to {@link #getDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getConsensusDigest(); - - /** - * Return the SHA-1 digest of this consensus, encoded as 40 lower-case - * hexadecimal characters that directory authorities use to sign the - * consensus. - * * @since 1.7.0 */ public String getDigestSha1Hex(); diff --git a/src/main/java/org/torproject/descriptor/RelayNetworkStatusVote.java b/src/main/java/org/torproject/descriptor/RelayNetworkStatusVote.java index 5d9d039..1e4b0a5 100644 --- a/src/main/java/org/torproject/descriptor/RelayNetworkStatusVote.java +++ b/src/main/java/org/torproject/descriptor/RelayNetworkStatusVote.java @@ -408,21 +408,6 @@ public interface RelayNetworkStatusVote extends Descriptor { public String getDirSigningKey(); /** - * Return the SHA-1 digest of the authority's signing key, encoded as - * 40 upper-case hexadecimal characters, or null if this digest cannot - * be obtained from the directory signature. - * - * @deprecated Removed in order to be more explicit that authorities may - * use different digest algorithms than "sha1"; see - * {@link #getSignatures()} and - * {@link DirectorySignature#getSigningKeyDigestSha1Hex()} for - * alternatives. - * - * @since 1.0.0 - */ - public String getSigningKeyDigest(); - - /** * Return the signature of the authority's identity key made using the * authority's signing key, or null if the vote does not contain such a * signature. @@ -470,20 +455,6 @@ public interface RelayNetworkStatusVote extends Descriptor { public NetworkStatusEntry getStatusEntry(String fingerprint); /** - * Return the directory signature of this vote, with the single map key - * being the SHA-1 digest of the authority's identity key in the version - * 3 directory protocol, encoded as 40 upper-case hexadecimal - * characters. - * - * @deprecated Replaced by {@link #getSignatures()} which permits an - * arbitrary number of signatures made by the authority using the - * same identity key digest and different algorithms. - * - * @since 1.0.0 - */ - public SortedMap<String, DirectorySignature> getDirectorySignatures(); - - /** * Return a list of signatures contained in this vote, which is * typically a single signature made by the authority but which may also * be more than one signature made with different keys or algorithms. diff --git a/src/main/java/org/torproject/descriptor/ServerDescriptor.java b/src/main/java/org/torproject/descriptor/ServerDescriptor.java index 3ac0cc9..0aeb543 100644 --- a/src/main/java/org/torproject/descriptor/ServerDescriptor.java +++ b/src/main/java/org/torproject/descriptor/ServerDescriptor.java @@ -51,18 +51,6 @@ public interface ServerDescriptor extends Descriptor { * characters, that is used to reference this descriptor from a network * status descriptor. * - * @deprecated Renamed to {@link #getDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getServerDescriptorDigest(); - - /** - * Return the SHA-1 descriptor digest, encoded as 40 lower-case (relay - * descriptors) or upper-case (bridge descriptors) hexadecimal - * characters, that is used to reference this descriptor from a network - * status descriptor. - * * @since 1.7.0 */ public String getDigestSha1Hex(); @@ -72,17 +60,6 @@ public interface ServerDescriptor extends Descriptor { * characters without padding characters, that may be used to reference * this server descriptor from a network status descriptor. * - * @deprecated Renamed to {@link #getDigestSha256Base64()}. - * - * @since 1.1.0 - */ - public String getServerDescriptorDigestSha256(); - - /** - * Return the SHA-256 descriptor digest, encoded as 43 base64 - * characters without padding characters, that may be used to reference - * this server descriptor from a network status descriptor. - * * @since 1.7.0 */ public String getDigestSha256Base64(); @@ -113,7 +90,7 @@ public interface ServerDescriptor extends Descriptor { /** * Return the TCP port where this server accepts SOCKS connections, - * which is deprecated and should always be 0. + * which is deprecated in the Tor Protocol and should always be 0. * * @since 1.0.0 */ @@ -322,17 +299,6 @@ public interface ServerDescriptor extends Descriptor { * encoded as 40 upper-case hexadecimal characters, or null if the * server did not upload a corresponding extra-info descriptor. * - * @deprecated Renamed to {@link #getExtraInfoDigestSha1Hex()}. - * - * @since 1.0.0 - */ - public String getExtraInfoDigest(); - - /** - * Return the SHA-1 digest of the server's extra-info descriptor, - * encoded as 40 upper-case hexadecimal characters, or null if the - * server did not upload a corresponding extra-info descriptor. - * * @since 1.7.0 */ public String getExtraInfoDigestSha1Hex(); @@ -343,18 +309,6 @@ public interface ServerDescriptor extends Descriptor { * if the server either did not upload a corresponding extra-info * descriptor or did not refer to it using a SHA-256 digest. * - * @deprecated Renamed to {@link #getExtraInfoDigestSha256Base64()}. - * - * @since 1.1.0 - */ - public String getExtraInfoDigestSha256(); - - /** - * Return the SHA-256 digest of the server's extra-info descriptor, - * encoded as 43 base64 characters without padding characters, or null - * if the server either did not upload a corresponding extra-info - * descriptor or did not refer to it using a SHA-256 digest. - * * @since 1.7.0 */ public String getExtraInfoDigestSha256Base64(); diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java index b2b3e99..d8941a6 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java @@ -12,10 +12,9 @@ public class BridgeExtraInfoDescriptorImpl extends ExtraInfoDescriptorImpl implements BridgeExtraInfoDescriptor { protected BridgeExtraInfoDescriptorImpl(byte[] descriptorBytes, - int[] offsetAndLimit, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, offsetAndLimit, descriptorFile, - failUnrecognizedDescriptorLines); + int[] offsetAndLimit, File descriptorFile) + throws DescriptorParseException { + super(descriptorBytes, offsetAndLimit, descriptorFile); } } diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java index a54c91c..01cd221 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java @@ -20,10 +20,10 @@ public class BridgeNetworkStatusImpl extends NetworkStatusImpl implements BridgeNetworkStatus { protected BridgeNetworkStatusImpl(byte[] rawDescriptorBytes, - int[] offsetAndLength, File descriptorFile, String fileName, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + int[] offsetAndLength, File descriptorFile, String fileName) + throws DescriptorParseException { super(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, false, false); + false, false); this.setPublishedMillisFromFileName(fileName); } @@ -86,14 +86,8 @@ public class BridgeNetworkStatusImpl extends NetworkStatusImpl this.parseFlagThresholdsLine(line, parts); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in bridge network status."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } } } diff --git a/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java index b2015ff..195690b 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java @@ -18,8 +18,7 @@ public class BridgePoolAssignmentImpl extends DescriptorImpl protected BridgePoolAssignmentImpl(byte[] rawDescriptorBytes, int[] offsetAndlength, File descriptorFile, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(rawDescriptorBytes, offsetAndlength, descriptorFile, - failUnrecognizedDescriptorLines, false); + super(rawDescriptorBytes, offsetAndlength, descriptorFile, false); this.parseDescriptorBytes(); this.checkExactlyOnceKeys(EnumSet.of(Key.BRIDGE_POOL_ASSIGNMENT)); this.checkFirstKey(Key.BRIDGE_POOL_ASSIGNMENT); diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java index 30da220..8125102 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java @@ -12,10 +12,9 @@ public class BridgeServerDescriptorImpl extends ServerDescriptorImpl implements BridgeServerDescriptor { protected BridgeServerDescriptorImpl(byte[] rawDescriptorBytes, - int[] offsetAndLength, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines); + int[] offsetAndLength, File descriptorFile) + throws DescriptorParseException { + super(rawDescriptorBytes, offsetAndLength, descriptorFile); } } diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorCollectorImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorCollectorImpl.java deleted file mode 100644 index 214227f..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorCollectorImpl.java +++ /dev/null @@ -1,264 +0,0 @@ -/* Copyright 2015--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import org.torproject.descriptor.DescriptorCollector; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.BufferedInputStream; -import java.io.BufferedOutputStream; -import java.io.BufferedReader; -import java.io.File; -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.InputStream; -import java.io.InputStreamReader; -import java.net.HttpURLConnection; -import java.net.URL; -import java.text.DateFormat; -import java.text.ParseException; -import java.util.Arrays; -import java.util.Map; -import java.util.Scanner; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.Stack; -import java.util.TreeMap; -import java.util.TreeSet; -import java.util.regex.Matcher; -import java.util.regex.Pattern; -import java.util.zip.GZIPInputStream; - -/** - * Download files from a CollecTor instance based on the remote instance's - * directory listings. - * - * @deprecated Replaced by {@link DescriptorIndexCollector} which uses the - * remote instance's index.json file as a more robust alternative to parsing - * the remote instance's directory listings. - */ -public class DescriptorCollectorImpl implements DescriptorCollector { - - private static Logger log = LoggerFactory - .getLogger(DescriptorCollectorImpl.class); - - @Override - public void collectDescriptors(String collecTorBaseUrl, - String[] remoteDirectories, long minLastModified, - File localDirectory, boolean deleteExtraneousLocalFiles) { - collecTorBaseUrl = collecTorBaseUrl.endsWith("/") - ? collecTorBaseUrl.substring(0, collecTorBaseUrl.length() - 1) - : collecTorBaseUrl; - if (minLastModified < 0) { - throw new IllegalArgumentException("A negative minimum " - + "last-modified time is not permitted."); - } - if (localDirectory.exists() && !localDirectory.isDirectory()) { - throw new IllegalArgumentException("Local directory already exists " - + "and is not a directory."); - } - SortedMap<String, Long> localFiles = - this.statLocalDirectory(localDirectory); - SortedMap<String, String> fetchedDirectoryListings = - this.fetchRemoteDirectories(collecTorBaseUrl, remoteDirectories); - SortedSet<String> parsedDirectories = new TreeSet<>(); - SortedMap<String, Long> remoteFiles = new TreeMap<>(); - for (Map.Entry<String, String> e - : fetchedDirectoryListings.entrySet()) { - String remoteDirectory = e.getKey(); - String directoryListing = e.getValue(); - SortedMap<String, Long> parsedRemoteFiles = - this.parseDirectoryListing(remoteDirectory, directoryListing); - if (parsedRemoteFiles == null) { - continue; - } - parsedDirectories.add(remoteDirectory); - remoteFiles.putAll(parsedRemoteFiles); - } - this.fetchRemoteFiles(collecTorBaseUrl, remoteFiles, minLastModified, - localDirectory, localFiles); - if (deleteExtraneousLocalFiles) { - this.deleteExtraneousLocalFiles(parsedDirectories, remoteFiles, - localDirectory, localFiles); - } - } - - SortedMap<String, Long> statLocalDirectory( - File localDirectory) { - SortedMap<String, Long> localFiles = new TreeMap<>(); - if (!localDirectory.exists()) { - return localFiles; - } - Stack<File> files = new Stack<>(); - files.add(localDirectory); - while (!files.isEmpty()) { - File file = files.pop(); - if (file.isDirectory()) { - files.addAll(Arrays.asList(file.listFiles())); - } else { - String localPath = file.getPath().substring( - localDirectory.getPath().length()); - localFiles.put(localPath, file.lastModified()); - } - } - return localFiles; - } - - SortedMap<String, String> fetchRemoteDirectories( - String collecTorBaseUrl, String[] remoteDirectories) { - SortedMap<String, String> fetchedDirectoryListings = new TreeMap<>(); - for (String remoteDirectory : remoteDirectories) { - String remoteDirectoryWithSlashAtBeginAndEnd = - (remoteDirectory.startsWith("/") ? "" : "/") + remoteDirectory - + (remoteDirectory.endsWith("/") ? "" : "/"); - String directoryUrl = collecTorBaseUrl - + remoteDirectoryWithSlashAtBeginAndEnd; - String directoryListing = this.fetchRemoteDirectory(directoryUrl); - if (directoryListing.length() > 0) { - fetchedDirectoryListings.put( - remoteDirectoryWithSlashAtBeginAndEnd, directoryListing); - } - } - return fetchedDirectoryListings; - } - - String fetchRemoteDirectory(String urlString) { - StringBuilder sb = new StringBuilder(); - HttpURLConnection huc = null; - try { - URL url = new URL(urlString); - huc = (HttpURLConnection) url.openConnection(); - huc.setRequestMethod("GET"); - huc.connect(); - int responseCode = huc.getResponseCode(); - if (responseCode == 200) { - BufferedReader br = new BufferedReader(new InputStreamReader( - huc.getInputStream())); - String line; - while ((line = br.readLine()) != null) { - sb.append(line).append("\n"); - } - br.close(); - } - } catch (IOException e) { - log.error("Cannot fetch remote directory.", e); - if (huc != null) { - huc.disconnect(); - } - return ""; - } - return sb.toString(); - } - - final Pattern directoryListingLinePattern = - Pattern.compile(".* href=\"([^\"/]+)\"" /* filename */ - + ".*>(\\d{4}-\\w{2}-\\d{2} \\d{2}:\\d{2})\\s*<.*"); /* dateTime */ - - SortedMap<String, Long> parseDirectoryListing( - String remoteDirectory, String directoryListing) { - SortedMap<String, Long> remoteFiles = new TreeMap<>(); - DateFormat dateTimeFormat = ParseHelper.getDateFormat( - "yyyy-MM-dd HH:mm"); - try { - Scanner scanner = new Scanner(directoryListing); - scanner.useDelimiter("\n"); - while (scanner.hasNext()) { - String line = scanner.next(); - Matcher matcher = directoryListingLinePattern.matcher(line); - if (matcher.matches()) { - String filename = matcher.group(1); - long lastModifiedMillis = dateTimeFormat.parse( - matcher.group(2)).getTime(); - remoteFiles.put(remoteDirectory + filename, lastModifiedMillis); - } - } - scanner.close(); - } catch (ParseException e) { - log.error("Cannot parse directory listing: " + directoryListing, e); - return null; - } - return remoteFiles; - } - - void fetchRemoteFiles(String collecTorBaseUrl, - SortedMap<String, Long> remoteFiles, long minLastModified, - File localDirectory, SortedMap<String, Long> localFiles) { - for (Map.Entry<String, Long> e : remoteFiles.entrySet()) { - String filename = e.getKey(); - long lastModifiedMillis = e.getValue(); - if (lastModifiedMillis < minLastModified - || (localFiles.containsKey(filename) - && localFiles.get(filename) >= lastModifiedMillis)) { - continue; - } - String url = collecTorBaseUrl + filename; - File destinationFile = new File(localDirectory.getPath() - + filename); - this.fetchRemoteFile(url, destinationFile, lastModifiedMillis); - } - } - - void fetchRemoteFile(String urlString, File destinationFile, - long lastModifiedMillis) { - HttpURLConnection huc = null; - try { - File destinationDirectory = destinationFile.getParentFile(); - destinationDirectory.mkdirs(); - File tempDestinationFile = new File(destinationDirectory, "." - + destinationFile.getName()); - BufferedOutputStream bos = new BufferedOutputStream( - new FileOutputStream(tempDestinationFile)); - URL url = new URL(urlString); - huc = (HttpURLConnection) url.openConnection(); - huc.setRequestMethod("GET"); - if (!urlString.endsWith(".xz")) { - huc.addRequestProperty("Accept-Encoding", "gzip"); - } - huc.connect(); - int responseCode = huc.getResponseCode(); - if (responseCode == 200) { - InputStream is; - if (huc.getContentEncoding() != null - && huc.getContentEncoding().equalsIgnoreCase("gzip")) { - is = new GZIPInputStream(huc.getInputStream()); - } else { - is = huc.getInputStream(); - } - BufferedInputStream bis = new BufferedInputStream(is); - int len; - byte[] data = new byte[8192]; - while ((len = bis.read(data, 0, 8192)) >= 0) { - bos.write(data, 0, len); - } - bis.close(); - bos.close(); - tempDestinationFile.renameTo(destinationFile); - destinationFile.setLastModified(lastModifiedMillis); - } - } catch (IOException e) { - log.error("Cannot fetch remote file.", e); - if (huc != null) { - huc.disconnect(); - } - } - } - - void deleteExtraneousLocalFiles( - SortedSet<String> parsedDirectories, - SortedMap<String, Long> remoteFiles, File localDirectory, - SortedMap<String, Long> localFiles) { - for (String localPath : localFiles.keySet()) { - for (String remoteDirectory : parsedDirectories) { - if (localPath.startsWith(remoteDirectory)) { - if (!remoteFiles.containsKey(localPath)) { - new File(localDirectory.getPath() + localPath).delete(); - } - } - } - } - } -} - diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorDownloaderImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorDownloaderImpl.java deleted file mode 100644 index 807abac..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorDownloaderImpl.java +++ /dev/null @@ -1,294 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import org.torproject.descriptor.DescriptorDownloader; -import org.torproject.descriptor.DescriptorRequest; - -import java.util.HashSet; -import java.util.Iterator; -import java.util.Set; -import java.util.SortedMap; -import java.util.TreeMap; - -public class DescriptorDownloaderImpl - implements DescriptorDownloader { - - private boolean hasStartedDownloading = false; - - private SortedMap<String, DirectoryDownloader> directoryAuthorities = - new TreeMap<>(); - - @Override - public void addDirectoryAuthority(String nickname, String ip, - int dirPort) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.checkDirectoryParameters(nickname, ip, dirPort); - DirectoryDownloader directoryAuthority = new DirectoryDownloader( - nickname, ip, dirPort); - this.directoryAuthorities.put(nickname, directoryAuthority); - } - - private SortedMap<String, DirectoryDownloader> directoryMirrors = - new TreeMap<>(); - - @Override - public void addDirectoryMirror(String nickname, String ip, - int dirPort) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.checkDirectoryParameters(nickname, ip, dirPort); - DirectoryDownloader directoryMirror = new DirectoryDownloader( - nickname, ip, dirPort); - this.directoryMirrors.put(nickname, directoryMirror); - /* TODO Implement prioritizing mirrors for non-vote downloads. */ - throw new UnsupportedOperationException("Prioritizing directory " - + "mirrors over directory authorities is not implemented yet. " - + "Until it is, configuring directory mirrors is misleading and " - + "therefore not supported."); - } - - private void checkDirectoryParameters(String nickname, String ip, - int dirPort) { - if (nickname == null || nickname.length() < 1) { - throw new IllegalArgumentException("'" + nickname + "' is not a " - + "valid nickname."); - } - if (ip == null || ip.length() < 7 || ip.split("\\.").length != 4) { - throw new IllegalArgumentException("'" + ip + "' is not a valid IP " - + "address."); - } - if (dirPort < 1 || dirPort > 65535) { - throw new IllegalArgumentException(String.valueOf(dirPort) + " is " - + "not a valid DirPort."); - } - /* TODO Relax the requirement for directory nicknames to be unique. - * In theory, we can identify them by ip+port. */ - if (this.directoryAuthorities.containsKey(nickname) - || this.directoryMirrors.containsKey(nickname)) { - throw new IllegalArgumentException("Directory nicknames must be " - + "unique."); - } - } - - private boolean downloadConsensus = false; - - @Override - public void setIncludeCurrentConsensus() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.downloadConsensus = true; - } - - private boolean downloadConsensusFromAllAuthorities = false; - - @Override - public void setIncludeCurrentConsensusFromAllDirectoryAuthorities() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.downloadConsensusFromAllAuthorities = true; - } - - private boolean includeCurrentReferencedVotes = false; - - @Override - public void setIncludeCurrentReferencedVotes() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.includeCurrentReferencedVotes = true; - } - - private Set<String> downloadVotes = new HashSet<>(); - - @Override - public void setIncludeCurrentVote(String fingerprint) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.checkVoteFingerprint(fingerprint); - this.downloadVotes.add(fingerprint); - } - - @Override - public void setIncludeCurrentVotes(Set<String> fingerprints) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - if (fingerprints == null) { - throw new IllegalArgumentException("Set of fingerprints must not " - + "be null."); - } - for (String fingerprint : fingerprints) { - this.checkVoteFingerprint(fingerprint); - } - for (String fingerprint : fingerprints) { - this.setIncludeCurrentVote(fingerprint); - } - } - - private void checkVoteFingerprint(String fingerprint) { - if (fingerprint == null || fingerprint.length() != 40) { - throw new IllegalArgumentException("'" + fingerprint + "' is not a " - + "valid fingerprint."); - } - } - - @Override - public void setIncludeReferencedServerDescriptors() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - /* TODO Implement me. */ - throw new UnsupportedOperationException("Downloading server " - + "descriptors is not implemented yet."); - } - - @Override - public void setExcludeServerDescriptor(String identifier) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - /* TODO Implement me. */ - throw new UnsupportedOperationException("Downloading server " - + "descriptors is not implemented yet."); - } - - @Override - public void setExcludeServerDescriptors(Set<String> identifier) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - /* TODO Implement me. */ - throw new UnsupportedOperationException("Downloading server " - + "descriptors is not implemented yet."); - } - - @Override - public void setIncludeReferencedExtraInfoDescriptors() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - /* TODO Implement me. */ - throw new UnsupportedOperationException("Downloading extra-info " - + "descriptors is not implemented yet."); - } - - @Override - public void setExcludeExtraInfoDescriptor(String identifier) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - /* TODO Implement me. */ - throw new UnsupportedOperationException("Downloading extra-info " - + "descriptors is not implemented yet."); - } - - @Override - public void setExcludeExtraInfoDescriptors(Set<String> identifiers) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - /* TODO Implement me. */ - throw new UnsupportedOperationException("Downloading extra-info " - + "descriptors is not implemented yet."); - } - - private long readTimeoutMillis = 60L * 1000L; - - @Override - public void setReadTimeout(long readTimeoutMillis) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - if (readTimeoutMillis < 0L) { - throw new IllegalArgumentException("Read timeout value " - + String.valueOf(readTimeoutMillis) + " may not be " - + "negative."); - } - this.readTimeoutMillis = readTimeoutMillis; - } - - private long connectTimeoutMillis = 60L * 1000L; - - @Override - public void setConnectTimeout(long connectTimeoutMillis) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - if (connectTimeoutMillis < 0L) { - throw new IllegalArgumentException("Connect timeout value " - + String.valueOf(connectTimeoutMillis) + " may not be " - + "negative."); - } - this.connectTimeoutMillis = connectTimeoutMillis; - } - - private long globalTimeoutMillis = 60L * 60L * 1000L; - - @Override - public void setGlobalTimeout(long globalTimeoutMillis) { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - if (globalTimeoutMillis < 0L) { - throw new IllegalArgumentException("Global timeout value " - + String.valueOf(globalTimeoutMillis) + " may not be " - + "negative."); - } - this.globalTimeoutMillis = globalTimeoutMillis; - } - - private boolean failUnrecognizedDescriptorLines = false; - - @Override - public void setFailUnrecognizedDescriptorLines() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to download."); - } - this.failUnrecognizedDescriptorLines = true; - } - - @Override - public Iterator<DescriptorRequest> downloadDescriptors() { - if (this.hasStartedDownloading) { - throw new IllegalStateException("Initiating downloads is only " - + "permitted once."); - } - this.hasStartedDownloading = true; - DownloadCoordinatorImpl downloadCoordinator = - new DownloadCoordinatorImpl(this.directoryAuthorities, - this.directoryMirrors, this.downloadConsensus, - this.downloadConsensusFromAllAuthorities, this.downloadVotes, - this.includeCurrentReferencedVotes, this.connectTimeoutMillis, - this.readTimeoutMillis, this.globalTimeoutMillis, - this.failUnrecognizedDescriptorLines); - Iterator<DescriptorRequest> descriptorQueue = downloadCoordinator - .getDescriptorQueue(); - return descriptorQueue; - } -} - diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorFileImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorFileImpl.java deleted file mode 100644 index 561e0b8..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorFileImpl.java +++ /dev/null @@ -1,93 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorFile; - -import java.io.File; -import java.util.ArrayList; -import java.util.List; - -public class DescriptorFileImpl implements DescriptorFile { - - private File directory; - - protected void setDirectory(File directory) { - this.directory = directory; - } - - @Override - public File getDirectory() { - return this.directory; - } - - private File tarball; - - protected void setTarball(File tarball) { - this.tarball = tarball; - } - - @Override - public File getTarball() { - return this.tarball; - } - - private File file; - - protected void setFile(File file) { - this.file = file; - } - - @Override - public File getFile() { - return this.file; - } - - private String fileName; - - protected void setFileName(String fileName) { - this.fileName = fileName; - } - - @Override - public String getFileName() { - return this.fileName; - } - - private long lastModified; - - protected void setLastModified(long lastModified) { - this.lastModified = lastModified; - } - - @Override - public long getLastModified() { - return this.lastModified; - } - - private List<Descriptor> descriptors; - - protected void setDescriptors(List<Descriptor> descriptors) { - this.descriptors = descriptors; - } - - @Override - public List<Descriptor> getDescriptors() { - return this.descriptors == null ? new ArrayList<Descriptor>() - : new ArrayList<>(this.descriptors); - } - - private Exception exception; - - protected void setException(Exception exception) { - this.exception = exception; - } - - @Override - public Exception getException() { - return this.exception; - } -} - diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java index acbee50..eec89a4 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java @@ -184,8 +184,6 @@ public abstract class DescriptorImpl implements Descriptor { return splitParts; } - protected boolean failUnrecognizedDescriptorLines = false; - protected List<String> unrecognizedLines; @Override @@ -195,7 +193,7 @@ public abstract class DescriptorImpl implements Descriptor { } protected DescriptorImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines) { + File descriptorFile) { int offset = offsetAndLength[0]; int length = offsetAndLength[1]; if (offset < 0 || offset + length > rawDescriptorBytes.length @@ -208,15 +206,12 @@ public abstract class DescriptorImpl implements Descriptor { this.offset = offset; this.length = length; this.descriptorFile = descriptorFile; - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; } protected DescriptorImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines, - boolean blankLinesAllowed) throws DescriptorParseException { - this(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines); + File descriptorFile, boolean blankLinesAllowed) + throws DescriptorParseException { + this(rawDescriptorBytes, offsetAndLength, descriptorFile); this.cutOffAnnotations(); this.countKeys(rawDescriptorBytes, blankLinesAllowed); } diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorParseException.java b/src/main/java/org/torproject/descriptor/impl/DescriptorParseException.java deleted file mode 100644 index 8583ef2..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorParseException.java +++ /dev/null @@ -1,23 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -/** - * @deprecated Replaced by - * org.torproject.descriptor.DescriptorParseException - */ -@Deprecated public class DescriptorParseException extends Exception { - - private static final long serialVersionUID = 100L; - - protected DescriptorParseException(String message) { - super(message); - } - - protected DescriptorParseException(String message, Exception ex) { - super(message, ex); - } - -} - diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java index a0be85c..dbdfca9 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java @@ -19,15 +19,6 @@ import java.util.List; public class DescriptorParserImpl implements DescriptorParser { - private boolean failUnrecognizedDescriptorLines; - - @Override - public void setFailUnrecognizedDescriptorLines( - boolean failUnrecognizedDescriptorLines) { - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; - } - @Override public List<Descriptor> parseDescriptors( byte[] rawDescriptorBytes, String fileName) @@ -46,8 +37,7 @@ public class DescriptorParserImpl implements DescriptorParser { * below and ran into a parse issue. */ List<Descriptor> parsedDescriptors = new ArrayList<>(); parsedDescriptors.add(new UnparseableDescriptorImpl(rawDescriptorBytes, - new int[] { 0, rawDescriptorBytes.length }, descriptorFile, - failUnrecognizedDescriptorLines, e)); + new int[] { 0, rawDescriptorBytes.length }, descriptorFile, e)); return parsedDescriptors; } } @@ -71,7 +61,7 @@ public class DescriptorParserImpl implements DescriptorParser { NL + Key.VOTE_STATUS.keyword + SP + "consensus" + NL))) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.NETWORK_STATUS_VERSION, RelayNetworkStatusConsensusImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type network-status-vote-3 1.") || ((firstLines.startsWith( Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL) @@ -81,59 +71,59 @@ public class DescriptorParserImpl implements DescriptorParser { NL + Key.VOTE_STATUS.keyword + SP + "vote" + NL))) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.NETWORK_STATUS_VERSION, RelayNetworkStatusVoteImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type bridge-network-status 1.") || firstLines.startsWith(Key.R.keyword + SP)) { List<Descriptor> parsedDescriptors = new ArrayList<>(); parsedDescriptors.add(new BridgeNetworkStatusImpl( rawDescriptorBytes, new int[] { 0, rawDescriptorBytes.length }, - descriptorFile, fileName, this.failUnrecognizedDescriptorLines)); + descriptorFile, fileName)); return parsedDescriptors; } else if (firstLines.startsWith("@type bridge-server-descriptor 1.")) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.ROUTER, BridgeServerDescriptorImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type server-descriptor 1.") || firstLines.startsWith(Key.ROUTER.keyword + SP) || firstLines.contains(NL + Key.ROUTER.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.ROUTER, RelayServerDescriptorImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type bridge-extra-info 1.")) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.EXTRA_INFO, BridgeExtraInfoDescriptorImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type extra-info 1.") || firstLines.startsWith(Key.EXTRA_INFO.keyword + SP) || firstLines.contains(NL + Key.EXTRA_INFO.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.EXTRA_INFO, RelayExtraInfoDescriptorImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type microdescriptor 1.") || firstLines.startsWith(Key.ONION_KEY.keyword + NL) || firstLines.contains(NL + Key.ONION_KEY.keyword + NL)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.ONION_KEY, MicrodescriptorImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type bridge-pool-assignment 1.") || firstLines.startsWith(Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP) || firstLines.contains(NL + Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.BRIDGE_POOL_ASSIGNMENT, BridgePoolAssignmentImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type dir-key-certificate-3 1.") || firstLines.startsWith(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP) || firstLines.contains( NL + Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.DIR_KEY_CERTIFICATE_VERSION, DirectoryKeyCertificateImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type tordnsel 1.") || firstLines.startsWith("ExitNode" + SP) || firstLines.contains(NL + "ExitNode" + SP)) { List<Descriptor> parsedDescriptors = new ArrayList<>(); parsedDescriptors.add(new ExitListImpl(rawDescriptorBytes, descriptorFile, - fileName, this.failUnrecognizedDescriptorLines)); + fileName)); return parsedDescriptors; } else if (firstLines.startsWith("@type network-status-2 1.") || firstLines.startsWith( @@ -142,16 +132,16 @@ public class DescriptorParserImpl implements DescriptorParser { NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.NETWORK_STATUS_VERSION, RelayNetworkStatusImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type directory 1.") || firstLines.startsWith(Key.SIGNED_DIRECTORY.keyword + NL) || firstLines.contains(NL + Key.SIGNED_DIRECTORY.keyword + NL)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, Key.SIGNED_DIRECTORY, RelayDirectoryImpl.class, - this.failUnrecognizedDescriptorLines, includeUnparseableDescriptors); + includeUnparseableDescriptors); } else if (firstLines.startsWith("@type torperf 1.")) { return TorperfResultImpl.parseTorperfResults(rawDescriptorBytes, - descriptorFile, this.failUnrecognizedDescriptorLines); + descriptorFile); } else { throw new DescriptorParseException("Could not detect descriptor " + "type in descriptor starting with '" + firstLines + "'."); @@ -161,7 +151,6 @@ public class DescriptorParserImpl implements DescriptorParser { private List<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, File descriptorFile, Key key, Class<? extends DescriptorImpl> descriptorClass, - boolean failUnrecognizedDescriptorLines, boolean includeUnparseableDescriptors) throws DescriptorParseException { List<Descriptor> parsedDescriptors = new ArrayList<>(); Constructor<? extends DescriptorImpl> constructor; @@ -213,13 +202,11 @@ public class DescriptorParserImpl implements DescriptorParser { endDescriptor - startAnnotations }; try { parsedDescriptors.add(this.parseDescriptor(rawDescriptorBytes, - offsetAndLength, descriptorFile, constructor, - failUnrecognizedDescriptorLines)); + offsetAndLength, descriptorFile, constructor)); } catch (DescriptorParseException e) { if (includeUnparseableDescriptors) { parsedDescriptors.add(new UnparseableDescriptorImpl( - rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, e)); + rawDescriptorBytes, offsetAndLength, descriptorFile, e)); } else { throw e; } @@ -231,11 +218,11 @@ public class DescriptorParserImpl implements DescriptorParser { Descriptor parseDescriptor(byte[] rawDescriptorBytes, int[] offsetAndLength, File descriptorFile, - Constructor<? extends DescriptorImpl> constructor, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + Constructor<? extends DescriptorImpl> constructor) + throws DescriptorParseException { try { - return constructor.newInstance(rawDescriptorBytes, - offsetAndLength, descriptorFile, failUnrecognizedDescriptorLines); + return constructor.newInstance(rawDescriptorBytes, offsetAndLength, + descriptorFile); } catch (InvocationTargetException e) { if (null != e.getCause() && e.getCause() instanceof DescriptorParseException) { diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java index 54e26fc..e2bf987 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java @@ -4,7 +4,6 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorFile; import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.DescriptorParser; import org.torproject.descriptor.DescriptorReader; @@ -27,7 +26,6 @@ import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.util.ArrayList; import java.util.Arrays; -import java.util.Iterator; import java.util.List; import java.util.Map; import java.util.SortedMap; @@ -42,40 +40,13 @@ public class DescriptorReaderImpl implements DescriptorReader { private List<File> directories = new ArrayList<>(); - @Override - public void addDirectory(File directory) { - if (this.hasStartedReading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to read."); - } - this.directories.add(directory); - } - private List<File> tarballs = new ArrayList<>(); - @Override - public void addTarball(File tarball) { - if (this.hasStartedReading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to read."); - } - this.tarballs.add(tarball); - } - private File autoSaveHistoryFile; private File manualSaveHistoryFile; @Override - public void setExcludeFiles(File historyFile) { - if (this.hasStartedReading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to read."); - } - this.autoSaveHistoryFile = historyFile; - } - - @Override public void setHistoryFile(File historyFile) { if (this.hasStartedReading) { throw new IllegalStateException("Reconfiguration is not permitted " @@ -113,28 +84,8 @@ public class DescriptorReaderImpl implements DescriptorReader { return new TreeMap<>(this.reader.parsedFilesAfter); } - private boolean failUnrecognizedDescriptorLines = false; - - @Override - public void setFailUnrecognizedDescriptorLines() { - if (this.hasStartedReading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to read."); - } - this.failUnrecognizedDescriptorLines = true; - } - private Integer maxDescriptorFilesInQueue = null; - @Override - public void setMaxDescriptorFilesInQueue(int max) { - if (this.hasStartedReading) { - throw new IllegalStateException("Reconfiguration is not permitted " - + "after starting to read."); - } - this.maxDescriptorFilesInQueue = max; - } - private int maxDescriptorsInQueue = 100; @Override @@ -149,28 +100,6 @@ public class DescriptorReaderImpl implements DescriptorReader { private DescriptorReaderRunnable reader; @Override - public Iterator<DescriptorFile> readDescriptors() { - if (this.hasStartedReading) { - throw new IllegalStateException("Initiating reading is only " - + "permitted once."); - } - this.hasStartedReading = true; - BlockingIteratorImpl<DescriptorFile> descriptorQueue = - this.maxDescriptorFilesInQueue == null - ? new BlockingIteratorImpl<DescriptorFile>() - : new BlockingIteratorImpl<DescriptorFile>( - this.maxDescriptorFilesInQueue); - this.reader = new DescriptorReaderRunnable(this.directories, - this.tarballs, descriptorQueue, this.autoSaveHistoryFile, - this.manualSaveHistoryFile, this.excludedFiles, - this.failUnrecognizedDescriptorLines); - Thread readerThread = new Thread(this.reader); - readerThread.setDaemon(true); - readerThread.start(); - return descriptorQueue; - } - - @Override public Iterable<Descriptor> readDescriptors(File... descriptorFiles) { if (this.hasStartedReading) { throw new IllegalStateException("Initiating reading is only " @@ -181,8 +110,7 @@ public class DescriptorReaderImpl implements DescriptorReader { new BlockingIteratorImpl<>(this.maxDescriptorsInQueue); this.reader = new DescriptorReaderRunnable( descriptorFiles, descriptorQueue, this.autoSaveHistoryFile, - this.manualSaveHistoryFile, this.excludedFiles, - this.failUnrecognizedDescriptorLines); + this.manualSaveHistoryFile, this.excludedFiles); Thread readerThread = new Thread(this.reader); readerThread.setDaemon(true); readerThread.start(); @@ -208,8 +136,6 @@ public class DescriptorReaderImpl implements DescriptorReader { private BlockingIteratorImpl<Descriptor> descriptorQueue; - private BlockingIteratorImpl<DescriptorFile> descriptorFileQueue; - private File autoSaveHistoryFile; private File manualSaveHistoryFile; @@ -226,31 +152,10 @@ public class DescriptorReaderImpl implements DescriptorReader { private boolean hasFinishedReading = false; - private DescriptorReaderRunnable(List<File> directories, - List<File> tarballs, - BlockingIteratorImpl<DescriptorFile> descriptorFileQueue, - File autoSaveHistoryFile, File manualSaveHistoryFile, - SortedMap<String, Long> excludedFiles, - boolean failUnrecognizedDescriptorLines) { - this.directories = directories; - this.tarballs = tarballs; - this.descriptorFileQueue = descriptorFileQueue; - this.autoSaveHistoryFile = autoSaveHistoryFile; - this.manualSaveHistoryFile = manualSaveHistoryFile; - if (excludedFiles != null) { - this.excludedFilesBefore = excludedFiles; - } - this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; - this.descriptorParser = new DescriptorParserImpl(); - this.descriptorParser.setFailUnrecognizedDescriptorLines( - failUnrecognizedDescriptorLines); - } - private DescriptorReaderRunnable(File[] descriptorFiles, BlockingIteratorImpl<Descriptor> descriptorQueue, File autoSaveHistoryFile, File manualSaveHistoryFile, - SortedMap<String, Long> excludedFiles, - boolean failUnrecognizedDescriptorLines) { + SortedMap<String, Long> excludedFiles) { this.descriptorFiles = descriptorFiles; this.descriptorQueue = descriptorQueue; this.autoSaveHistoryFile = autoSaveHistoryFile; @@ -260,8 +165,6 @@ public class DescriptorReaderImpl implements DescriptorReader { } this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; this.descriptorParser = new DescriptorParserImpl(); - this.descriptorParser.setFailUnrecognizedDescriptorLines( - failUnrecognizedDescriptorLines); } public void run() { @@ -276,9 +179,6 @@ public class DescriptorReaderImpl implements DescriptorReader { log.error("Bug: uncaught exception or error while " + "reading descriptors: " + t.getMessage(), t); } finally { - if (null != this.descriptorFileQueue) { - this.descriptorFileQueue.setOutOfDescriptors(); - } if (null != this.descriptorQueue) { this.descriptorQueue.setOutOfDescriptors(); } @@ -453,20 +353,6 @@ public class DescriptorReaderImpl implements DescriptorReader { continue; } this.parsedFilesAfter.put(absolutePath, lastModifiedMillis); - DescriptorFileImpl descriptorFile = new DescriptorFileImpl(); - try { - descriptorFile.setDirectory(directory); - descriptorFile.setFile(file); - descriptorFile.setFileName(file.getAbsolutePath()); - descriptorFile.setLastModified(lastModifiedMillis); - descriptorFile.setDescriptors(this.readFile(file)); - } catch (DescriptorParseException e) { - descriptorFile.setException(e); - } catch (IOException e) { - descriptorFile.setException(e); - abortReading = true; - } - this.descriptorFileQueue.add(descriptorFile); } } } @@ -513,33 +399,6 @@ public class DescriptorReaderImpl implements DescriptorReader { if (tae.isDirectory()) { continue; } - DescriptorFileImpl descriptorFile = - new DescriptorFileImpl(); - descriptorFile.setTarball(tarball); - descriptorFile.setFileName(tae.getName()); - descriptorFile.setLastModified(tae.getLastModifiedDate() - .getTime()); - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - int len; - byte[] data = new byte[1024]; - while ((len = bis.read(data, 0, 1024)) >= 0) { - baos.write(data, 0, len); - } - byte[] rawDescriptorBytes = baos.toByteArray(); - if (rawDescriptorBytes.length < 1) { - continue; - } - try { - String fileName = tae.getName().substring( - tae.getName().lastIndexOf("/") + 1); - List<Descriptor> parsedDescriptors = - this.descriptorParser.parseDescriptors( - rawDescriptorBytes, fileName); - descriptorFile.setDescriptors(parsedDescriptors); - } catch (DescriptorParseException e) { - descriptorFile.setException(e); - } - this.descriptorFileQueue.add(descriptorFile); } } } catch (IOException e) { diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorRequestImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorRequestImpl.java deleted file mode 100644 index 7362935..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorRequestImpl.java +++ /dev/null @@ -1,137 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorRequest; - -import java.util.List; - -public class DescriptorRequestImpl implements DescriptorRequest { - - private String requestedResource; - - protected void setRequestedResource(String requestedResource) { - this.requestedResource = requestedResource; - } - - protected String getRequestedResource() { - return this.requestedResource; - } - - private String descriptorType; - - protected void setDescriptorType(String descriptorType) { - this.descriptorType = descriptorType; - } - - protected String getDescriptorType() { - return this.descriptorType; - } - - private byte[] responseBytes; - - protected byte[] getResponseBytes() { - return this.responseBytes; - } - - protected void setResponseBytes(byte[] responseBytes) { - this.responseBytes = responseBytes; - } - - private String requestUrl; - - @Override - public String getRequestUrl() { - return this.requestUrl; - } - - private String directoryNickname; - - protected void setDirectoryNickname(String directoryNickname) { - this.directoryNickname = directoryNickname; - } - - @Override - public String getDirectoryNickname() { - return this.directoryNickname; - } - - private int responseCode; - - protected void setResponseCode(int responseCode) { - this.responseCode = responseCode; - } - - @Override - public int getResponseCode() { - return this.responseCode; - } - - private long requestStart; - - protected void setRequestStart(long requestStart) { - this.requestStart = requestStart; - } - - @Override - public long getRequestStart() { - return this.requestStart; - } - - private long requestEnd; - - protected void setRequestEnd(long requestEnd) { - this.requestEnd = requestEnd; - } - - @Override - public long getRequestEnd() { - return this.requestEnd; - } - - private boolean connectTimeoutHasExpired; - - @Override - public boolean connectTimeoutHasExpired() { - return this.connectTimeoutHasExpired; - } - - private boolean readTimeoutHasExpired; - - @Override - public boolean readTimeoutHasExpired() { - return this.readTimeoutHasExpired; - } - - private boolean globalTimeoutHasExpired; - - @Override - public boolean globalTimeoutHasExpired() { - return this.globalTimeoutHasExpired; - } - - private List<Descriptor> descriptors; - - protected void setDescriptors(List<Descriptor> descriptors) { - this.descriptors = descriptors; - } - - @Override - public List<Descriptor> getDescriptors() { - return this.descriptors; - } - - private Exception exception; - - protected void setException(Exception exception) { - this.exception = exception; - } - - @Override - public Exception getException() { - return this.exception; - } -} - diff --git a/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java b/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java index 770ca61..630b886 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java @@ -31,8 +31,6 @@ public class DirSourceEntryImpl implements DirSourceEntry { return this.parent.getRawDescriptorBytes(this.offset, this.length); } - private boolean failUnrecognizedDescriptorLines; - private List<String> unrecognizedLines; protected List<String> getAndClearUnrecognizedLines() { @@ -41,14 +39,11 @@ public class DirSourceEntryImpl implements DirSourceEntry { return lines; } - protected DirSourceEntryImpl(DescriptorImpl parent, int offset, int length, - boolean failUnrecognizedDescriptorLines) + protected DirSourceEntryImpl(DescriptorImpl parent, int offset, int length) throws DescriptorParseException { this.parent = parent; this.offset = offset; this.length = length; - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; this.parseDirSourceEntryBytes(); this.checkAndClearKeys(); } @@ -117,15 +112,10 @@ public class DirSourceEntryImpl implements DirSourceEntry { break; default: if (!skipCrypto) { - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in dir-source entry."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -237,11 +227,6 @@ public class DirSourceEntryImpl implements DirSourceEntry { private String voteDigest; @Override - public String getVoteDigest() { - return this.getVoteDigestSha1Hex(); - } - - @Override public String getVoteDigestSha1Hex() { return this.voteDigest; } diff --git a/src/main/java/org/torproject/descriptor/impl/DirectoryDownloader.java b/src/main/java/org/torproject/descriptor/impl/DirectoryDownloader.java deleted file mode 100644 index 7b3f977..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DirectoryDownloader.java +++ /dev/null @@ -1,111 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import org.torproject.descriptor.DescriptorParser; -import org.torproject.descriptor.DescriptorSourceFactory; - -import java.io.BufferedInputStream; -import java.io.ByteArrayOutputStream; -import java.net.HttpURLConnection; -import java.net.URL; -import java.util.zip.InflaterInputStream; - -/* Download descriptors from one directory authority or mirror. First, - * ask the coordinator thread to create a request, run it, and deliver - * the response. Repeat until the coordinator thread says there are no - * further requests to make. */ -public class DirectoryDownloader implements Runnable { - - private String nickname; - - private String ipPort; - - private DescriptorParser descriptorParser; - - protected DirectoryDownloader(String nickname, String ip, int dirPort) { - this.nickname = nickname; - this.ipPort = ip + ":" + String.valueOf(dirPort); - this.descriptorParser = - DescriptorSourceFactory.createDescriptorParser(); - } - - private DownloadCoordinator downloadCoordinator; - - protected void setDownloadCoordinator( - DownloadCoordinator downloadCoordinator) { - this.downloadCoordinator = downloadCoordinator; - } - - private long connectTimeout; - - protected void setConnectTimeout(long connectTimeout) { - this.connectTimeout = connectTimeout; - } - - private long readTimeout; - - protected void setReadTimeout(long readTimeout) { - this.readTimeout = readTimeout; - } - - protected void setFailUnrecognizedDescriptorLines( - boolean failUnrecognizedDescriptorLines) { - this.descriptorParser.setFailUnrecognizedDescriptorLines( - failUnrecognizedDescriptorLines); - } - - @Override - public void run() { - boolean keepRunning = true; - do { - DescriptorRequestImpl request = - this.downloadCoordinator.createRequest(this.nickname); - if (request != null) { - String urlString = "http://" + this.ipPort - + request.getRequestedResource(); - request.setRequestStart(System.currentTimeMillis()); - HttpURLConnection huc = null; - try { - URL url = new URL(urlString); - huc = (HttpURLConnection) url.openConnection(); - huc.setConnectTimeout((int) this.connectTimeout); - huc.setReadTimeout((int) this.readTimeout); - huc.setRequestMethod("GET"); - huc.connect(); - int responseCode = huc.getResponseCode(); - request.setResponseCode(responseCode); - if (responseCode == 200) { - BufferedInputStream in = new BufferedInputStream( - new InflaterInputStream(huc.getInputStream())); - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - int len; - byte[] data = new byte[8192]; - while ((len = in.read(data, 0, 8192)) >= 0) { - baos.write(data, 0, len); - } - in.close(); - byte[] responseBytes = baos.toByteArray(); - request.setResponseBytes(responseBytes); - request.setRequestEnd(System.currentTimeMillis()); - request.setDescriptors(this.descriptorParser.parseDescriptors( - responseBytes, null)); - } - } catch (Exception e) { - request.setException(e); - if (huc != null) { - huc.disconnect(); - } - /* Stop downloading from this directory if there are any - * problems, e.g., refused connections. */ - keepRunning = false; - } - this.downloadCoordinator.deliverResponse(request); - } else { - keepRunning = false; - } - } while (keepRunning); - } -} - diff --git a/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java b/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java index 6beb9b5..59357d5 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java @@ -18,8 +18,7 @@ public class DirectoryKeyCertificateImpl extends DescriptorImpl protected DirectoryKeyCertificateImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, File descriptorFile, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, false); + super(rawDescriptorBytes, offsetAndLength, descriptorFile, false); this.parseDescriptorBytes(); this.calculateDigestSha1Hex(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP, NL + Key.DIR_KEY_CERTIFICATION.keyword + NL); @@ -107,15 +106,10 @@ public class DirectoryKeyCertificateImpl extends DescriptorImpl if (crypto != null) { crypto.append(line).append(NL); } else { - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in directory key certificate."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -261,9 +255,5 @@ public class DirectoryKeyCertificateImpl extends DescriptorImpl return this.dirKeyCertification; } - @Override - public String getCertificateDigest() { - return this.getDigestSha1Hex(); - } } diff --git a/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java b/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java index be0abf8..b638002 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java @@ -21,8 +21,6 @@ public class DirectorySignatureImpl implements DirectorySignature { private int length; - private boolean failUnrecognizedDescriptorLines; - private List<String> unrecognizedLines; protected List<String> getAndClearUnrecognizedLines() { @@ -32,13 +30,10 @@ public class DirectorySignatureImpl implements DirectorySignature { } protected DirectorySignatureImpl(DescriptorImpl parent, int offset, - int length, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { + int length) throws DescriptorParseException { this.parent = parent; this.offset = offset; this.length = length; - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; this.parseDirectorySignatureBytes(); } @@ -84,15 +79,10 @@ public class DirectorySignatureImpl implements DirectorySignature { if (crypto != null) { crypto.append(line).append(NL); } else { - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in dir-source entry."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -117,11 +107,6 @@ public class DirectorySignatureImpl implements DirectorySignature { private String signingKeyDigest; @Override - public String getSigningKeyDigest() { - return this.getSigningKeyDigestSha1Hex(); - } - - @Override public String getSigningKeyDigestSha1Hex() { return this.signingKeyDigest; } diff --git a/src/main/java/org/torproject/descriptor/impl/DownloadCoordinator.java b/src/main/java/org/torproject/descriptor/impl/DownloadCoordinator.java deleted file mode 100644 index 0d78f06..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DownloadCoordinator.java +++ /dev/null @@ -1,11 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -public interface DownloadCoordinator { - - public DescriptorRequestImpl createRequest(String nickname); - - public void deliverResponse(DescriptorRequestImpl request); -} diff --git a/src/main/java/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java b/src/main/java/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java deleted file mode 100644 index c79a7cc..0000000 --- a/src/main/java/org/torproject/descriptor/impl/DownloadCoordinatorImpl.java +++ /dev/null @@ -1,323 +0,0 @@ -/* Copyright 2011--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorRequest; -import org.torproject.descriptor.DirSourceEntry; -import org.torproject.descriptor.RelayNetworkStatusConsensus; - -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.HashMap; -import java.util.HashSet; -import java.util.Iterator; -import java.util.Map; -import java.util.Set; -import java.util.SortedMap; -import java.util.SortedSet; -import java.util.TreeSet; - -/* TODO This whole download logic is a mess and needs a cleanup. */ -public class DownloadCoordinatorImpl implements DownloadCoordinator { - - private static Logger log = LoggerFactory - .getLogger(DownloadCoordinatorImpl.class); - - private BlockingIteratorImpl<DescriptorRequest> descriptorQueue = - new BlockingIteratorImpl<>(); - - protected Iterator<DescriptorRequest> getDescriptorQueue() { - return this.descriptorQueue; - } - - private SortedSet<String> runningDirectories; - - private SortedMap<String, DirectoryDownloader> directoryAuthorities; - - private SortedMap<String, DirectoryDownloader> directoryMirrors; - - private boolean downloadConsensusFromAllAuthorities; - - private boolean includeCurrentReferencedVotes; - - private long connectTimeoutMillis; - - private long readTimeoutMillis; - - private long globalTimeoutMillis; - - private boolean failUnrecognizedDescriptorLines; - - protected DownloadCoordinatorImpl( - SortedMap<String, DirectoryDownloader> directoryAuthorities, - SortedMap<String, DirectoryDownloader> directoryMirrors, - boolean downloadConsensus, - boolean downloadConsensusFromAllAuthorities, - Set<String> downloadVotes, boolean includeCurrentReferencedVotes, - long connectTimeoutMillis, long readTimeoutMillis, - long globalTimeoutMillis, boolean failUnrecognizedDescriptorLines) { - this.directoryAuthorities = directoryAuthorities; - this.directoryMirrors = directoryMirrors; - this.runningDirectories = new TreeSet<>(); - this.runningDirectories.addAll(directoryAuthorities.keySet()); - this.runningDirectories.addAll(directoryMirrors.keySet()); - this.missingConsensus = downloadConsensus; - this.downloadConsensusFromAllAuthorities = - downloadConsensusFromAllAuthorities; - this.missingVotes = downloadVotes; - this.includeCurrentReferencedVotes = includeCurrentReferencedVotes; - this.connectTimeoutMillis = connectTimeoutMillis; - this.readTimeoutMillis = readTimeoutMillis; - this.globalTimeoutMillis = globalTimeoutMillis; - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; - if (this.directoryMirrors.isEmpty() - && this.directoryAuthorities.isEmpty()) { - this.descriptorQueue.setOutOfDescriptors(); - log.warn("There were no directories configured. Nothing to download."); - } else { - GlobalTimer globalTimer = new GlobalTimer(this.globalTimeoutMillis, - this); - this.globalTimerThread = new Thread(globalTimer); - this.globalTimerThread.start(); - for (DirectoryDownloader directoryMirror - : this.directoryMirrors.values()) { - directoryMirror.setDownloadCoordinator(this); - directoryMirror.setConnectTimeout(this.connectTimeoutMillis); - directoryMirror.setReadTimeout(this.readTimeoutMillis); - directoryMirror.setFailUnrecognizedDescriptorLines( - this.failUnrecognizedDescriptorLines); - new Thread(directoryMirror).start(); - } - for (DirectoryDownloader directoryAuthority - : this.directoryAuthorities.values()) { - directoryAuthority.setDownloadCoordinator(this); - directoryAuthority.setConnectTimeout(this.connectTimeoutMillis); - directoryAuthority.setReadTimeout(this.readTimeoutMillis); - directoryAuthority.setFailUnrecognizedDescriptorLines( - this.failUnrecognizedDescriptorLines); - new Thread(directoryAuthority).start(); - } - } - } - - /* Interrupt all downloads if the total download time exceeds a given - * time. */ - private Thread globalTimerThread; - - private static class GlobalTimer implements Runnable { - - private long timeoutMillis; - - private DownloadCoordinatorImpl downloadCoordinator; - - private GlobalTimer(long timeoutMillis, - DownloadCoordinatorImpl downloadCoordinator) { - this.timeoutMillis = timeoutMillis; - this.downloadCoordinator = downloadCoordinator; - } - - public void run() { - long started = System.currentTimeMillis(); - long sleep; - while ((sleep = started + this.timeoutMillis - - System.currentTimeMillis()) > 0L) { - try { - Thread.sleep(sleep); - } catch (InterruptedException e) { - return; - } - } - this.downloadCoordinator.interruptAllDownloads(); - } - } - - /* Are we missing the consensus, and should the next directory that - * hasn't tried downloading it before attempt to download it? */ - private boolean missingConsensus = false; - - /* Which directories are currently attempting to download the - * consensus? */ - private Set<String> requestingConsensuses = new HashSet<>(); - - /* Which directories have attempted to download the consensus so far, - * including those directories that are currently attempting it? */ - private Set<String> requestedConsensuses = new HashSet<>(); - - /* Which votes are we currently missing? */ - private Set<String> missingVotes = new HashSet<>(); - - /* Which vote (map value) is a given directory (map key) currently - * attempting to download? */ - private Map<String, String> requestingVotes = new HashMap<>(); - - /* Which votes (map value) has a given directory (map key) attempted or - * is currently attempting to download? */ - private Map<String, Set<String>> requestedVotes = new HashMap<>(); - - private boolean hasFinishedDownloading = false; - - /* Look up what request a directory should make next. If there is - * nothing to do right now, but maybe later, block the caller. If - * we're done downloading, return null to notify the caller. */ - @Override - public synchronized DescriptorRequestImpl createRequest( - String nickname) { - while (!this.hasFinishedDownloading) { - DescriptorRequestImpl request = new DescriptorRequestImpl(); - request.setDirectoryNickname(nickname); - if ((this.missingConsensus - || (this.downloadConsensusFromAllAuthorities - && this.directoryAuthorities.containsKey(nickname))) - && !this.requestedConsensuses.contains(nickname)) { - if (!this.downloadConsensusFromAllAuthorities) { - this.missingConsensus = false; - } - this.requestingConsensuses.add(nickname); - this.requestedConsensuses.add(nickname); - request.setRequestedResource( - "/tor/status-vote/current/consensus.z"); - request.setDescriptorType("consensus"); - return request; - } - if (!this.missingVotes.isEmpty() - && this.directoryAuthorities.containsKey(nickname)) { - String requestingVote = null; - for (String missingVote : this.missingVotes) { - if (!this.requestedVotes.containsKey(nickname) - || !this.requestedVotes.get(nickname).contains( - missingVote)) { - requestingVote = missingVote; - } - } - if (requestingVote != null) { - this.requestingVotes.put(nickname, requestingVote); - if (!this.requestedVotes.containsKey(nickname)) { - this.requestedVotes.put(nickname, new HashSet<String>()); - } - this.requestedVotes.get(nickname).add(requestingVote); - this.missingVotes.remove(requestingVote); - request.setRequestedResource("/tor/status-vote/current/" - + requestingVote + ".z"); - request.setDescriptorType("vote"); - return request; - } - } - /* TODO Add server descriptors and extra-info descriptors later. */ - try { - this.wait(); - } catch (InterruptedException e) { - /* TODO What shall we do? */ - } - } - return null; - } - - /* Deliver a response which may either contain one or more descriptors - * or a failure response code. Update the lists of missing descriptors, - * decide if there are more descriptors to download, and wake up any - * waiting downloader threads. */ - @Override - public synchronized void deliverResponse( - DescriptorRequestImpl response) { - String nickname = response.getDirectoryNickname(); - if (response.getException() != null) { - this.runningDirectories.remove(nickname); - } - switch (response.getDescriptorType()) { - case "consensus": - this.requestingConsensuses.remove(nickname); - if (response.getResponseCode() == 200 - && response.getDescriptors() != null) { - if (this.includeCurrentReferencedVotes) { - /* TODO Only add votes if the consensus is not older than one - * hour. Or does that make no sense? */ - for (Descriptor parsedDescriptor - : response.getDescriptors()) { - if (!(parsedDescriptor - instanceof RelayNetworkStatusConsensus)) { - continue; - } - RelayNetworkStatusConsensus parsedConsensus = - (RelayNetworkStatusConsensus) parsedDescriptor; - for (DirSourceEntry dirSource - : parsedConsensus.getDirSourceEntries().values()) { - String identity = dirSource.getIdentity(); - if (!this.missingVotes.contains(identity)) { - boolean alreadyRequested = false; - for (Set<String> requestedBefore - : this.requestedVotes.values()) { - if (requestedBefore.contains(identity)) { - alreadyRequested = true; - break; - } - } - if (!alreadyRequested) { - this.missingVotes.add(identity); - } - } - } - } - /* TODO Later, add referenced server descriptors. */ - } - } else { - this.missingConsensus = true; - } - break; - case "vote": - String requestedVote = requestingVotes.remove(nickname); - if (response.getResponseCode() != 200) { - this.missingVotes.add(requestedVote); - } - break; - default: - /* nothing to be done */ - } - if (response.getRequestEnd() != 0L) { - this.descriptorQueue.add(response); - } - boolean doneDownloading = true; - if ((this.missingConsensus - || this.downloadConsensusFromAllAuthorities) - && (!this.requestedConsensuses.containsAll( - this.runningDirectories) - || !this.requestingConsensuses.isEmpty())) { - doneDownloading = false; - } - if (!this.requestingVotes.isEmpty()) { - doneDownloading = false; - } else if (!this.missingVotes.isEmpty()) { - if (!this.requestedVotes.keySet().containsAll( - this.runningDirectories)) { - doneDownloading = false; - } else { - for (String missingVote : this.missingVotes) { - for (String runningDirectory : this.runningDirectories) { - Set<String> reqVotes = this.requestedVotes.get( - runningDirectory); - if (!reqVotes.contains(missingVote)) { - doneDownloading = false; - } - } - } - } - } - if (doneDownloading) { - this.hasFinishedDownloading = true; - this.globalTimerThread.interrupt(); - this.descriptorQueue.setOutOfDescriptors(); - } - /* Wake up all waiting downloader threads. Maybe they can now - * download something, or they'll realize we're done downloading. */ - this.notifyAll(); - } - - private synchronized void interruptAllDownloads() { - this.hasFinishedDownloading = true; - this.notifyAll(); - } -} - diff --git a/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java b/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java index c0d8b52..06b6a96 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java @@ -5,7 +5,6 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.ExitList; -import org.torproject.descriptor.ExitListEntry; import java.util.ArrayList; import java.util.HashMap; @@ -15,12 +14,10 @@ import java.util.Scanner; import java.util.SortedSet; import java.util.TreeSet; -public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry { +public class ExitListEntryImpl implements ExitList.Entry { private String exitListEntryString; - private boolean failUnrecognizedDescriptorLines; - private List<String> unrecognizedLines; protected List<String> getAndClearUnrecognizedLines() { @@ -29,37 +26,9 @@ public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry { return lines; } - @Deprecated - private ExitListEntryImpl(String fingerprint, long publishedMillis, - long lastStatusMillis, String exitAddress, long scanMillis) { - this.fingerprint = fingerprint; - this.publishedMillis = publishedMillis; - this.lastStatusMillis = lastStatusMillis; - this.exitAddresses.put(exitAddress, scanMillis); - } - - @Deprecated - List<ExitListEntry> oldEntries() { - List<ExitListEntry> result = new ArrayList<>(); - if (this.exitAddresses.size() > 1) { - for (Map.Entry<String, Long> entry - : this.exitAddresses.entrySet()) { - result.add(new ExitListEntryImpl(this.fingerprint, - this.publishedMillis, this.lastStatusMillis, entry.getKey(), - entry.getValue())); - } - } else { - result.add(this); - } - return result; - } - - protected ExitListEntryImpl(String exitListEntryString, - boolean failUnrecognizedDescriptorLines) + protected ExitListEntryImpl(String exitListEntryString) throws DescriptorParseException { this.exitListEntryString = exitListEntryString; - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; this.initializeKeywords(); this.parseExitListEntry(); this.checkAndClearKeywords(); @@ -114,15 +83,10 @@ public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry { this.parseExitAddressLine(line, parts); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in exit list entry."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -192,34 +156,11 @@ public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry { return this.lastStatusMillis; } - private String exitAddress; - - @Override - public String getExitAddress() { - if (null == exitAddress) { - Map.Entry<String, Long> randomEntry = - this.exitAddresses.entrySet().iterator().next(); - this.exitAddress = randomEntry.getKey(); - this.scanMillis = randomEntry.getValue(); - } - return this.exitAddress; - } - private Map<String, Long> exitAddresses = new HashMap<>(); @Override public Map<String, Long> getExitAddresses() { return new HashMap<>(this.exitAddresses); } - - private long scanMillis; - - @Override - public long getScanMillis() { - if (null == exitAddress) { - getExitAddress(); - } - return scanMillis; - } } diff --git a/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java b/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java index d16856a..d67ccd9 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java @@ -5,7 +5,6 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.ExitList; -import org.torproject.descriptor.ExitListEntry; import java.io.File; import java.text.ParseException; @@ -20,10 +19,9 @@ import java.util.TimeZone; public class ExitListImpl extends DescriptorImpl implements ExitList { protected ExitListImpl(byte[] rawDescriptorBytes, File descriptorfile, - String fileName, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { + String fileName) throws DescriptorParseException { super(rawDescriptorBytes, new int[] { 0, rawDescriptorBytes.length }, - descriptorfile, failUnrecognizedDescriptorLines, false); + descriptorfile, false); this.splitAndParseExitListEntries(); this.setPublishedMillisFromFileName(fileName); } @@ -88,14 +86,8 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { sb.append(line).append(ExitList.EOL); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in exit list."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } } } @@ -106,9 +98,8 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { protected void parseExitListEntry(String exitListEntryString) throws DescriptorParseException { ExitListEntryImpl exitListEntry = new ExitListEntryImpl( - exitListEntryString, this.failUnrecognizedDescriptorLines); + exitListEntryString); this.exitListEntries.add(exitListEntry); - this.oldExitListEntries.addAll(exitListEntry.oldEntries()); List<String> unrecognizedExitListEntryLines = exitListEntry .getAndClearUnrecognizedLines(); if (unrecognizedExitListEntryLines != null) { @@ -126,13 +117,7 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { return this.downloadedMillis; } - private Set<ExitListEntry> oldExitListEntries = new HashSet<>(); - - @Deprecated - @Override - public Set<ExitListEntry> getExitListEntries() { - return new HashSet<>(this.oldExitListEntries); - } + private Set<ExitList.Entry> oldExitListEntries = new HashSet<>(); private Set<ExitList.Entry> exitListEntries = new HashSet<>(); diff --git a/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java index 32a4dcc..3f84902 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java @@ -35,10 +35,9 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl Key.PADDING_COUNTS); protected ExtraInfoDescriptorImpl(byte[] descriptorBytes, - int[] offsetAndLimit, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, offsetAndLimit, descriptorFile, - failUnrecognizedDescriptorLines, false); + int[] offsetAndLimit, File descriptorFile) + throws DescriptorParseException { + super(descriptorBytes, offsetAndLimit, descriptorFile, false); this.parseDescriptorBytes(); this.checkExactlyOnceKeys(exactlyOnceKeys); Set<Key> dirreqStatsKeys = EnumSet.of( @@ -260,16 +259,10 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl this.parseIdentityEd25519CryptoBlock(cryptoString); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized crypto " - + "block '" + cryptoString + "' in extra-info " - + "descriptor."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.addAll(cryptoLines); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.addAll(cryptoLines); } cryptoLines = null; nextCrypto = Key.EMPTY; @@ -279,15 +272,10 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl cryptoLines.add(line); } else { ParseHelper.parseKeyword(line, partsNoOpt[0]); - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in extra-info descriptor."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -833,16 +821,6 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl this.setDigestSha256Base64(partsNoOpt[1]); } - @Override - public String getExtraInfoDigest() { - return this.getDigestSha1Hex(); - } - - @Override - public String getExtraInfoDigestSha256() { - return this.getDigestSha256Base64(); - } - private String nickname; @Override @@ -881,11 +859,6 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl private String geoipDbDigest; @Override - public String getGeoipDbDigest() { - return this.getGeoipDbDigestSha1Hex(); - } - - @Override public String getGeoipDbDigestSha1Hex() { return this.geoipDbDigest; } @@ -893,11 +866,6 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl private String geoip6DbDigest; @Override - public String getGeoip6DbDigest() { - return this.getGeoip6DbDigestSha1Hex(); - } - - @Override public String getGeoip6DbDigestSha1Hex() { return this.geoip6DbDigest; } diff --git a/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java index d9c284e..6acb845 100644 --- a/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java @@ -19,10 +19,9 @@ public class MicrodescriptorImpl extends DescriptorImpl implements Microdescriptor { protected MicrodescriptorImpl(byte[] descriptorBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines) + File descriptorFile) throws DescriptorParseException { - super(descriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, false); + super(descriptorBytes, offsetAndLength, descriptorFile, false); this.parseDescriptorBytes(); this.calculateDigestSha256Base64(Key.ONION_KEY.keyword + NL); this.checkExactlyOnceKeys(EnumSet.of(Key.ONION_KEY)); @@ -89,15 +88,10 @@ public class MicrodescriptorImpl extends DescriptorImpl crypto.append(line).append(NL); } else { ParseHelper.parseKeyword(line, parts[0]); - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in microdescriptor."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -213,11 +207,6 @@ public class MicrodescriptorImpl extends DescriptorImpl } } - @Override - public String getMicrodescriptorDigest() { - return this.getDigestSha256Base64(); - } - private String onionKey; @Override diff --git a/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java b/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java index 64c688e..e6f7bad 100644 --- a/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java @@ -40,8 +40,6 @@ public class NetworkStatusEntryImpl implements NetworkStatusEntry { private boolean microdescConsensus; - private boolean failUnrecognizedDescriptorLines; - private List<String> unrecognizedLines; protected List<String> getAndClearUnrecognizedLines() { @@ -51,14 +49,12 @@ public class NetworkStatusEntryImpl implements NetworkStatusEntry { } protected NetworkStatusEntryImpl(DescriptorImpl parent, int offset, - int length, boolean microdescConsensus, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + int length, boolean microdescConsensus) + throws DescriptorParseException { this.parent = parent; this.offset = offset; this.length = length; this.microdescConsensus = microdescConsensus; - this.failUnrecognizedDescriptorLines = - failUnrecognizedDescriptorLines; this.parseStatusEntryBytes(); this.clearAtMostOnceKeys(); } @@ -120,15 +116,10 @@ public class NetworkStatusEntryImpl implements NetworkStatusEntry { this.parseIdLine(line, parts); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in status entry."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -344,11 +335,6 @@ public class NetworkStatusEntryImpl implements NetworkStatusEntry { private Set<String> microdescriptorDigests; @Override - public Set<String> getMicrodescriptorDigests() { - return this.getMicrodescriptorDigestsSha256Base64(); - } - - @Override public Set<String> getMicrodescriptorDigestsSha256Base64() { return this.microdescriptorDigests == null ? null : new HashSet<>(this.microdescriptorDigests); diff --git a/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java index 5695353..b025c1b 100644 --- a/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java @@ -20,11 +20,10 @@ import java.util.TreeMap; public abstract class NetworkStatusImpl extends DescriptorImpl { protected NetworkStatusImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines, - boolean containsDirSourceEntries, boolean blankLinesAllowed) - throws DescriptorParseException { + File descriptorFile, boolean containsDirSourceEntries, + boolean blankLinesAllowed) throws DescriptorParseException { super(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, blankLinesAllowed); + blankLinesAllowed); this.splitAndParseParts(containsDirSourceEntries); } @@ -102,7 +101,7 @@ public abstract class NetworkStatusImpl extends DescriptorImpl { protected void parseDirSource(int offset, int length) throws DescriptorParseException { DirSourceEntryImpl dirSourceEntry = new DirSourceEntryImpl( - this, offset, length, this.failUnrecognizedDescriptorLines); + this, offset, length); this.dirSourceEntries.put(dirSourceEntry.getIdentity(), dirSourceEntry); List<String> unrecognizedDirSourceLines = dirSourceEntry @@ -141,7 +140,7 @@ public abstract class NetworkStatusImpl extends DescriptorImpl { protected void parseStatusEntry(int offset, int length) throws DescriptorParseException { NetworkStatusEntryImpl statusEntry = new NetworkStatusEntryImpl( - this, offset, length, false, this.failUnrecognizedDescriptorLines); + this, offset, length, false); this.statusEntries.put(statusEntry.getFingerprint(), statusEntry); List<String> unrecognizedStatusEntryLines = statusEntry .getAndClearUnrecognizedLines(); @@ -162,7 +161,7 @@ public abstract class NetworkStatusImpl extends DescriptorImpl { this.signatures = new ArrayList<>(); } DirectorySignatureImpl signature = new DirectorySignatureImpl( - this, offset, length, failUnrecognizedDescriptorLines); + this, offset, length); this.signatures.add(signature); List<String> unrecognizedStatusEntryLines = signature .getAndClearUnrecognizedLines(); diff --git a/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java index 79ab862..a312f24 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java @@ -19,10 +19,8 @@ public class RelayDirectoryImpl extends DescriptorImpl implements RelayDirectory { protected RelayDirectoryImpl(byte[] directoryBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - super(directoryBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, true); + File descriptorFile) throws DescriptorParseException { + super(directoryBytes, offsetAndLength, descriptorFile, true); this.splitAndParseParts(); this.calculateDigestSha1Hex(Key.SIGNED_DIRECTORY.keyword + NL, NL + Key.DIRECTORY_SIGNATURE.keyword + SP); @@ -144,15 +142,10 @@ public class RelayDirectoryImpl extends DescriptorImpl if (crypto != null) { crypto.append(line).append(NL); } else { - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in v1 directory."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -195,8 +188,7 @@ public class RelayDirectoryImpl extends DescriptorImpl try { ServerDescriptorImpl serverDescriptor = new RelayServerDescriptorImpl(this.rawDescriptorBytes, - new int[] { offset, length }, this.getDescriptorFile(), - this.failUnrecognizedDescriptorLines); + new int[] { offset, length }, this.getDescriptorFile()); this.serverDescriptors.add(serverDescriptor); } catch (DescriptorParseException e) { this.serverDescriptorParseExceptions.add(e); @@ -238,9 +230,6 @@ public class RelayDirectoryImpl extends DescriptorImpl default: if (crypto != null) { crypto.append(line).append(NL); - } else if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in v2 network status."); } else { if (this.unrecognizedLines == null) { this.unrecognizedLines = new ArrayList<>(); @@ -449,9 +438,5 @@ public class RelayDirectoryImpl extends DescriptorImpl return this.nickname; } - @Override - public String getDirectoryDigest() { - return this.getDigestSha1Hex(); - } } diff --git a/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java index 9644f90..7ed696e 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java @@ -12,10 +12,9 @@ public class RelayExtraInfoDescriptorImpl extends ExtraInfoDescriptorImpl implements RelayExtraInfoDescriptor { protected RelayExtraInfoDescriptorImpl(byte[] descriptorBytes, - int[] offsetAndLimit, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, offsetAndLimit, descriptorFile, - failUnrecognizedDescriptorLines); + int[] offsetAndLimit, File descriptorFile) + throws DescriptorParseException { + super(descriptorBytes, offsetAndLimit, descriptorFile); this.calculateDigestSha1Hex(Key.EXTRA_INFO.keyword + SP, NL + Key.ROUTER_SIGNATURE.keyword + NL); this.calculateDigestSha256Base64(Key.EXTRA_INFO.keyword + SP, diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java index 0e853bb..7c6f7e0 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java @@ -23,10 +23,9 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl implements RelayNetworkStatusConsensus { protected RelayNetworkStatusConsensusImpl(byte[] consensusBytes, - int[] offsetAndLimit, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(consensusBytes, offsetAndLimit, descriptorFile, - failUnrecognizedDescriptorLines, true, false); + int[] offsetAndLimit, File descriptorFile) + throws DescriptorParseException { + super(consensusBytes, offsetAndLimit, descriptorFile, true, false); Set<Key> exactlyOnceKeys = EnumSet.of( Key.VOTE_STATUS, Key.CONSENSUS_METHOD, Key.VALID_AFTER, Key.FRESH_UNTIL, Key.VALID_UNTIL, Key.VOTING_DELAY, Key.KNOWN_FLAGS); @@ -107,14 +106,8 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl this.parseSharedRandCurrentValueLine(line, parts); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in consensus."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } } } @@ -125,8 +118,7 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl protected void parseStatusEntry(int offset, int length) throws DescriptorParseException { NetworkStatusEntryImpl statusEntry = new NetworkStatusEntryImpl(this, - offset, length, this.microdescConsensus, - this.failUnrecognizedDescriptorLines); + offset, length, this.microdescConsensus); this.statusEntries.put(statusEntry.getFingerprint(), statusEntry); List<String> unrecognizedStatusEntryLines = statusEntry .getAndClearUnrecognizedLines(); @@ -152,15 +144,10 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl this.parseBandwidthWeightsLine(line, parts); break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in consensus."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -345,11 +332,6 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl parts, 1); } - @Override - public String getConsensusDigest() { - return this.getDigestSha1Hex(); - } - private int networkStatusVersion; @Override diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java index 8ebea2d..f77e903 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java @@ -22,8 +22,7 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl protected RelayNetworkStatusImpl(byte[] statusBytes, int[] offsetAndLength, File descriptorFile, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(statusBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, false, true); + super(statusBytes, offsetAndLength, descriptorFile, false, true); Set<Key> exactlyOnceKeys = EnumSet.of( Key.NETWORK_STATUS_VERSION, Key.DIR_SOURCE, Key.FINGERPRINT, Key.CONTACT, Key.DIR_SIGNING_KEY, Key.PUBLISHED); @@ -97,9 +96,6 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl default: if (crypto != null) { crypto.append(line).append(NL); - } else if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in v2 network status."); } else { if (this.unrecognizedLines == null) { this.unrecognizedLines = new ArrayList<>(); @@ -149,9 +145,6 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl default: if (crypto != null) { crypto.append(line).append(NL); - } else if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in v2 network status."); } else { if (this.unrecognizedLines == null) { this.unrecognizedLines = new ArrayList<>(); @@ -247,11 +240,6 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl this.nickname = ParseHelper.parseNickname(line, parts[1]); } - @Override - public String getStatusDigest() { - return this.getDigestSha1Hex(); - } - private int networkStatusVersion; @Override diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java index 93d6f12..df36b26 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java @@ -4,7 +4,6 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.DescriptorParseException; -import org.torproject.descriptor.DirectorySignature; import org.torproject.descriptor.RelayNetworkStatusVote; import java.io.File; @@ -25,10 +24,9 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl implements RelayNetworkStatusVote { protected RelayNetworkStatusVoteImpl(byte[] voteBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines) + File descriptorFile) throws DescriptorParseException { - super(voteBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, false, false); + super(voteBytes, offsetAndLength, descriptorFile, false, false); Set<Key> exactlyOnceKeys = EnumSet.of( Key.VOTE_STATUS, Key.PUBLISHED, Key.VALID_AFTER, Key.FRESH_UNTIL, Key.VALID_UNTIL, Key.VOTING_DELAY, Key.KNOWN_FLAGS, Key.DIR_SOURCE, @@ -213,15 +211,10 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl if (crypto != null) { crypto.append(line).append(NL); } else { - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in vote."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -588,15 +581,10 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl while (scanner.hasNext()) { String line = scanner.next(); if (!line.equals(Key.DIRECTORY_FOOTER.keyword)) { - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in vote."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -748,21 +736,6 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl return this.dirKeyCertification; } - @Override - public String getSigningKeyDigest() { - String signingKeyDigest = null; - if (this.signatures != null && !this.signatures.isEmpty()) { - for (DirectorySignature signature : this.signatures) { - if (DirectorySignatureImpl.DEFAULT_ALGORITHM.equals( - signature.getAlgorithm())) { - signingKeyDigest = signature.getSigningKeyDigestSha1Hex(); - break; - } - } - } - return signingKeyDigest; - } - private int networkStatusVersion; @Override diff --git a/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java index 23e9152..fd2464b 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java @@ -12,10 +12,9 @@ public class RelayServerDescriptorImpl extends ServerDescriptorImpl implements RelayServerDescriptor { protected RelayServerDescriptorImpl(byte[] descriptorBytes, - int[] offsetAndLength, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines); + int[] offsetAndLength, File descriptorFile) + throws DescriptorParseException { + super(descriptorBytes, offsetAndLength, descriptorFile); this.calculateDigestSha1Hex(Key.ROUTER.keyword + SP, NL + Key.ROUTER_SIGNATURE.keyword + NL); this.calculateDigestSha256Base64(Key.ROUTER.keyword + SP, diff --git a/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java index f434d84..80ed473 100644 --- a/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java @@ -36,10 +36,8 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl Key.ROUTER, Key.BANDWIDTH, Key.PUBLISHED); protected ServerDescriptorImpl(byte[] descriptorBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - super(descriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines, false); + File descriptorFile) throws DescriptorParseException { + super(descriptorBytes, offsetAndLength, descriptorFile, false); this.parseDescriptorBytes(); this.checkExactlyOnceKeys(exactlyOnce); this.checkAtMostOnceKeys(atMostOnce); @@ -210,15 +208,10 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl this.ntorOnionKeyCrosscert = cryptoString; break; default: - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized crypto " - + "block '" + cryptoString + "' in server descriptor."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.addAll(cryptoLines); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.addAll(cryptoLines); } cryptoLines = null; nextCrypto = Key.EMPTY; @@ -229,15 +222,10 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl cryptoLines.add(line); } else { ParseHelper.parseKeyword(line, partsNoOpt[0]); - if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized line '" - + line + "' in server descriptor."); - } else { - if (this.unrecognizedLines == null) { - this.unrecognizedLines = new ArrayList<>(); - } - this.unrecognizedLines.add(line); + if (this.unrecognizedLines == null) { + this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } @@ -676,16 +664,6 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl this.setDigestSha256Base64(partsNoOpt[1]); } - @Override - public String getServerDescriptorDigest() { - return this.getDigestSha1Hex(); - } - - @Override - public String getServerDescriptorDigestSha256() { - return this.getDigestSha256Base64(); - } - private String nickname; @Override @@ -865,11 +843,6 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl private String extraInfoDigest; @Override - public String getExtraInfoDigest() { - return this.getExtraInfoDigestSha1Hex(); - } - - @Override public String getExtraInfoDigestSha1Hex() { return this.extraInfoDigest; } @@ -877,11 +850,6 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl private String extraInfoDigestSha256; @Override - public String getExtraInfoDigestSha256() { - return this.getExtraInfoDigestSha256Base64(); - } - - @Override public String getExtraInfoDigestSha256Base64() { return this.extraInfoDigestSha256; } diff --git a/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java b/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java index 1ff0fe3..408b08c 100644 --- a/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java @@ -22,8 +22,8 @@ public class TorperfResultImpl extends DescriptorImpl implements TorperfResult { protected static List<Descriptor> parseTorperfResults( - byte[] rawDescriptorBytes, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + byte[] rawDescriptorBytes, File descriptorFile) + throws DescriptorParseException { if (rawDescriptorBytes.length == 0) { throw new DescriptorParseException("Descriptor is empty."); } @@ -50,18 +50,17 @@ public class TorperfResultImpl extends DescriptorImpl /* XXX21932 */ parsedDescriptors.add(new TorperfResultImpl( (typeAnnotation + line).getBytes(StandardCharsets.UTF_8), - descriptorFile, failUnrecognizedDescriptorLines)); + descriptorFile)); typeAnnotation = ""; } } return parsedDescriptors; } - protected TorperfResultImpl(byte[] rawDescriptorBytes, File descriptorFile, - boolean failUnrecognizedDescriptorLines) + protected TorperfResultImpl(byte[] rawDescriptorBytes, File descriptorFile) throws DescriptorParseException { super(rawDescriptorBytes, new int[] { 0, rawDescriptorBytes.length }, - descriptorFile, failUnrecognizedDescriptorLines, false); + descriptorFile, false); this.parseTorperfResultLine(); } @@ -173,9 +172,6 @@ public class TorperfResultImpl extends DescriptorImpl default: if (key.startsWith("DATAPERC")) { this.parseDataPercentile(value, keyValue, line); - } else if (this.failUnrecognizedDescriptorLines) { - throw new DescriptorParseException("Unrecognized key '" + key - + "' in line '" + line + "'."); } else { if (this.unrecognizedKeys == null) { this.unrecognizedKeys = new TreeMap<>(); diff --git a/src/main/java/org/torproject/descriptor/impl/UnparseableDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/UnparseableDescriptorImpl.java index 4a0e772..7174621 100644 --- a/src/main/java/org/torproject/descriptor/impl/UnparseableDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/UnparseableDescriptorImpl.java @@ -14,10 +14,8 @@ public class UnparseableDescriptorImpl extends DescriptorImpl protected UnparseableDescriptorImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, File descriptorFile, - boolean failUnrecognizedDescriptorLines, DescriptorParseException descriptorParseException) { - super(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines); + super(rawDescriptorBytes, offsetAndLength, descriptorFile); this.descriptorParseException = descriptorParseException; } diff --git a/src/test/java/org/torproject/descriptor/DummyDownloaderImplementation.java b/src/test/java/org/torproject/descriptor/DummyDownloaderImplementation.java deleted file mode 100644 index dce3372..0000000 --- a/src/test/java/org/torproject/descriptor/DummyDownloaderImplementation.java +++ /dev/null @@ -1,15 +0,0 @@ -/* Copyright 2016--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor; - -import org.torproject.descriptor.impl.DescriptorDownloaderImpl; - -class DummyDownloaderImplementation extends DescriptorDownloaderImpl { - - static int count; - - public DummyDownloaderImplementation() { - count++; - } -} diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorCollectorImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorCollectorImplTest.java deleted file mode 100644 index c2f9674..0000000 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorCollectorImplTest.java +++ /dev/null @@ -1,121 +0,0 @@ -/* Copyright 2015--2017 The Tor Project - * See LICENSE for licensing information */ - -package org.torproject.descriptor.impl; - -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertSame; -import static org.junit.Assert.assertTrue; - -import org.junit.Test; - -import java.util.SortedMap; - -public class DescriptorCollectorImplTest { - - private static final String REMOTE_DIRECTORY_CONSENSUSES = - "/recent/relay-descriptors/consensuses/"; - - @Test - public void testOneFile() { - String remoteFilename = "2015-05-24-12-00-00-consensus"; - String directoryListing = "<tr><td valign=\"top\">" - + "<img src=\"/icons/unknown.gif\" alt=\"[ ]\"></td><td>" - + "<a href=\"" + remoteFilename + "\">" - + "2015-05-24-12-00-00-consensus</a></td>" - + "<td align=\"right\">2015-05-24 12:08 </td>" - + "<td align=\"right\">1.5M</td><td> </td></tr>"; - SortedMap<String, Long> remoteFiles = - new DescriptorCollectorImpl().parseDirectoryListing( - REMOTE_DIRECTORY_CONSENSUSES, directoryListing); - assertNotNull(remoteFiles); - assertSame(1, remoteFiles.size()); - assertEquals(REMOTE_DIRECTORY_CONSENSUSES + remoteFilename, - remoteFiles.firstKey()); - assertEquals((Long) 1432469280000L, - remoteFiles.get(remoteFiles.firstKey())); - } - - @Test - public void testSameFileTwoTimestampsLastWins() { - String remoteFilename = "2015-05-24-12-00-00-consensus"; - String firstTimestamp = "2015-05-24 12:04"; - String secondTimestamp = "2015-05-24 12:08"; - String lineFormat = "<tr><td valign=\"top\">" - + "<img src=\"/icons/unknown.gif\" alt=\"[ ]\"></td><td>" - + "<a href=\"%s\">2015-05-24-12-00-00-consensus</a></td>" - + "<td align=\"right\">%s </td>" - + "<td align=\"right\">1.5M</td><td> </td></tr>\n"; - String directoryListing = String.format(lineFormat + lineFormat, - remoteFilename, firstTimestamp, remoteFilename, secondTimestamp); - SortedMap<String, Long> remoteFiles = - new DescriptorCollectorImpl().parseDirectoryListing( - REMOTE_DIRECTORY_CONSENSUSES, directoryListing); - assertNotNull(remoteFiles); - assertSame(1, remoteFiles.size()); - assertEquals(REMOTE_DIRECTORY_CONSENSUSES + remoteFilename, - remoteFiles.firstKey()); - assertEquals((Long) 1432469280000L, - remoteFiles.get(remoteFiles.firstKey())); - } - - @Test - public void testSubDirectoryOnly() { - String directoryListing = "<tr><td valign=\"top\">" - + "<img src=\"/icons/folder.gif\" alt=\"[DIR]\"></td><td>" - + "<a href=\"subdir/\">subdir/</a></td>" - + "<td align=\"right\">2015-05-27 14:07 </td>" - + "<td align=\"right\"> - </td><td> </td></tr>"; - DescriptorCollectorImpl collector = new DescriptorCollectorImpl(); - SortedMap<String, Long> remoteFiles = collector.parseDirectoryListing( - REMOTE_DIRECTORY_CONSENSUSES, directoryListing); - assertNotNull(remoteFiles); - assertTrue(remoteFiles.isEmpty()); - } - - @Test - public void testParentDirectoryOnly() { - String directoryListing = "<tr><td valign=\"top\">" - + "<img src=\"/icons/back.gif\" alt=\"[DIR]\"></td><td>" - + "<a href=\"/recent/relay-descriptors/\">Parent Directory</a>" - + "</td><td> </td><td align=\"right\"> - </td>" - + "<td> </td></tr>"; - DescriptorCollectorImpl collector = new DescriptorCollectorImpl(); - SortedMap<String, Long> remoteFiles = collector.parseDirectoryListing( - REMOTE_DIRECTORY_CONSENSUSES, directoryListing); - assertNotNull(remoteFiles); - assertTrue(remoteFiles.isEmpty()); - } - - @Test - public void testUnexpectedDateFormat() { - String directoryListing = "<tr><td valign=\"top\">" - + "<img src=\"/icons/unknown.gif\" alt=\"[ ]\"></td><td>" - + "<a href=\"2015-05-24-12-00-00-consensus\">" - + "2015-05-24-12-00-00-consensus</a></td>" - + "<td align=\"right\">24-May-2015 12:08 </td>" - + "<td align=\"right\">1.5M</td><td> </td></tr>"; - SortedMap<String, Long> remoteFiles = - new DescriptorCollectorImpl().parseDirectoryListing( - REMOTE_DIRECTORY_CONSENSUSES, directoryListing); - assertNotNull(remoteFiles); - assertTrue(remoteFiles.isEmpty()); - } - - @Test - public void testInvalidDate() { - String directoryListing = "<tr><td valign=\"top\">" - + "<img src=\"/icons/unknown.gif\" alt=\"[ ]\"></td><td>" - + "<a href=\"2015-05-24-12-00-00-consensus\">" - + "2015-05-24-12-00-00-consensus</a></td>" - + "<td align=\"right\">2015-05-34 12:08 </td>" - + "<td align=\"right\">1.5M</td><td> </td></tr>"; - SortedMap<String, Long> remoteFiles = - new DescriptorCollectorImpl().parseDirectoryListing( - REMOTE_DIRECTORY_CONSENSUSES, directoryListing); - assertNull(remoteFiles); - } -} -

1 0

[metrics-lib/master] Make tests pass again.
by karsten＠torproject.org 27 Jun '17

27 Jun '17

commit c57517c7a950e5f814ed00da5753a35c00dbf574 Author: iwakeh <iwakeh(a)torproject.org> Date: Fri Jun 23 08:37:48 2017 +0000 Make tests pass again. (one still fails: testDirectorySignaturesLinesTwoAlgorithms in RelayNetworkStatusVoteImplTest) Part of task-22154. --- .../descriptor/impl/DescriptorParserImpl.java | 47 ++++------- .../descriptor/benchmark/MeasurePerformance.java | 13 ++- .../descriptor/impl/DescriptorImplTest.java | 4 +- .../descriptor/impl/DescriptorParserImplTest.java | 8 +- .../descriptor/impl/DescriptorReaderImplTest.java | 34 ++++---- .../descriptor/impl/ExitListImplTest.java | 30 ------- .../impl/ExtraInfoDescriptorImplTest.java | 20 ----- .../impl/RelayNetworkStatusConsensusImplTest.java | 96 ++-------------------- .../impl/RelayNetworkStatusImplTest.java | 3 +- .../impl/RelayNetworkStatusVoteImplTest.java | 47 +---------- .../descriptor/impl/ServerDescriptorImplTest.java | 33 +------- .../torproject/descriptor/impl/TestDescriptor.java | 5 ++ 12 files changed, 65 insertions(+), 275 deletions(-) diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java index dbdfca9..320139d 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java @@ -60,8 +60,7 @@ public class DescriptorParserImpl implements DescriptorParser { && firstLines.contains( NL + Key.VOTE_STATUS.keyword + SP + "consensus" + NL))) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.NETWORK_STATUS_VERSION, RelayNetworkStatusConsensusImpl.class, - includeUnparseableDescriptors); + Key.NETWORK_STATUS_VERSION, RelayNetworkStatusConsensusImpl.class); } else if (firstLines.startsWith("@type network-status-vote-3 1.") || ((firstLines.startsWith( Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL) @@ -70,8 +69,7 @@ public class DescriptorParserImpl implements DescriptorParser { && firstLines.contains( NL + Key.VOTE_STATUS.keyword + SP + "vote" + NL))) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.NETWORK_STATUS_VERSION, RelayNetworkStatusVoteImpl.class, - includeUnparseableDescriptors); + Key.NETWORK_STATUS_VERSION, RelayNetworkStatusVoteImpl.class); } else if (firstLines.startsWith("@type bridge-network-status 1.") || firstLines.startsWith(Key.R.keyword + SP)) { List<Descriptor> parsedDescriptors = new ArrayList<>(); @@ -81,43 +79,36 @@ public class DescriptorParserImpl implements DescriptorParser { return parsedDescriptors; } else if (firstLines.startsWith("@type bridge-server-descriptor 1.")) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.ROUTER, BridgeServerDescriptorImpl.class, - includeUnparseableDescriptors); + Key.ROUTER, BridgeServerDescriptorImpl.class); } else if (firstLines.startsWith("@type server-descriptor 1.") || firstLines.startsWith(Key.ROUTER.keyword + SP) || firstLines.contains(NL + Key.ROUTER.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.ROUTER, RelayServerDescriptorImpl.class, - includeUnparseableDescriptors); + Key.ROUTER, RelayServerDescriptorImpl.class); } else if (firstLines.startsWith("@type bridge-extra-info 1.")) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.EXTRA_INFO, BridgeExtraInfoDescriptorImpl.class, - includeUnparseableDescriptors); + Key.EXTRA_INFO, BridgeExtraInfoDescriptorImpl.class); } else if (firstLines.startsWith("@type extra-info 1.") || firstLines.startsWith(Key.EXTRA_INFO.keyword + SP) || firstLines.contains(NL + Key.EXTRA_INFO.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.EXTRA_INFO, RelayExtraInfoDescriptorImpl.class, - includeUnparseableDescriptors); + Key.EXTRA_INFO, RelayExtraInfoDescriptorImpl.class); } else if (firstLines.startsWith("@type microdescriptor 1.") || firstLines.startsWith(Key.ONION_KEY.keyword + NL) || firstLines.contains(NL + Key.ONION_KEY.keyword + NL)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.ONION_KEY, MicrodescriptorImpl.class, - includeUnparseableDescriptors); + Key.ONION_KEY, MicrodescriptorImpl.class); } else if (firstLines.startsWith("@type bridge-pool-assignment 1.") || firstLines.startsWith(Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP) || firstLines.contains(NL + Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.BRIDGE_POOL_ASSIGNMENT, BridgePoolAssignmentImpl.class, - includeUnparseableDescriptors); + Key.BRIDGE_POOL_ASSIGNMENT, BridgePoolAssignmentImpl.class); } else if (firstLines.startsWith("@type dir-key-certificate-3 1.") || firstLines.startsWith(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP) || firstLines.contains( NL + Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.DIR_KEY_CERTIFICATE_VERSION, DirectoryKeyCertificateImpl.class, - includeUnparseableDescriptors); + Key.DIR_KEY_CERTIFICATE_VERSION, DirectoryKeyCertificateImpl.class); } else if (firstLines.startsWith("@type tordnsel 1.") || firstLines.startsWith("ExitNode" + SP) || firstLines.contains(NL + "ExitNode" + SP)) { @@ -131,14 +122,12 @@ public class DescriptorParserImpl implements DescriptorParser { || firstLines.contains( NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.NETWORK_STATUS_VERSION, RelayNetworkStatusImpl.class, - includeUnparseableDescriptors); + Key.NETWORK_STATUS_VERSION, RelayNetworkStatusImpl.class); } else if (firstLines.startsWith("@type directory 1.") || firstLines.startsWith(Key.SIGNED_DIRECTORY.keyword + NL) || firstLines.contains(NL + Key.SIGNED_DIRECTORY.keyword + NL)) { return this.parseDescriptors(rawDescriptorBytes, descriptorFile, - Key.SIGNED_DIRECTORY, RelayDirectoryImpl.class, - includeUnparseableDescriptors); + Key.SIGNED_DIRECTORY, RelayDirectoryImpl.class); } else if (firstLines.startsWith("@type torperf 1.")) { return TorperfResultImpl.parseTorperfResults(rawDescriptorBytes, descriptorFile); @@ -150,13 +139,13 @@ public class DescriptorParserImpl implements DescriptorParser { private List<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, File descriptorFile, Key key, - Class<? extends DescriptorImpl> descriptorClass, - boolean includeUnparseableDescriptors) throws DescriptorParseException { + Class<? extends DescriptorImpl> descriptorClass) + throws DescriptorParseException { List<Descriptor> parsedDescriptors = new ArrayList<>(); Constructor<? extends DescriptorImpl> constructor; try { constructor = descriptorClass.getDeclaredConstructor(byte[].class, - int[].class, File.class, boolean.class); + int[].class, File.class); } catch (NoSuchMethodException e) { throw new RuntimeException(e); } @@ -204,12 +193,8 @@ public class DescriptorParserImpl implements DescriptorParser { parsedDescriptors.add(this.parseDescriptor(rawDescriptorBytes, offsetAndLength, descriptorFile, constructor)); } catch (DescriptorParseException e) { - if (includeUnparseableDescriptors) { - parsedDescriptors.add(new UnparseableDescriptorImpl( - rawDescriptorBytes, offsetAndLength, descriptorFile, e)); - } else { - throw e; - } + parsedDescriptors.add(new UnparseableDescriptorImpl( + rawDescriptorBytes, offsetAndLength, descriptorFile, e)); } startAnnotations = endDescriptor; } diff --git a/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java b/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java index 83faf05..09c581d 100644 --- a/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java +++ b/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java @@ -4,7 +4,6 @@ package org.torproject.descriptor.benchmark; import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.DescriptorReader; import org.torproject.descriptor.DescriptorSourceFactory; import org.torproject.descriptor.ExtraInfoDescriptor; @@ -136,9 +135,9 @@ public class MeasurePerformance { continue; } ExtraInfoDescriptor extraInfoDescriptor = - (ExtraInfoDescriptor) descriptor; + (ExtraInfoDescriptor) descriptor; SortedMap<String, Integer> dirreqV3Reqs = - extraInfoDescriptor.getDirreqV3Reqs(); + extraInfoDescriptor.getDirreqV3Reqs(); if (dirreqV3Reqs != null) { countries.addAll(dirreqV3Reqs.keySet()); } @@ -174,9 +173,8 @@ public class MeasurePerformance { continue; } RelayNetworkStatusConsensus consensus = - (RelayNetworkStatusConsensus) descriptor; - for (NetworkStatusEntry entry : - consensus.getStatusEntries().values()) { + (RelayNetworkStatusConsensus) descriptor; + for (NetworkStatusEntry entry : consensus.getStatusEntries().values()) { if (entry.getFlags().contains("Exit")) { totalRelaysWithExitFlag++; } @@ -217,8 +215,7 @@ public class MeasurePerformance { continue; } countedMicrodescriptors++; - Microdescriptor microdescriptor = - (Microdescriptor) descriptor; + Microdescriptor microdescriptor = (Microdescriptor) descriptor; String defaultPolicy = microdescriptor.getDefaultPolicy(); if (defaultPolicy == null) { continue; diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java index cc97d1d..09bac87 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java @@ -3,11 +3,11 @@ package org.torproject.descriptor.impl; -import org.torproject.descriptor.DescriptorParseException; - import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNull; +import org.torproject.descriptor.DescriptorParseException; + import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExpectedException; diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java index a2a3d9e..61dc119 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java @@ -19,8 +19,9 @@ public class DescriptorParserImplTest { @Rule public ExpectedException thrown = ExpectedException.none(); - private TestDescriptor makeTestDesc(byte[] bytes) throws Exception { - return new TestDescriptor(bytes, new int[]{0, bytes.length}, false); + private TestDescriptor makeTestDesc(byte[] bytes) + throws Exception { + return new TestDescriptor(bytes, new int[]{0, bytes.length}); } private static final String MICRO = @@ -47,8 +48,7 @@ public class DescriptorParserImplTest { Constructor<? extends DescriptorImpl> constructor; try { constructor = TestServerDescriptor.class - .getDeclaredConstructor(byte[].class, int[].class, File.class, - boolean.class); + .getDeclaredConstructor(byte[].class, int[].class, File.class); } catch (NoSuchMethodException e) { throw new RuntimeException(e); } diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java index fa791a2..a00e5ba 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java @@ -75,10 +75,8 @@ public class DescriptorReaderImplTest { } private void assertExcludedFilesParsedFilesAndHistoryFileLines( - int expectedExcludedFiles, int expectedParsedFiles, + int expectedParsedFiles, int expectedHistoryFileLines) throws IOException { - assertEquals(expectedExcludedFiles, - this.descriptorReader.getExcludedFiles().size()); assertEquals(expectedParsedFiles, this.descriptorReader.getParsedFiles().size()); assertEquals(expectedHistoryFileLines, @@ -89,13 +87,21 @@ public class DescriptorReaderImplTest { @Test public void testDescriptors() throws IOException { this.readAllDescriptors(this.inputDirectory); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 2, 1); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 1); } @Test - public void testNoDescriptors() throws IOException { - this.readAllDescriptors(null); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 1); + public void testNoDescriptors() throws Exception { + // calling readAllDescriptors fails with IAE + Iterator<Descriptor> descriptors = this.descriptorReader + .readDescriptors(null).iterator(); + int count = 0; + while (descriptors.hasNext()) { + count++; + descriptors.next(); + } + assertEquals(count, 0); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 1); } @Test @@ -104,7 +110,7 @@ public class DescriptorReaderImplTest { IOException { this.readAllDescriptors(this.inputDirectory); Thread.sleep(100L); /* It may take a moment to write the history file. */ - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 2); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 1); } @Test @@ -112,7 +118,7 @@ public class DescriptorReaderImplTest { IOException { this.readAllDescriptors(this.inputDirectory); Thread.sleep(100L); /* It may take a moment to write the history file. */ - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 0); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 1); } @Test @@ -120,7 +126,7 @@ public class DescriptorReaderImplTest { this.descriptorReader.setHistoryFile(this.historyFile); this.readAllDescriptors(this.inputDirectory); descriptorReader.saveHistoryFile(this.historyFile); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 2); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 2); } @Test @@ -128,21 +134,21 @@ public class DescriptorReaderImplTest { this.descriptorReader.setHistoryFile(this.historyFile); this.readAllDescriptors(this.inputDirectory); this.descriptorReader.saveHistoryFile(this.historyFile); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 0); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 2); } @Test public void testSetExcludedFilesDescriptors() throws IOException { this.descriptorReader.setExcludedFiles(this.historyMap); this.readAllDescriptors(this.inputDirectory); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 1); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1); } @Test public void testSetExcludedFilesNoDescriptors() throws IOException { this.descriptorReader.setExcludedFiles(this.historyMap); this.readAllDescriptors(this.inputDirectory); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 1); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1); } @Test @@ -153,7 +159,7 @@ public class DescriptorReaderImplTest { int count = this.readAllDescriptors(this.inputDirectory); assertEquals("Two files should have been parsed.", 2, count); descriptorReader.saveHistoryFile(this.historyFile); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 2, 2); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 2); } } diff --git a/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java b/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java index dabea1e..8365d4c 100644 --- a/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java @@ -7,13 +7,11 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import org.torproject.descriptor.DescriptorParseException; -import org.torproject.descriptor.ExitList; import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExpectedException; -import java.util.HashMap; import java.util.Map; public class ExitListImplTest { @@ -32,39 +30,11 @@ public class ExitListImplTest { assertEquals(1441065722000L, result.getDownloadedMillis()); assertTrue("Unrecognized lines: " + result.getUnrecognizedLines(), result.getUnrecognizedLines().isEmpty()); - assertEquals("Found: " + result.getEntries(), 7, - result.getEntries().size()); assertEquals("Found: " + result.getEntries(), 5, result.getEntries().size()); } @Test - public void testMultipleOldExitAddresses() throws Exception { - ExitListImpl result = new ExitListImpl( - (tordnselAnnotation + multiExitAddressInput) - .getBytes("US-ASCII"), null, fileName); - assertTrue("Unrecognized lines: " + result.getUnrecognizedLines(), - result.getUnrecognizedLines().isEmpty()); - assertEquals("Found: " + result.getEntries(), - 3, result.getEntries().size()); - Map<String, Long> testMap = new HashMap(); - testMap.put("81.7.17.171", 1441044592000L); - testMap.put("81.7.17.172", 1441044652000L); - testMap.put("81.7.17.173", 1441044712000L); - for (ExitList.Entry ele : result.getEntries()) { - Map<String, Long> map = ele.getExitAddresses(); - assertEquals("Found: " + map, 1, map.size()); - Map.Entry<String, Long> ea = map.entrySet().iterator().next(); - assertTrue("Map: " + testMap, - testMap.keySet().contains(ea.getKey())); - assertTrue("Map: " + testMap + " exitaddress: " + ea, - testMap.values().contains(ea.getValue())); - testMap.remove(ea.getKey()); - } - assertTrue("Map: " + testMap, testMap.isEmpty()); - } - - @Test public void testMultipleExitAddresses() throws Exception { ExitListImpl result = new ExitListImpl( (tordnselAnnotation + multiExitAddressInput) diff --git a/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java index 901aca8..9633f2b 100644 --- a/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java @@ -2160,15 +2160,6 @@ public class ExtraInfoDescriptorImplTest { } @Test - public void testUnrecognizedLineFail() throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'unrecognized-line 1' in " - + "extra-info descriptor."); - String unrecognizedLine = "unrecognized-line 1"; - DescriptorBuilder.createWithUnrecognizedLine(unrecognizedLine); - } - - @Test public void testUnrecognizedLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -2292,17 +2283,6 @@ public class ExtraInfoDescriptorImplTest { } @Test - public void testEd25519FollowedbyUnrecognizedLine() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in extra-info descriptor."); - DescriptorBuilder.createWithEd25519Lines(IDENTITY_ED25519_LINES, - MASTER_KEY_ED25519_LINE, ROUTER_SIG_ED25519_LINE - + "\nunrecognized-line 1"); - } - - @Test public void testExtraInfoDigestSha256Relay() throws DescriptorParseException { byte[] descriptorBytes = ("extra-info Unnamed " diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java index 02b89e1..cafbf31 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java @@ -436,21 +436,13 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage( - "Unrecognized line 'directory-footer' in consensus."); + "Keyword 'directory-footer' is contained 2 times, but must be " + + "contained at most once."); ConsensusBuilder.createWithNetworkStatusVersionLine( "directory-footer\nnetwork-status-version 3"); } @Test - public void testNetworkStatusVersionPrefixLinePoundChar() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line '#consensus' in consensus."); - ConsensusBuilder.createWithNetworkStatusVersionLine( - "#consensus\nnetwork-status-version 3"); - } - - @Test public void testNetworkStatusVersionNoSpace() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); @@ -1478,19 +1470,6 @@ public class RelayNetworkStatusConsensusImplTest { } @Test - public void testDirectoryFooterNoLine() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'bandwidth-weights Wbd=285 " - + "Wbe=0 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=1021 Wee=10000 " - + "Weg=1021 Wem=10000 Wgb=10000 Wgd=8694 Wgg=10000 Wgm=10000 Wmb=10000 " - + "Wmd=285 Wme=0 Wmg=0 Wmm=10000' in status entry."); - /* This breaks, because a bandwidth-weights line without a preceding - * directory-footer line is not allowed. */ - ConsensusBuilder.createWithDirectoryFooterLine(null); - } - - @Test public void testDirectoryFooterMissing() throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); @@ -1593,8 +1572,8 @@ public class RelayNetworkStatusConsensusImplTest { @Test public void testNonAsciiByte20() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'network-status-versi�n 3' in " - + "consensus."); + this.thrown.expectMessage("Keyword 'network-status-version' must be " + + "contained in the first line."); ConsensusBuilder cb = new ConsensusBuilder(); byte[] consensusBytes = cb.buildConsensusBytes(); consensusBytes[20] = (byte) 200; @@ -1606,8 +1585,8 @@ public class RelayNetworkStatusConsensusImplTest { public void testNonAsciiByteMinusOne() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line '�network-status-version 3' in consensus."); + this.thrown.expectMessage("Keyword 'network-status-version' " + + "must be contained in the first line."); ConsensusBuilder cb = new ConsensusBuilder(); cb.networkStatusVersionLine = "Xnetwork-status-version 3"; byte[] consensusBytes = cb.buildConsensusBytes(); @@ -1617,37 +1596,6 @@ public class RelayNetworkStatusConsensusImplTest { } @Test - public void testUnrecognizedHeaderLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in consensus."); - String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedHeaderLine(unrecognizedLine); - } - - @Test - public void testUnrecognizedHeaderLineIgnore() - throws DescriptorParseException { - String unrecognizedLine = "unrecognized-line 1"; - RelayNetworkStatusConsensus consensus = ConsensusBuilder - .createWithUnrecognizedHeaderLine(unrecognizedLine); - List<String> unrecognizedLines = new ArrayList<>(); - unrecognizedLines.add(unrecognizedLine); - assertEquals(unrecognizedLines, consensus.getUnrecognizedLines()); - } - - @Test - public void testUnrecognizedDirSourceLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in dir-source entry."); - String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedDirSourceLine(unrecognizedLine); - } - - @Test public void testUnrecognizedDirSourceLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1659,17 +1607,6 @@ public class RelayNetworkStatusConsensusImplTest { } @Test - public void testUnrecognizedStatusEntryLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'unrecognized-line 1' in " - + "status entry."); - String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedStatusEntryLine( - unrecognizedLine); - } - - @Test public void testUnrecognizedStatusEntryLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1681,16 +1618,6 @@ public class RelayNetworkStatusConsensusImplTest { } @Test - public void testUnrecognizedDirectoryFooterLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in consensus."); - String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedFooterLine(unrecognizedLine); - } - - @Test public void testUnrecognizedDirectoryFooterLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1702,17 +1629,6 @@ public class RelayNetworkStatusConsensusImplTest { } @Test - public void testUnrecognizedDirectorySignatureLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in dir-source entry."); - String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedDirectorySignatureLine( - unrecognizedLine); - } - - @Test public void testUnrecognizedDirectorySignatureLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java index 760d04c..55ce6eb 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java @@ -44,7 +44,8 @@ public class RelayNetworkStatusImplTest { @Test public void testParseBrokenHeader() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'xyx' in v2 network status."); + this.thrown.expectMessage( + "Unrecognized crypto block in v2 network status."); String invalidHeader = "network-status-version 2\nxyx\nabc"; byte[] statusBytes = (validAnnotation + invalidHeader + validFooter) .getBytes(); diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java index 060cbbb..b0f53bd 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java @@ -791,7 +791,8 @@ public class RelayNetworkStatusVoteImplTest { public void testNetworkStatusVersionPrefixLine() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'directory-footer' in vote."); + this.thrown.expectMessage("Keyword 'directory-footer' is contained 2 " + + "times, but must be contained at most once."); VoteBuilder.createWithNetworkStatusVersionLine( "directory-footer\nnetwork-status-version 3"); } @@ -800,7 +801,8 @@ public class RelayNetworkStatusVoteImplTest { public void testNetworkStatusVersionPrefixLinePoundChar() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line '#vote' in vote."); + this.thrown.expectMessage("Keyword 'network-status-version' " + + "must be contained in the first line."); VoteBuilder.createWithNetworkStatusVersionLine( "#vote\nnetwork-status-version 3"); } @@ -965,16 +967,6 @@ public class RelayNetworkStatusVoteImplTest { } @Test - public void testConsensusMethodTwoLines() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'consensus-method 1' in vote."); - VoteBuilder.createWithConsensusMethodsLine( - "consensus-method 1\nconsensus-method 1"); - } - - @Test public void testPublishedNoLine() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage("Keyword 'published' is contained 0 times, " @@ -1802,16 +1794,6 @@ public class RelayNetworkStatusVoteImplTest { } @Test - public void testUnrecognizedHeaderLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in vote."); - String unrecognizedLine = "unrecognized-line 1"; - VoteBuilder.createWithUnrecognizedHeaderLine(unrecognizedLine, true); - } - - @Test public void testUnrecognizedHeaderLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1823,17 +1805,6 @@ public class RelayNetworkStatusVoteImplTest { } @Test - public void testUnrecognizedDirSourceLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in vote."); - String unrecognizedLine = "unrecognized-line 1"; - VoteBuilder.createWithUnrecognizedDirSourceLine(unrecognizedLine, - true); - } - - @Test public void testUnrecognizedDirSourceLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1845,16 +1816,6 @@ public class RelayNetworkStatusVoteImplTest { } @Test - public void testUnrecognizedFooterLineFail() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown - .expectMessage("Unrecognized line 'unrecognized-line 1' in vote."); - String unrecognizedLine = "unrecognized-line 1"; - VoteBuilder.createWithUnrecognizedFooterLine(unrecognizedLine, true); - } - - @Test public void testUnrecognizedFooterLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; diff --git a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java index 5f369ad..67145c7 100644 --- a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java @@ -1136,7 +1136,7 @@ public class ServerDescriptorImplTest { public void testExitPolicyProjectAll() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage( - "Unrecognized line 'project *:*' in server descriptor."); + "Either keyword 'accept' or 'reject' must be contained at least once."); DescriptorBuilder.createWithExitPolicyLines("project *:*"); } @@ -1677,16 +1677,6 @@ public class ServerDescriptorImplTest { } @Test - public void testTunnelledDirServerTypo() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'tunneled-dir-server' " - + "in server descriptor."); - DescriptorBuilder.createWithTunnelledDirServerLine( - "tunneled-dir-server"); - } - - @Test public void testTunnelledDirServerTwice() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); @@ -1706,16 +1696,6 @@ public class ServerDescriptorImplTest { } @Test - public void testUnrecognizedLineFail() - throws DescriptorParseException { - String unrecognizedLine = "unrecognized-line 1"; - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage( - "Unrecognized line 'unrecognized-line 1' in server descriptor."); - DescriptorBuilder.createWithUnrecognizedLine(unrecognizedLine); - } - - @Test public void testUnrecognizedLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1880,17 +1860,6 @@ public class ServerDescriptorImplTest { + ROUTER_SIG_ED25519_LINE); } - @Test - public void testEd25519FollowedbyUnrecognizedLine() - throws DescriptorParseException { - this.thrown.expect(DescriptorParseException.class); - this.thrown.expectMessage("Unrecognized line 'unrecognized-line 1' " - + "in server descriptor."); - DescriptorBuilder.createWithEd25519Lines(IDENTITY_ED25519_LINES, - MASTER_KEY_ED25519_LINE, ROUTER_SIG_ED25519_LINE - + "\nunrecognized-line 1"); - } - private static final String ONION_KEY_CROSSCERT_LINES = "onion-key-crosscert\n" + "-----BEGIN CROSSCERT-----\n" diff --git a/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java b/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java index cf5efd5..48dcb98 100644 --- a/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java +++ b/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java @@ -12,6 +12,11 @@ public class TestDescriptor extends DescriptorImpl { throws DescriptorParseException { super(rawDescriptorBytes, offsetAndLength, null, blankLinesAllowed); } + + protected TestDescriptor(byte[] rawDescriptorBytes, int[] offsetAndLength) + throws DescriptorParseException { + super(rawDescriptorBytes, offsetAndLength, null); + } }

1 0

[metrics-lib/master] Tweak tests for removing deprecated code for #22154.
by karsten＠torproject.org 27 Jun '17

27 Jun '17

commit 1b3fd15f92d5fe5f71e62d8a76ee847134cd5e07 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Jun 26 22:12:10 2017 +0200 Tweak tests for removing deprecated code for #22154. This patch makes the following changes: - Remove now unused import in DescriptorParser. - Rename methods in DescriptorParserImpl to please checkstyle. - Tweak a few things in DescriptorReaderImplTest, including accepting varargs for dirs and putting back expected excluded files. - Remove tests that were previously testing deprecated methods which are now gone. - Change the two tests ending in NoDescriptors() back so that they really don't read any descriptors. - Put back tests for unrecognized lines, but don't expect DescriptorParseExceptions anymore and instead check getUnrecognizedLines(). --- .../torproject/descriptor/DescriptorParser.java | 1 - .../descriptor/impl/DescriptorParserImpl.java | 28 +++++++------- .../descriptor/impl/DescriptorParserImplTest.java | 5 +-- .../descriptor/impl/DescriptorReaderImplTest.java | 45 ++++++++-------------- .../impl/RelayNetworkStatusVoteImplTest.java | 42 +++++++++++++++++++- .../descriptor/impl/ServerDescriptorImplTest.java | 21 ++++++++++ 6 files changed, 93 insertions(+), 49 deletions(-) diff --git a/src/main/java/org/torproject/descriptor/DescriptorParser.java b/src/main/java/org/torproject/descriptor/DescriptorParser.java index 0ad2790..259510f 100644 --- a/src/main/java/org/torproject/descriptor/DescriptorParser.java +++ b/src/main/java/org/torproject/descriptor/DescriptorParser.java @@ -4,7 +4,6 @@ package org.torproject.descriptor; import java.io.File; -import java.util.List; /** * Descriptor source that parses descriptors from raw descriptor contents. diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java index d39343d..ac7ff9c 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java @@ -52,7 +52,7 @@ public class DescriptorParserImpl implements DescriptorParser { NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "3")) && firstLines.contains( NL + Key.VOTE_STATUS.keyword + SP + "consensus" + NL))) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.NETWORK_STATUS_VERSION, RelayNetworkStatusConsensusImpl.class); } else if (firstLines.startsWith("@type network-status-vote-3 1.") || ((firstLines.startsWith( @@ -61,7 +61,7 @@ public class DescriptorParserImpl implements DescriptorParser { NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL)) && firstLines.contains( NL + Key.VOTE_STATUS.keyword + SP + "vote" + NL))) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.NETWORK_STATUS_VERSION, RelayNetworkStatusVoteImpl.class); } else if (firstLines.startsWith("@type bridge-network-status 1.") || firstLines.startsWith(Key.R.keyword + SP)) { @@ -71,36 +71,36 @@ public class DescriptorParserImpl implements DescriptorParser { descriptorFile, fileName)); return parsedDescriptors; } else if (firstLines.startsWith("@type bridge-server-descriptor 1.")) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.ROUTER, BridgeServerDescriptorImpl.class); } else if (firstLines.startsWith("@type server-descriptor 1.") || firstLines.startsWith(Key.ROUTER.keyword + SP) || firstLines.contains(NL + Key.ROUTER.keyword + SP)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.ROUTER, RelayServerDescriptorImpl.class); } else if (firstLines.startsWith("@type bridge-extra-info 1.")) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.EXTRA_INFO, BridgeExtraInfoDescriptorImpl.class); } else if (firstLines.startsWith("@type extra-info 1.") || firstLines.startsWith(Key.EXTRA_INFO.keyword + SP) || firstLines.contains(NL + Key.EXTRA_INFO.keyword + SP)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.EXTRA_INFO, RelayExtraInfoDescriptorImpl.class); } else if (firstLines.startsWith("@type microdescriptor 1.") || firstLines.startsWith(Key.ONION_KEY.keyword + NL) || firstLines.contains(NL + Key.ONION_KEY.keyword + NL)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.ONION_KEY, MicrodescriptorImpl.class); } else if (firstLines.startsWith("@type bridge-pool-assignment 1.") || firstLines.startsWith(Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP) || firstLines.contains(NL + Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.BRIDGE_POOL_ASSIGNMENT, BridgePoolAssignmentImpl.class); } else if (firstLines.startsWith("@type dir-key-certificate-3 1.") || firstLines.startsWith(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP) || firstLines.contains( NL + Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.DIR_KEY_CERTIFICATE_VERSION, DirectoryKeyCertificateImpl.class); } else if (firstLines.startsWith("@type tordnsel 1.") || firstLines.startsWith("ExitNode" + SP) @@ -114,12 +114,12 @@ public class DescriptorParserImpl implements DescriptorParser { Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL) || firstLines.contains( NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.NETWORK_STATUS_VERSION, RelayNetworkStatusImpl.class); } else if (firstLines.startsWith("@type directory 1.") || firstLines.startsWith(Key.SIGNED_DIRECTORY.keyword + NL) || firstLines.contains(NL + Key.SIGNED_DIRECTORY.keyword + NL)) { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, + return this.parseOneOrMoreDescriptors(rawDescriptorBytes, descriptorFile, Key.SIGNED_DIRECTORY, RelayDirectoryImpl.class); } else if (firstLines.startsWith("@type torperf 1.")) { return TorperfResultImpl.parseTorperfResults(rawDescriptorBytes, @@ -130,7 +130,7 @@ public class DescriptorParserImpl implements DescriptorParser { } } - private List<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, + private List<Descriptor> parseOneOrMoreDescriptors(byte[] rawDescriptorBytes, File descriptorFile, Key key, Class<? extends DescriptorImpl> descriptorClass) throws DescriptorParseException { @@ -183,7 +183,7 @@ public class DescriptorParserImpl implements DescriptorParser { int[] offsetAndLength = new int[] { startAnnotations, endDescriptor - startAnnotations }; try { - parsedDescriptors.add(this.parseDescriptor(rawDescriptorBytes, + parsedDescriptors.add(this.parseOneDescriptor(rawDescriptorBytes, offsetAndLength, descriptorFile, constructor)); } catch (DescriptorParseException e) { parsedDescriptors.add(new UnparseableDescriptorImpl( @@ -194,7 +194,7 @@ public class DescriptorParserImpl implements DescriptorParser { return parsedDescriptors; } - Descriptor parseDescriptor(byte[] rawDescriptorBytes, + Descriptor parseOneDescriptor(byte[] rawDescriptorBytes, int[] offsetAndLength, File descriptorFile, Constructor<? extends DescriptorImpl> constructor) throws DescriptorParseException { diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java index df441f3..42d0cde 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java @@ -20,8 +20,7 @@ public class DescriptorParserImplTest { @Rule public ExpectedException thrown = ExpectedException.none(); - private TestDescriptor makeTestDesc(byte[] bytes) - throws Exception { + private TestDescriptor makeTestDesc(byte[] bytes) throws Exception { return new TestDescriptor(bytes, new int[]{0, bytes.length}); } @@ -61,7 +60,7 @@ public class DescriptorParserImplTest { this.thrown.expectMessage("'176x.158.53.63' in line 'router UbuntuCore169 " + "176x.158.53.63 44583 0 0' is not a valid IPv4 address."); DescriptorParserImpl dpi = new DescriptorParserImpl(); - dpi.parseDescriptor(DEFECT.getBytes(), + dpi.parseOneDescriptor(DEFECT.getBytes(), new int[]{0, DEFECT.getBytes().length}, null, constructor); } diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java index a00e5ba..0a3791e 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java @@ -63,9 +63,9 @@ public class DescriptorReaderImplTest { parseHistoryContents.getBytes(StandardCharsets.UTF_8)); } - private int readAllDescriptors(File dir) { + private int readAllDescriptors(File... dirs) { Iterator<Descriptor> descriptors = this.descriptorReader - .readDescriptors(dir).iterator(); + .readDescriptors(dirs).iterator(); int count = 0; while (descriptors.hasNext()) { count++; @@ -75,8 +75,10 @@ public class DescriptorReaderImplTest { } private void assertExcludedFilesParsedFilesAndHistoryFileLines( - int expectedParsedFiles, + int expectedExcludedFiles, int expectedParsedFiles, int expectedHistoryFileLines) throws IOException { + assertEquals(expectedExcludedFiles, + this.descriptorReader.getExcludedFiles().size()); assertEquals(expectedParsedFiles, this.descriptorReader.getParsedFiles().size()); assertEquals(expectedHistoryFileLines, @@ -87,38 +89,21 @@ public class DescriptorReaderImplTest { @Test public void testDescriptors() throws IOException { this.readAllDescriptors(this.inputDirectory); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 1); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 2, 1); } @Test public void testNoDescriptors() throws Exception { // calling readAllDescriptors fails with IAE Iterator<Descriptor> descriptors = this.descriptorReader - .readDescriptors(null).iterator(); + .readDescriptors().iterator(); int count = 0; while (descriptors.hasNext()) { count++; descriptors.next(); } assertEquals(count, 0); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 1); - } - - @Test - @SuppressWarnings("deprecation") - public void testSetExcludeFilesDescriptors() throws InterruptedException, - IOException { - this.readAllDescriptors(this.inputDirectory); - Thread.sleep(100L); /* It may take a moment to write the history file. */ - this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 1); - } - - @Test - public void testSetExcludeFilesNoDescriptors() throws InterruptedException, - IOException { - this.readAllDescriptors(this.inputDirectory); - Thread.sleep(100L); /* It may take a moment to write the history file. */ - this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 1); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 1); } @Test @@ -126,29 +111,29 @@ public class DescriptorReaderImplTest { this.descriptorReader.setHistoryFile(this.historyFile); this.readAllDescriptors(this.inputDirectory); descriptorReader.saveHistoryFile(this.historyFile); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 2); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 2); } @Test public void testSetHistoryFileNoDescriptors() throws IOException { this.descriptorReader.setHistoryFile(this.historyFile); - this.readAllDescriptors(this.inputDirectory); + this.readAllDescriptors(); this.descriptorReader.saveHistoryFile(this.historyFile); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 2); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 0); } @Test public void testSetExcludedFilesDescriptors() throws IOException { this.descriptorReader.setExcludedFiles(this.historyMap); this.readAllDescriptors(this.inputDirectory); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 1); } @Test public void testSetExcludedFilesNoDescriptors() throws IOException { this.descriptorReader.setExcludedFiles(this.historyMap); - this.readAllDescriptors(this.inputDirectory); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1); + this.readAllDescriptors(); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 1); } @Test @@ -159,7 +144,7 @@ public class DescriptorReaderImplTest { int count = this.readAllDescriptors(this.inputDirectory); assertEquals("Two files should have been parsed.", 2, count); descriptorReader.saveHistoryFile(this.historyFile); - this.assertExcludedFilesParsedFilesAndHistoryFileLines(2, 2); + this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 2, 2); } } diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java index 8dc89bd..aba55f4 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java @@ -962,6 +962,16 @@ public class RelayNetworkStatusVoteImplTest { } @Test + public void testConsensusMethodTwoLines() + throws DescriptorParseException { + String consensusMethodLine = "consensus-method 1"; + RelayNetworkStatusVote vote = VoteBuilder.createWithConsensusMethodsLine( + consensusMethodLine + "\n" + consensusMethodLine); + assertEquals(Arrays.asList(consensusMethodLine, consensusMethodLine), + vote.getUnrecognizedLines()); + } + + @Test public void testPublishedNoLine() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage("Keyword 'published' is contained 0 times, " @@ -1770,7 +1780,7 @@ public class RelayNetworkStatusVoteImplTest { assertEquals(signingKeyDigestSha1, secondSignature.getSigningKeyDigestSha1Hex()); assertEquals(signatureSha1 + "\n", secondSignature.getSignature()); - assertEquals(signingKeyDigestSha1, vote.getSignatures().get(0) + assertEquals(signingKeyDigestSha1, vote.getSignatures().get(1) .getSigningKeyDigestSha1Hex()); assertEquals("c0d58c8d3c3695526f6eb5c0d9f8452b2234d303", vote.getDigestSha1Hex()); @@ -1789,6 +1799,16 @@ public class RelayNetworkStatusVoteImplTest { } @Test + public void testUnrecognizedHeaderLineFail() + throws DescriptorParseException { + String unrecognizedLine = "unrecognized-line 1"; + RelayNetworkStatusVote vote = VoteBuilder.createWithUnrecognizedHeaderLine( + unrecognizedLine); + assertEquals(Arrays.asList(unrecognizedLine), + vote.getUnrecognizedLines()); + } + + @Test public void testUnrecognizedHeaderLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1800,6 +1820,16 @@ public class RelayNetworkStatusVoteImplTest { } @Test + public void testUnrecognizedDirSourceLineFail() + throws DescriptorParseException { + String unrecognizedLine = "unrecognized-line 1"; + RelayNetworkStatusVote vote = + VoteBuilder.createWithUnrecognizedDirSourceLine(unrecognizedLine); + assertEquals(Arrays.asList(unrecognizedLine), + vote.getUnrecognizedLines()); + } + + @Test public void testUnrecognizedDirSourceLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; @@ -1811,6 +1841,16 @@ public class RelayNetworkStatusVoteImplTest { } @Test + public void testUnrecognizedFooterLineFail() + throws DescriptorParseException { + String unrecognizedLine = "unrecognized-line 1"; + RelayNetworkStatusVote vote = VoteBuilder.createWithUnrecognizedFooterLine( + unrecognizedLine); + assertEquals(Arrays.asList(unrecognizedLine), + vote.getUnrecognizedLines()); + } + + @Test public void testUnrecognizedFooterLineIgnore() throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; diff --git a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java index 972e207..534cb0e 100644 --- a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java @@ -1675,6 +1675,16 @@ public class ServerDescriptorImplTest { } @Test + public void testTunnelledDirServerTypo() + throws DescriptorParseException { + String tunneledDirServerLine = "tunneled-dir-server"; + ServerDescriptor descriptor = DescriptorBuilder + .createWithTunnelledDirServerLine(tunneledDirServerLine); + assertEquals(Arrays.asList(tunneledDirServerLine), + descriptor.getUnrecognizedLines()); + } + + @Test public void testTunnelledDirServerTwice() throws DescriptorParseException { this.thrown.expect(DescriptorParseException.class); @@ -1858,6 +1868,17 @@ public class ServerDescriptorImplTest { + ROUTER_SIG_ED25519_LINE); } + @Test + public void testEd25519FollowedbyUnrecognizedLine() + throws DescriptorParseException { + String unrecognizedLine = "unrecognized-line 1"; + ServerDescriptor serverDecriptor = DescriptorBuilder.createWithEd25519Lines( + IDENTITY_ED25519_LINES, MASTER_KEY_ED25519_LINE, ROUTER_SIG_ED25519_LINE + + "\n" + unrecognizedLine); + assertEquals(Arrays.asList(unrecognizedLine), + serverDecriptor.getUnrecognizedLines()); + } + private static final String ONION_KEY_CROSSCERT_LINES = "onion-key-crosscert\n" + "-----BEGIN CROSSCERT-----\n"

1 0

[metrics-lib/master] Remove more deprecated code for #22154.
by karsten＠torproject.org 27 Jun '17

27 Jun '17

commit 946b261e6ca9a9c5115621db7314d4073d4bcb9b Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Jun 26 18:12:45 2017 +0200 Remove more deprecated code for #22154. This patch makes the following changes: - Remove (non-deprecated) method DescriptorParser#parseDescriptors(byte[], String) which we clearly forgot to deprecate earlier and which we shouldn't support in the future. It returns a List (rather than Iterable) and throws DescriptorParseException rather than include UnparseableDescriptor in the result. Might deserve an extra notice in the change log. - Put back a few lines `this.unrecognizedLines.add(line);` which got lost in previous commits. - Remove `failUnrecognizedDescriptorLines` parameters everywhere. - Remove more obsolete code from DescriptorReaderImpl. - Remove now obsolete attribute oldExitListEntries from ExitListImpl. --- .../torproject/descriptor/DescriptorParser.java | 10 -- .../descriptor/impl/BridgeNetworkStatusImpl.java | 1 + .../descriptor/impl/BridgePoolAssignmentImpl.java | 4 +- .../descriptor/impl/DescriptorParserImpl.java | 17 +-- .../descriptor/impl/DescriptorReaderImpl.java | 116 +-------------------- .../impl/DirectoryKeyCertificateImpl.java | 4 +- .../torproject/descriptor/impl/ExitListImpl.java | 3 +- .../impl/RelayNetworkStatusConsensusImpl.java | 1 + .../descriptor/impl/RelayNetworkStatusImpl.java | 3 +- .../descriptor/impl/DescriptorParserImplTest.java | 9 +- .../impl/RelayNetworkStatusImplTest.java | 4 +- .../impl/RelayNetworkStatusVoteImplTest.java | 21 ++-- .../descriptor/impl/ServerDescriptorImplTest.java | 6 +- 13 files changed, 34 insertions(+), 165 deletions(-) diff --git a/src/main/java/org/torproject/descriptor/DescriptorParser.java b/src/main/java/org/torproject/descriptor/DescriptorParser.java index 0082b5f..0ad2790 100644 --- a/src/main/java/org/torproject/descriptor/DescriptorParser.java +++ b/src/main/java/org/torproject/descriptor/DescriptorParser.java @@ -24,16 +24,6 @@ import java.util.List; public interface DescriptorParser { /** - * Parse descriptors in the given byte array, possibly parsing the - * publication time from the file name, depending on the descriptor - * type. - * - * @since 1.0.0 - */ - public List<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, - String fileName) throws DescriptorParseException; - - /** * Parse descriptors in the given byte array and return the parsed/unparseable * descriptors. * diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java index 01cd221..66a238e 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java @@ -89,6 +89,7 @@ public class BridgeNetworkStatusImpl extends NetworkStatusImpl if (this.unrecognizedLines == null) { this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } diff --git a/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java index 195690b..e0e5cd8 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java @@ -16,8 +16,8 @@ public class BridgePoolAssignmentImpl extends DescriptorImpl implements BridgePoolAssignment { protected BridgePoolAssignmentImpl(byte[] rawDescriptorBytes, - int[] offsetAndlength, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + int[] offsetAndlength, File descriptorFile) + throws DescriptorParseException { super(rawDescriptorBytes, offsetAndlength, descriptorFile, false); this.parseDescriptorBytes(); this.checkExactlyOnceKeys(EnumSet.of(Key.BRIDGE_POOL_ASSIGNMENT)); diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java index 320139d..d39343d 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java @@ -20,18 +20,11 @@ import java.util.List; public class DescriptorParserImpl implements DescriptorParser { @Override - public List<Descriptor> parseDescriptors( - byte[] rawDescriptorBytes, String fileName) - throws DescriptorParseException { - return this.parseDescriptors(rawDescriptorBytes, null, fileName, false); - } - - @Override public Iterable<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, File descriptorFile, String fileName) { try { - return this.parseDescriptors(rawDescriptorBytes, descriptorFile, fileName, - true); + return this.detectTypeAndParseDescriptors(rawDescriptorBytes, + descriptorFile, fileName); } catch (DescriptorParseException e) { /* Looks like we attempted to parse the whole raw descriptor bytes at once * below and ran into a parse issue. */ @@ -42,9 +35,9 @@ public class DescriptorParserImpl implements DescriptorParser { } } - private List<Descriptor> parseDescriptors( - byte[] rawDescriptorBytes, File descriptorFile, String fileName, - boolean includeUnparseableDescriptors) throws DescriptorParseException { + private List<Descriptor> detectTypeAndParseDescriptors( + byte[] rawDescriptorBytes, File descriptorFile, String fileName) + throws DescriptorParseException { byte[] first100Chars = new byte[Math.min(100, rawDescriptorBytes.length)]; System.arraycopy(rawDescriptorBytes, 0, first100Chars, 0, diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java index e2bf987..2d58573 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorReaderImpl.java @@ -4,7 +4,6 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.DescriptorParser; import org.torproject.descriptor.DescriptorReader; @@ -24,7 +23,6 @@ import java.io.FileInputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.nio.file.Files; -import java.util.ArrayList; import java.util.Arrays; import java.util.List; import java.util.Map; @@ -38,12 +36,6 @@ public class DescriptorReaderImpl implements DescriptorReader { DescriptorReaderImpl.class); private boolean hasStartedReading = false; - private List<File> directories = new ArrayList<>(); - - private List<File> tarballs = new ArrayList<>(); - - private File autoSaveHistoryFile; - private File manualSaveHistoryFile; @Override @@ -84,8 +76,6 @@ public class DescriptorReaderImpl implements DescriptorReader { return new TreeMap<>(this.reader.parsedFilesAfter); } - private Integer maxDescriptorFilesInQueue = null; - private int maxDescriptorsInQueue = 100; @Override @@ -108,8 +98,7 @@ public class DescriptorReaderImpl implements DescriptorReader { this.hasStartedReading = true; BlockingIteratorImpl<Descriptor> descriptorQueue = new BlockingIteratorImpl<>(this.maxDescriptorsInQueue); - this.reader = new DescriptorReaderRunnable( - descriptorFiles, descriptorQueue, this.autoSaveHistoryFile, + this.reader = new DescriptorReaderRunnable(descriptorFiles, descriptorQueue, this.manualSaveHistoryFile, this.excludedFiles); Thread readerThread = new Thread(this.reader); readerThread.setDaemon(true); @@ -130,10 +119,6 @@ public class DescriptorReaderImpl implements DescriptorReader { private File[] descriptorFiles; - private List<File> directories; - - private List<File> tarballs; - private BlockingIteratorImpl<Descriptor> descriptorQueue; private File autoSaveHistoryFile; @@ -146,16 +131,13 @@ public class DescriptorReaderImpl implements DescriptorReader { private SortedMap<String, Long> parsedFilesAfter = new TreeMap<>(); - private boolean failUnrecognizedDescriptorLines; - private DescriptorParser descriptorParser; private boolean hasFinishedReading = false; private DescriptorReaderRunnable(File[] descriptorFiles, BlockingIteratorImpl<Descriptor> descriptorQueue, - File autoSaveHistoryFile, File manualSaveHistoryFile, - SortedMap<String, Long> excludedFiles) { + File manualSaveHistoryFile, SortedMap<String, Long> excludedFiles) { this.descriptorFiles = descriptorFiles; this.descriptorQueue = descriptorQueue; this.autoSaveHistoryFile = autoSaveHistoryFile; @@ -163,7 +145,6 @@ public class DescriptorReaderImpl implements DescriptorReader { if (excludedFiles != null) { this.excludedFilesBefore = excludedFiles; } - this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; this.descriptorParser = new DescriptorParserImpl(); } @@ -172,8 +153,6 @@ public class DescriptorReaderImpl implements DescriptorReader { this.readOldHistory(this.autoSaveHistoryFile); this.readOldHistory(this.manualSaveHistoryFile); this.readDescriptorFiles(); - this.readDescriptors(); - this.readTarballs(); this.hasFinishedReading = true; } catch (Throwable t) { log.error("Bug: uncaught exception or error while " @@ -322,97 +301,6 @@ public class DescriptorReaderImpl implements DescriptorReader { this.descriptorQueue.add(descriptor); } } - - private void readDescriptors() { - if (null == this.directories) { - return; - } - for (File directory : this.directories) { - if (!directory.exists() || !directory.isDirectory()) { - continue; - } - Stack<File> files = new Stack<>(); - files.add(directory); - boolean abortReading = false; - while (!abortReading && !files.isEmpty()) { - File file = files.pop(); - if (file.isDirectory()) { - files.addAll(Arrays.asList(file.listFiles())); - } else if (file.getName().endsWith(".tar") - || file.getName().endsWith(".tar.bz2") - || file.getName().endsWith(".tar.xz")) { - this.tarballs.add(file); - } else { - String absolutePath = file.getAbsolutePath(); - long lastModifiedMillis = file.lastModified(); - if (this.excludedFilesBefore.containsKey(absolutePath) - && this.excludedFilesBefore.get(absolutePath) - == lastModifiedMillis) { - this.excludedFilesAfter.put(absolutePath, - lastModifiedMillis); - continue; - } - this.parsedFilesAfter.put(absolutePath, lastModifiedMillis); - } - } - } - } - - private void readTarballs() { - if (null == this.tarballs) { - return; - } - List<File> files = new ArrayList<>(this.tarballs); - boolean abortReading = false; - while (!abortReading && !files.isEmpty()) { - File tarball = files.remove(0); - if (!tarball.getName().endsWith(".tar") - && !tarball.getName().endsWith(".tar.bz2") - && !tarball.getName().endsWith(".tar.xz")) { - continue; - } - String absolutePath = tarball.getAbsolutePath(); - long lastModifiedMillis = tarball.lastModified(); - if (this.excludedFilesBefore.containsKey(absolutePath) - && this.excludedFilesBefore.get(absolutePath) - == lastModifiedMillis) { - this.excludedFilesAfter.put(absolutePath, lastModifiedMillis); - continue; - } - this.parsedFilesAfter.put(absolutePath, lastModifiedMillis); - try { - FileInputStream in = new FileInputStream(tarball); - if (in.available() > 0) { - TarArchiveInputStream tais = null; - if (tarball.getName().endsWith(".tar.bz2")) { - tais = new TarArchiveInputStream( - new BZip2CompressorInputStream(in)); - } else if (tarball.getName().endsWith(".tar.xz")) { - tais = new TarArchiveInputStream( - new XZCompressorInputStream(in)); - } else if (tarball.getName().endsWith(".tar")) { - tais = new TarArchiveInputStream(in); - } - BufferedInputStream bis = new BufferedInputStream(tais); - TarArchiveEntry tae = null; - while ((tae = tais.getNextTarEntry()) != null) { - if (tae.isDirectory()) { - continue; - } - } - } - } catch (IOException e) { - abortReading = true; - } - } - } - - private List<Descriptor> readFile(File file) throws IOException, - DescriptorParseException { - byte[] rawDescriptorBytes = Files.readAllBytes(file.toPath()); - return this.descriptorParser.parseDescriptors(rawDescriptorBytes, - file.getName()); - } } } diff --git a/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java b/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java index 59357d5..81ece3c 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java @@ -16,8 +16,8 @@ public class DirectoryKeyCertificateImpl extends DescriptorImpl implements DirectoryKeyCertificate { protected DirectoryKeyCertificateImpl(byte[] rawDescriptorBytes, - int[] offsetAndLength, File descriptorFile, - boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + int[] offsetAndLength, File descriptorFile) + throws DescriptorParseException { super(rawDescriptorBytes, offsetAndLength, descriptorFile, false); this.parseDescriptorBytes(); this.calculateDigestSha1Hex(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP, diff --git a/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java b/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java index d67ccd9..454efdc 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java @@ -89,6 +89,7 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { if (this.unrecognizedLines == null) { this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } /* Parse the last entry. */ @@ -117,8 +118,6 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { return this.downloadedMillis; } - private Set<ExitList.Entry> oldExitListEntries = new HashSet<>(); - private Set<ExitList.Entry> exitListEntries = new HashSet<>(); @Override diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java index 7c6f7e0..87ce59a 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java @@ -109,6 +109,7 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl if (this.unrecognizedLines == null) { this.unrecognizedLines = new ArrayList<>(); } + this.unrecognizedLines.add(line); } } } diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java index f77e903..d2be570 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java @@ -20,8 +20,7 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl implements RelayNetworkStatus { protected RelayNetworkStatusImpl(byte[] statusBytes, int[] offsetAndLength, - File descriptorFile, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { + File descriptorFile) throws DescriptorParseException { super(statusBytes, offsetAndLength, descriptorFile, false, true); Set<Key> exactlyOnceKeys = EnumSet.of( Key.NETWORK_STATUS_VERSION, Key.DIR_SOURCE, Key.FINGERPRINT, diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java index 61dc119..df441f3 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java @@ -5,6 +5,7 @@ package org.torproject.descriptor.impl; import static org.junit.Assert.assertEquals; +import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.DescriptorParseException; import org.junit.Rule; @@ -39,8 +40,12 @@ public class DescriptorParserImplTest { public void testAnnotation() throws Exception { TestDescriptor des = makeTestDesc(MICRO.getBytes()); DescriptorParserImpl dpi = new DescriptorParserImpl(); - assertEquals(1, - dpi.parseDescriptors(des.getRawDescriptorBytes(), "dummy.file").size()); + int parsedDescriptors = 0; + for (Descriptor d : dpi.parseDescriptors(des.getRawDescriptorBytes(), null, + "dummy.file")) { + parsedDescriptors++; + } + assertEquals(1, parsedDescriptors); } @Test diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java index 55ce6eb..a2328d3 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java @@ -50,14 +50,14 @@ public class RelayNetworkStatusImplTest { byte[] statusBytes = (validAnnotation + invalidHeader + validFooter) .getBytes(); new RelayNetworkStatusImpl(statusBytes, new int[] { 0, statusBytes.length }, - null, true); + null); } @Test public void testValidHeader() throws DescriptorParseException { byte[] statusBytes = validStatus.getBytes(); RelayNetworkStatusImpl rnsi = new RelayNetworkStatusImpl(statusBytes, - new int[] { 0, statusBytes.length }, null, true); + new int[] { 0, statusBytes.length }, null); assertEquals(rnsi.getContactLine(), "1024R/8D56913D Alex de Joode <adejoode(a)sabotage.org>"); } diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java index b0f53bd..8dc89bd 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java @@ -470,8 +470,7 @@ public class RelayNetworkStatusVoteImplTest { private String unrecognizedHeaderLine = null; protected static RelayNetworkStatusVote - createWithUnrecognizedHeaderLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedHeaderLine(String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedHeaderLine = line; @@ -481,8 +480,7 @@ public class RelayNetworkStatusVoteImplTest { private String unrecognizedDirSourceLine = null; protected static RelayNetworkStatusVote - createWithUnrecognizedDirSourceLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedDirSourceLine(String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedDirSourceLine = line; @@ -492,8 +490,7 @@ public class RelayNetworkStatusVoteImplTest { private String unrecognizedStatusEntryLine = null; protected static RelayNetworkStatusVote - createWithUnrecognizedStatusEntryLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedStatusEntryLine(String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedStatusEntryLine = line; @@ -503,8 +500,7 @@ public class RelayNetworkStatusVoteImplTest { private String unrecognizedFooterLine = null; protected static RelayNetworkStatusVote - createWithUnrecognizedFooterLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedFooterLine(String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedFooterLine = line; @@ -514,8 +510,7 @@ public class RelayNetworkStatusVoteImplTest { private String unrecognizedDirectorySignatureLine = null; protected static RelayNetworkStatusVote - createWithUnrecognizedDirectorySignatureLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedDirectorySignatureLine(String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedDirectorySignatureLine = line; @@ -1798,7 +1793,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusVote vote = VoteBuilder - .createWithUnrecognizedHeaderLine(unrecognizedLine, false); + .createWithUnrecognizedHeaderLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, vote.getUnrecognizedLines()); @@ -1809,7 +1804,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusVote vote = VoteBuilder - .createWithUnrecognizedDirSourceLine(unrecognizedLine, false); + .createWithUnrecognizedDirSourceLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, vote.getUnrecognizedLines()); @@ -1820,7 +1815,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusVote vote = VoteBuilder - .createWithUnrecognizedFooterLine(unrecognizedLine, false); + .createWithUnrecognizedFooterLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, vote.getUnrecognizedLines()); diff --git a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java index 67145c7..972e207 100644 --- a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java @@ -315,8 +315,7 @@ public class ServerDescriptorImplTest { private byte[] nonAsciiLineBytes = null; private static ServerDescriptor createWithNonAsciiLineBytes( - byte[] lineBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { + byte[] lineBytes) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.nonAsciiLineBytes = lineBytes; return db.buildDescriptor(); @@ -1558,8 +1557,7 @@ public class ServerDescriptorImplTest { 0x61, 0x6c, 0x6c, 0x6f, 0x77, 0x2d, // "allow-" 0x73, 0x69, 0x6e, 0x67, 0x6c, 0x65, 0x2d, // "single-" 0x68, 0x6f, 0x70, 0x2d, // "hop-" - 0x65, 0x78, 0x69, 0x74, 0x73 }, // "exits" (no newline) - false); + 0x65, 0x78, 0x69, 0x74, 0x73 }); // "exits" (no newline) } @Test

1 0

[metrics-lib/master] Make tests and benchmark compile; tests fail currently.
by karsten＠torproject.org 27 Jun '17

27 Jun '17

commit 5148de226742ac2d451db01dfc4f0233850ce303 Author: iwakeh <iwakeh(a)torproject.org> Date: Fri Jun 23 08:37:47 2017 +0000 Make tests and benchmark compile; tests fail currently. Part of task-22154. --- .../descriptor/DescriptorSourceFactoryTest.java | 15 +- .../descriptor/benchmark/MeasurePerformance.java | 160 ++++++++++----------- .../descriptor/impl/BridgeNetworkStatusTest.java | 25 ++-- .../descriptor/impl/ConsensusBuilder.java | 71 +++++---- .../descriptor/impl/DescriptorImplTest.java | 4 +- .../descriptor/impl/DescriptorParserImplTest.java | 4 +- .../descriptor/impl/DescriptorReaderImplTest.java | 38 ++--- .../descriptor/impl/ExitListImplTest.java | 22 +-- .../impl/ExtraInfoDescriptorImplTest.java | 70 +++++---- .../descriptor/impl/MicrodescriptorImplTest.java | 10 +- .../impl/RelayNetworkStatusConsensusImplTest.java | 52 ++++--- .../impl/RelayNetworkStatusVoteImplTest.java | 97 +++++++------ .../descriptor/impl/ServerDescriptorImplTest.java | 83 ++++++----- .../torproject/descriptor/impl/TestDescriptor.java | 5 +- .../descriptor/impl/TestServerDescriptor.java | 8 +- .../descriptor/impl/TorperfResultImplTest.java | 10 +- 16 files changed, 309 insertions(+), 365 deletions(-) diff --git a/src/test/java/org/torproject/descriptor/DescriptorSourceFactoryTest.java b/src/test/java/org/torproject/descriptor/DescriptorSourceFactoryTest.java index 69d4193..3a9d502 100644 --- a/src/test/java/org/torproject/descriptor/DescriptorSourceFactoryTest.java +++ b/src/test/java/org/torproject/descriptor/DescriptorSourceFactoryTest.java @@ -8,14 +8,11 @@ import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; import static org.torproject.descriptor.DescriptorSourceFactory.COLLECTOR_DEFAULT; import static org.torproject.descriptor.DescriptorSourceFactory.COLLECTOR_PROPERTY; -import static org.torproject.descriptor.DescriptorSourceFactory.DOWNLOADER_DEFAULT; -import static org.torproject.descriptor.DescriptorSourceFactory.DOWNLOADER_PROPERTY; import static org.torproject.descriptor.DescriptorSourceFactory.PARSER_DEFAULT; import static org.torproject.descriptor.DescriptorSourceFactory.PARSER_PROPERTY; import static org.torproject.descriptor.DescriptorSourceFactory.READER_DEFAULT; import static org.torproject.descriptor.DescriptorSourceFactory.READER_PROPERTY; -import org.torproject.descriptor.impl.DescriptorDownloaderImpl; import org.torproject.descriptor.impl.DescriptorParserImpl; import org.torproject.descriptor.impl.DescriptorReaderImpl; import org.torproject.descriptor.index.DescriptorIndexCollector; @@ -28,10 +25,10 @@ import java.lang.reflect.Method; public class DescriptorSourceFactoryTest { private static final String[] properties = new String[] { COLLECTOR_PROPERTY, - DOWNLOADER_PROPERTY, PARSER_PROPERTY, READER_PROPERTY }; + PARSER_PROPERTY, READER_PROPERTY }; private static final String[] defaults = new String[] { COLLECTOR_DEFAULT, - DOWNLOADER_DEFAULT, PARSER_DEFAULT, READER_DEFAULT }; + PARSER_DEFAULT, READER_DEFAULT }; @Test public void testDefaults() { @@ -39,9 +36,6 @@ public class DescriptorSourceFactoryTest { DescriptorCollector dc = DescriptorSourceFactory.createDescriptorCollector(); assertTrue(dc instanceof DescriptorIndexCollector); - DescriptorDownloader dd = - DescriptorSourceFactory.createDescriptorDownloader(); - assertTrue(dd instanceof DescriptorDownloaderImpl); DescriptorParser dp = DescriptorSourceFactory.createDescriptorParser(); assertTrue(dp instanceof DescriptorParserImpl); DescriptorReader dr = DescriptorSourceFactory.createDescriptorReader(); @@ -85,17 +79,12 @@ public class DescriptorSourceFactoryTest { public void testProperties() { setProperties(new String[] { "org.torproject.descriptor.DummyCollectorImplementation", - "org.torproject.descriptor.DummyDownloaderImplementation", "org.torproject.descriptor.DummyParserImplementation", "org.torproject.descriptor.DummyReaderImplementation" }); DescriptorCollector dc = DescriptorSourceFactory.createDescriptorCollector(); assertTrue(dc instanceof DummyCollectorImplementation); assertEquals(1, DummyCollectorImplementation.count); - DescriptorDownloader dd = - DescriptorSourceFactory.createDescriptorDownloader(); - assertTrue(dd instanceof DummyDownloaderImplementation); - assertEquals(1, DummyDownloaderImplementation.count); DescriptorParser dp = DescriptorSourceFactory.createDescriptorParser(); assertTrue(dp instanceof DummyParserImplementation); assertEquals(1, DummyParserImplementation.count); diff --git a/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java b/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java index a0c923c..83faf05 100644 --- a/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java +++ b/src/test/java/org/torproject/descriptor/benchmark/MeasurePerformance.java @@ -4,7 +4,7 @@ package org.torproject.descriptor.benchmark; import org.torproject.descriptor.Descriptor; -import org.torproject.descriptor.DescriptorFile; +import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.DescriptorReader; import org.torproject.descriptor.DescriptorSourceFactory; import org.torproject.descriptor.ExtraInfoDescriptor; @@ -94,23 +94,19 @@ public class MeasurePerformance { long countedServerDescriptors = 0; DescriptorReader descriptorReader = DescriptorSourceFactory.createDescriptorReader(); - descriptorReader.addTarball(tarballFileOrDirectory); - descriptorReader.addDirectory(tarballFileOrDirectory); - Iterator<DescriptorFile> descriptorFiles = - descriptorReader.readDescriptors(); - while (descriptorFiles.hasNext()) { - DescriptorFile descriptorFile = descriptorFiles.next(); - for (Descriptor descriptor : descriptorFile.getDescriptors()) { - if (!(descriptor instanceof ServerDescriptor)) { - continue; - } - ServerDescriptor serverDescriptor = (ServerDescriptor) descriptor; - sumAdvertisedBandwidth += (long) Math.min(Math.min( - serverDescriptor.getBandwidthRate(), - serverDescriptor.getBandwidthBurst()), - serverDescriptor.getBandwidthObserved()); - countedServerDescriptors++; + Iterator<Descriptor> descriptors = + descriptorReader.readDescriptors(tarballFileOrDirectory).iterator(); + while (descriptors.hasNext()) { + Descriptor descriptor = descriptors.next(); + if (!(descriptor instanceof ServerDescriptor)) { + continue; } + ServerDescriptor serverDescriptor = (ServerDescriptor) descriptor; + sumAdvertisedBandwidth += (long) Math.min(Math.min( + serverDescriptor.getBandwidthRate(), + serverDescriptor.getBandwidthBurst()), + serverDescriptor.getBandwidthObserved()); + countedServerDescriptors++; } long endedMillis = System.currentTimeMillis(); System.out.println("Ending measureAverageAdvertisedBandwidth"); @@ -132,24 +128,21 @@ public class MeasurePerformance { long countedExtraInfoDescriptors = 0; DescriptorReader descriptorReader = DescriptorSourceFactory.createDescriptorReader(); - descriptorReader.addTarball(tarballFile); - Iterator<DescriptorFile> descriptorFiles = - descriptorReader.readDescriptors(); - while (descriptorFiles.hasNext()) { - DescriptorFile descriptorFile = descriptorFiles.next(); - for (Descriptor descriptor : descriptorFile.getDescriptors()) { - if (!(descriptor instanceof ExtraInfoDescriptor)) { - continue; - } - ExtraInfoDescriptor extraInfoDescriptor = - (ExtraInfoDescriptor) descriptor; - SortedMap<String, Integer> dirreqV3Reqs = - extraInfoDescriptor.getDirreqV3Reqs(); - if (dirreqV3Reqs != null) { - countries.addAll(dirreqV3Reqs.keySet()); - } - countedExtraInfoDescriptors++; + Iterator<Descriptor> descriptors = + descriptorReader.readDescriptors(tarballFile).iterator(); + while (descriptors.hasNext()) { + Descriptor descriptor = descriptors.next(); + if (!(descriptor instanceof ExtraInfoDescriptor)) { + continue; } + ExtraInfoDescriptor extraInfoDescriptor = + (ExtraInfoDescriptor) descriptor; + SortedMap<String, Integer> dirreqV3Reqs = + extraInfoDescriptor.getDirreqV3Reqs(); + if (dirreqV3Reqs != null) { + countries.addAll(dirreqV3Reqs.keySet()); + } + countedExtraInfoDescriptors++; } long endedMillis = System.currentTimeMillis(); System.out.println("Ending measureCountriesV3Requests"); @@ -173,27 +166,23 @@ public class MeasurePerformance { long countedConsensuses = 0L; DescriptorReader descriptorReader = DescriptorSourceFactory.createDescriptorReader(); - descriptorReader.addTarball(tarballFileOrDirectory); - descriptorReader.addDirectory(tarballFileOrDirectory); - Iterator<DescriptorFile> descriptorFiles = - descriptorReader.readDescriptors(); - while (descriptorFiles.hasNext()) { - DescriptorFile descriptorFile = descriptorFiles.next(); - for (Descriptor descriptor : descriptorFile.getDescriptors()) { - if (!(descriptor instanceof RelayNetworkStatusConsensus)) { - continue; - } - RelayNetworkStatusConsensus consensus = - (RelayNetworkStatusConsensus) descriptor; - for (NetworkStatusEntry entry : - consensus.getStatusEntries().values()) { - if (entry.getFlags().contains("Exit")) { - totalRelaysWithExitFlag++; - } - totalRelays++; + Iterator<Descriptor> descriptors = + descriptorReader.readDescriptors(tarballFileOrDirectory).iterator(); + while (descriptors.hasNext()) { + Descriptor descriptor = descriptors.next(); + if (!(descriptor instanceof RelayNetworkStatusConsensus)) { + continue; + } + RelayNetworkStatusConsensus consensus = + (RelayNetworkStatusConsensus) descriptor; + for (NetworkStatusEntry entry : + consensus.getStatusEntries().values()) { + if (entry.getFlags().contains("Exit")) { + totalRelaysWithExitFlag++; } - countedConsensuses++; + totalRelays++; } + countedConsensuses++; } long endedMillis = System.currentTimeMillis(); System.out.println("Ending measureAverageRelaysExit"); @@ -220,45 +209,42 @@ public class MeasurePerformance { long countedMicrodescriptors = 0L; DescriptorReader descriptorReader = DescriptorSourceFactory.createDescriptorReader(); - descriptorReader.addTarball(tarballFile); - Iterator<DescriptorFile> descriptorFiles = - descriptorReader.readDescriptors(); - while (descriptorFiles.hasNext()) { - DescriptorFile descriptorFile = descriptorFiles.next(); - for (Descriptor descriptor : descriptorFile.getDescriptors()) { - if (!(descriptor instanceof Microdescriptor)) { - continue; - } - countedMicrodescriptors++; - Microdescriptor microdescriptor = - (Microdescriptor) descriptor; - String defaultPolicy = microdescriptor.getDefaultPolicy(); - if (defaultPolicy == null) { - continue; - } - boolean accept = "accept".equals( - microdescriptor.getDefaultPolicy()); - for (String ports : microdescriptor.getPortList().split(",")) { - if (ports.contains("-")) { - String[] parts = ports.split("-"); - int from = Integer.parseInt(parts[0]); - int to = Integer.parseInt(parts[1]); - if (from <= 80 && to >= 80) { - if (accept) { - totalRelaysWithExitFlag++; - } - } else if (to > 80) { - if (!accept) { - totalRelaysWithExitFlag++; - } - break; - } - } else if ("80".equals(ports)) { + Iterator<Descriptor> descriptors = + descriptorReader.readDescriptors(tarballFile).iterator(); + while (descriptors.hasNext()) { + Descriptor descriptor = descriptors.next(); + if (!(descriptor instanceof Microdescriptor)) { + continue; + } + countedMicrodescriptors++; + Microdescriptor microdescriptor = + (Microdescriptor) descriptor; + String defaultPolicy = microdescriptor.getDefaultPolicy(); + if (defaultPolicy == null) { + continue; + } + boolean accept = "accept".equals( + microdescriptor.getDefaultPolicy()); + for (String ports : microdescriptor.getPortList().split(",")) { + if (ports.contains("-")) { + String[] parts = ports.split("-"); + int from = Integer.parseInt(parts[0]); + int to = Integer.parseInt(parts[1]); + if (from <= 80 && to >= 80) { if (accept) { totalRelaysWithExitFlag++; } + } else if (to > 80) { + if (!accept) { + totalRelaysWithExitFlag++; + } break; } + } else if ("80".equals(ports)) { + if (accept) { + totalRelaysWithExitFlag++; + } + break; } } } diff --git a/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java b/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java index c44de6d..22d24fd 100644 --- a/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java +++ b/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java @@ -30,7 +30,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.fileName = fileName; - return sb.buildStatus(true); + return sb.buildStatus(); } private String publishedLine = "published 2015-11-21 17:39:36"; @@ -40,7 +40,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.publishedLine = line; - return sb.buildStatus(true); + return sb.buildStatus(); } private String flagThresholdsLine = "flag-thresholds " @@ -54,7 +54,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.flagThresholdsLine = line; - return sb.buildStatus(true); + return sb.buildStatus(); } private List<String> statusEntries = new ArrayList<>(); @@ -62,23 +62,21 @@ public class BridgeNetworkStatusTest { private String unrecognizedHeaderLine = null; protected static BridgeNetworkStatus - createWithUnrecognizedHeaderLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedHeaderLine(String line) throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.unrecognizedHeaderLine = line; - return sb.buildStatus(failUnrecognizedDescriptorLines); + return sb.buildStatus(); } private String unrecognizedStatusEntryLine = null; protected static BridgeNetworkStatus - createWithUnrecognizedStatusEntryLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedStatusEntryLine(String line) throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.unrecognizedStatusEntryLine = line; - return sb.buildStatus(failUnrecognizedDescriptorLines); + return sb.buildStatus(); } private StatusBuilder() { @@ -95,13 +93,10 @@ public class BridgeNetworkStatusTest { return sb.toString().getBytes(); } - private BridgeNetworkStatus buildStatus( - boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { + private BridgeNetworkStatus buildStatus() throws DescriptorParseException { byte[] statusBytes = this.buildStatusBytes(); return new BridgeNetworkStatusImpl(statusBytes, - new int[] { 0, statusBytes.length }, null, this.fileName, - failUnrecognizedDescriptorLines); + new int[] { 0, statusBytes.length }, null, this.fileName); } private void appendHeader(StringBuilder sb) { @@ -129,7 +124,7 @@ public class BridgeNetworkStatusTest { @Test public void testSampleStatus() throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); - BridgeNetworkStatus status = sb.buildStatus(true); + BridgeNetworkStatus status = sb.buildStatus(); assertEquals(1448127576000L, status.getPublishedMillis()); assertEquals(3105080L, status.getStableUptime()); assertEquals(2450615L, status.getStableMtbf()); diff --git a/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java b/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java index 71bcd94..cb8d0aa 100644 --- a/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java +++ b/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java @@ -20,7 +20,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.networkStatusVersionLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String voteStatusLine = "vote-status consensus"; @@ -30,7 +30,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.voteStatusLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String consensusMethodLine = "consensus-method 11"; @@ -40,7 +40,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.consensusMethodLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String validAfterLine = "valid-after 2011-11-30 09:00:00"; @@ -50,7 +50,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.validAfterLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String freshUntilLine = "fresh-until 2011-11-30 10:00:00"; @@ -60,7 +60,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.freshUntilLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String validUntilLine = "valid-until 2011-11-30 12:00:00"; @@ -70,7 +70,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.validUntilLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String votingDelayLine = "voting-delay 300 300"; @@ -80,7 +80,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.votingDelayLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } String clientVersionsLine = "client-versions 0.2.1.31," @@ -91,7 +91,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.clientVersionsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } String serverVersionsLine = "server-versions 0.2.1.31," @@ -102,7 +102,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.serverVersionsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String packageLines = null; @@ -112,7 +112,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.packageLines = lines; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String knownFlagsLine = "known-flags Authority BadExit Exit " @@ -123,7 +123,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.knownFlagsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String recommendedClientProtocolsLine = @@ -135,7 +135,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.recommendedClientProtocolsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String recommendedRelayProtocolsLine = @@ -147,7 +147,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.recommendedRelayProtocolsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String requiredClientProtocolsLine = @@ -159,7 +159,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.requiredClientProtocolsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String requiredRelayProtocolsLine = @@ -171,7 +171,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.requiredRelayProtocolsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String paramsLine = "params " @@ -185,7 +185,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.paramsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String sharedRandPreviousValueLine = @@ -197,7 +197,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.sharedRandPreviousValueLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String sharedRandCurrentValueLine = @@ -209,7 +209,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.sharedRandCurrentValueLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } List<String> dirSources = new ArrayList<>(); @@ -227,7 +227,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.directoryFooterLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String bandwidthWeightsLine = "bandwidth-weights Wbd=285 " @@ -244,7 +244,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.bandwidthWeightsLine = line; - return cb.buildConsensus(true); + return cb.buildConsensus(); } private List<String> directorySignatures = new ArrayList<>(); @@ -256,56 +256,51 @@ public class ConsensusBuilder { private String unrecognizedHeaderLine = null; protected static RelayNetworkStatusConsensus - createWithUnrecognizedHeaderLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedHeaderLine(String line) throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedHeaderLine = line; - return cb.buildConsensus(failUnrecognizedDescriptorLines); + return cb.buildConsensus(); } private String unrecognizedDirSourceLine = null; protected static RelayNetworkStatusConsensus - createWithUnrecognizedDirSourceLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedDirSourceLine(String line) throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedDirSourceLine = line; - return cb.buildConsensus(failUnrecognizedDescriptorLines); + return cb.buildConsensus(); } private String unrecognizedStatusEntryLine = null; protected static RelayNetworkStatusConsensus - createWithUnrecognizedStatusEntryLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedStatusEntryLine(String line) throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedStatusEntryLine = line; - return cb.buildConsensus(failUnrecognizedDescriptorLines); + return cb.buildConsensus(); } private String unrecognizedFooterLine = null; protected static RelayNetworkStatusConsensus - createWithUnrecognizedFooterLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedFooterLine(String line) throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedFooterLine = line; - return cb.buildConsensus(failUnrecognizedDescriptorLines); + return cb.buildConsensus(); } private String unrecognizedDirectorySignatureLine = null; protected static RelayNetworkStatusConsensus - createWithUnrecognizedDirectorySignatureLine(String line, - boolean failUnrecognizedDescriptorLines) + createWithUnrecognizedDirectorySignatureLine(String line) throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedDirectorySignatureLine = line; - return cb.buildConsensus(failUnrecognizedDescriptorLines); + return cb.buildConsensus(); } protected ConsensusBuilder() { @@ -355,13 +350,11 @@ public class ConsensusBuilder { return sb.toString().getBytes(); } - protected RelayNetworkStatusConsensus buildConsensus( - boolean failUnrecognizedDescriptorLines) + protected RelayNetworkStatusConsensus buildConsensus() throws DescriptorParseException { byte[] consensusBytes = this.buildConsensusBytes(); return new RelayNetworkStatusConsensusImpl(consensusBytes, - new int[] { 0, consensusBytes.length }, null, - failUnrecognizedDescriptorLines); + new int[] { 0, consensusBytes.length }, null); } private void appendHeader(StringBuilder sb) { diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java index 543254d..cc97d1d 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorImplTest.java @@ -3,6 +3,8 @@ package org.torproject.descriptor.impl; +import org.torproject.descriptor.DescriptorParseException; + import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNull; @@ -49,7 +51,7 @@ public class DescriptorImplTest { } private TestDescriptor makeTestDesc(byte[] bytes) throws Exception { - return new TestDescriptor(bytes, new int[]{0, bytes.length}, false, false); + return new TestDescriptor(bytes, new int[]{0, bytes.length}, false); } @Test diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java index 558a395..a2a3d9e 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorParserImplTest.java @@ -20,7 +20,7 @@ public class DescriptorParserImplTest { public ExpectedException thrown = ExpectedException.none(); private TestDescriptor makeTestDesc(byte[] bytes) throws Exception { - return new TestDescriptor(bytes, new int[]{0, bytes.length}, false, false); + return new TestDescriptor(bytes, new int[]{0, bytes.length}, false); } private static final String MICRO = @@ -57,7 +57,7 @@ public class DescriptorParserImplTest { + "176x.158.53.63 44583 0 0' is not a valid IPv4 address."); DescriptorParserImpl dpi = new DescriptorParserImpl(); dpi.parseDescriptor(DEFECT.getBytes(), - new int[]{0, DEFECT.getBytes().length}, null, constructor, false); + new int[]{0, DEFECT.getBytes().length}, null, constructor); } private static final String DEFECT = diff --git a/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java b/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java index 0d3bdcf..fa791a2 100644 --- a/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/DescriptorReaderImplTest.java @@ -5,7 +5,7 @@ package org.torproject.descriptor.impl; import static org.junit.Assert.assertEquals; -import org.torproject.descriptor.DescriptorFile; +import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.DescriptorReader; import org.junit.Before; @@ -63,13 +63,13 @@ public class DescriptorReaderImplTest { parseHistoryContents.getBytes(StandardCharsets.UTF_8)); } - private int readAllDescriptors() { - Iterator<DescriptorFile> descriptorFiles = - this.descriptorReader.readDescriptors(); + private int readAllDescriptors(File dir) { + Iterator<Descriptor> descriptors = this.descriptorReader + .readDescriptors(dir).iterator(); int count = 0; - while (descriptorFiles.hasNext()) { + while (descriptors.hasNext()) { count++; - descriptorFiles.next(); + descriptors.next(); } return count; } @@ -88,14 +88,13 @@ public class DescriptorReaderImplTest { @Test public void testDescriptors() throws IOException { - this.descriptorReader.addDirectory(this.inputDirectory); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 2, 1); } @Test public void testNoDescriptors() throws IOException { - this.readAllDescriptors(); + this.readAllDescriptors(null); this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 1); } @@ -103,19 +102,15 @@ public class DescriptorReaderImplTest { @SuppressWarnings("deprecation") public void testSetExcludeFilesDescriptors() throws InterruptedException, IOException { - this.descriptorReader.setExcludeFiles(this.historyFile); - this.descriptorReader.addDirectory(this.inputDirectory); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); Thread.sleep(100L); /* It may take a moment to write the history file. */ this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 2); } @Test - @SuppressWarnings("deprecation") public void testSetExcludeFilesNoDescriptors() throws InterruptedException, IOException { - this.descriptorReader.setExcludeFiles(this.historyFile); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); Thread.sleep(100L); /* It may take a moment to write the history file. */ this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 0); } @@ -123,8 +118,7 @@ public class DescriptorReaderImplTest { @Test public void testSetHistoryFileDescriptors() throws IOException { this.descriptorReader.setHistoryFile(this.historyFile); - this.descriptorReader.addDirectory(this.inputDirectory); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); descriptorReader.saveHistoryFile(this.historyFile); this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 2); } @@ -132,7 +126,7 @@ public class DescriptorReaderImplTest { @Test public void testSetHistoryFileNoDescriptors() throws IOException { this.descriptorReader.setHistoryFile(this.historyFile); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); this.descriptorReader.saveHistoryFile(this.historyFile); this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 0); } @@ -140,15 +134,14 @@ public class DescriptorReaderImplTest { @Test public void testSetExcludedFilesDescriptors() throws IOException { this.descriptorReader.setExcludedFiles(this.historyMap); - this.descriptorReader.addDirectory(this.inputDirectory); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); this.assertExcludedFilesParsedFilesAndHistoryFileLines(1, 1, 1); } @Test public void testSetExcludedFilesNoDescriptors() throws IOException { this.descriptorReader.setExcludedFiles(this.historyMap); - this.readAllDescriptors(); + this.readAllDescriptors(this.inputDirectory); this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 0, 1); } @@ -157,8 +150,7 @@ public class DescriptorReaderImplTest { File corruptHistoryFile = this.temporaryFolder.newFile("corruptHistory"); Files.write(corruptHistoryFile.toPath(),"1293145200000 \n ".getBytes()); this.descriptorReader.setHistoryFile(corruptHistoryFile); - this.descriptorReader.addDirectory(this.inputDirectory); - int count = this.readAllDescriptors(); + int count = this.readAllDescriptors(this.inputDirectory); assertEquals("Two files should have been parsed.", 2, count); descriptorReader.saveHistoryFile(this.historyFile); this.assertExcludedFilesParsedFilesAndHistoryFileLines(0, 2, 2); diff --git a/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java b/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java index 5f31341..dabea1e 100644 --- a/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ExitListImplTest.java @@ -7,7 +7,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertTrue; import org.torproject.descriptor.DescriptorParseException; -import org.torproject.descriptor.ExitListEntry; +import org.torproject.descriptor.ExitList; import org.junit.Rule; import org.junit.Test; @@ -24,7 +24,7 @@ public class ExitListImplTest { @Test public void testAnnotatedInput() throws Exception { ExitListImpl result = new ExitListImpl((tordnselAnnotation + input) - .getBytes("US-ASCII"), null, fileName, false); + .getBytes("US-ASCII"), null, fileName); assertEquals("Expected one annotation.", 1, result.getAnnotations().size()); assertEquals(tordnselAnnotation.substring(0, 18), @@ -32,8 +32,8 @@ public class ExitListImplTest { assertEquals(1441065722000L, result.getDownloadedMillis()); assertTrue("Unrecognized lines: " + result.getUnrecognizedLines(), result.getUnrecognizedLines().isEmpty()); - assertEquals("Found: " + result.getExitListEntries(), 7, - result.getExitListEntries().size()); + assertEquals("Found: " + result.getEntries(), 7, + result.getEntries().size()); assertEquals("Found: " + result.getEntries(), 5, result.getEntries().size()); } @@ -42,16 +42,16 @@ public class ExitListImplTest { public void testMultipleOldExitAddresses() throws Exception { ExitListImpl result = new ExitListImpl( (tordnselAnnotation + multiExitAddressInput) - .getBytes("US-ASCII"), null, fileName, false); + .getBytes("US-ASCII"), null, fileName); assertTrue("Unrecognized lines: " + result.getUnrecognizedLines(), result.getUnrecognizedLines().isEmpty()); - assertEquals("Found: " + result.getExitListEntries(), - 3, result.getExitListEntries().size()); + assertEquals("Found: " + result.getEntries(), + 3, result.getEntries().size()); Map<String, Long> testMap = new HashMap(); testMap.put("81.7.17.171", 1441044592000L); testMap.put("81.7.17.172", 1441044652000L); testMap.put("81.7.17.173", 1441044712000L); - for (ExitListEntry ele : result.getExitListEntries()) { + for (ExitList.Entry ele : result.getEntries()) { Map<String, Long> map = ele.getExitAddresses(); assertEquals("Found: " + map, 1, map.size()); Map.Entry<String, Long> ea = map.entrySet().iterator().next(); @@ -68,7 +68,7 @@ public class ExitListImplTest { public void testMultipleExitAddresses() throws Exception { ExitListImpl result = new ExitListImpl( (tordnselAnnotation + multiExitAddressInput) - .getBytes("US-ASCII"), null, fileName, false); + .getBytes("US-ASCII"), null, fileName); assertTrue("Unrecognized lines: " + result.getUnrecognizedLines(), result.getUnrecognizedLines().isEmpty()); Map<String, Long> map = result.getEntries() @@ -84,7 +84,7 @@ public class ExitListImplTest { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage("Missing 'ExitAddress' line in exit list entry."); new ExitListImpl((tordnselAnnotation + insufficientInput[0]) - .getBytes("US-ASCII"), null, fileName, false); + .getBytes("US-ASCII"), null, fileName); } @Test @@ -92,7 +92,7 @@ public class ExitListImplTest { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage("Missing 'Published' line in exit list entry."); new ExitListImpl((tordnselAnnotation + insufficientInput[1]) - .getBytes("US-ASCII"), null, fileName, false); + .getBytes("US-ASCII"), null, fileName); } private static final String tordnselAnnotation = "@type tordnsel 1.0\n"; diff --git a/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java index 82dd365..901aca8 100644 --- a/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java @@ -43,7 +43,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.extraInfoLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String publishedLine = "published 2012-02-11 09:08:36"; @@ -52,7 +52,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.publishedLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String writeHistoryLine = "write-history 2012-02-11 09:03:39 " @@ -62,7 +62,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.writeHistoryLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String readHistoryLine = "read-history 2012-02-11 09:03:39 " @@ -72,7 +72,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.readHistoryLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String dirreqWriteHistoryLine = "dirreq-write-history " @@ -83,7 +83,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.dirreqWriteHistoryLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String dirreqReadHistoryLine = "dirreq-read-history " @@ -94,7 +94,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.dirreqReadHistoryLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String geoipDbDigestLine = null; @@ -103,7 +103,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.geoipDbDigestLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String geoip6DbDigestLine = null; @@ -112,7 +112,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.geoip6DbDigestLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String geoipStatsLines = null; @@ -121,7 +121,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.geoipStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String dirreqStatsLines = null; @@ -130,7 +130,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.dirreqStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String entryStatsLines = null; @@ -139,7 +139,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.entryStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String cellStatsLines = null; @@ -148,7 +148,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.cellStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String connBiDirectLine = null; @@ -157,7 +157,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.connBiDirectLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String exitStatsLines = null; @@ -166,7 +166,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.exitStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String bridgeStatsLines = null; @@ -175,7 +175,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.bridgeStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String hidservStatsLines = null; @@ -184,7 +184,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.hidservStatsLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String paddingCountsLine = null; @@ -193,27 +193,27 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.paddingCountsLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String unrecognizedLine = null; private static ExtraInfoDescriptor createWithUnrecognizedLine( - String line, boolean failUnrecognizedDescriptorLines) + String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.unrecognizedLine = line; - return db.buildDescriptor(failUnrecognizedDescriptorLines); + return db.buildDescriptor(); } private byte[] nonAsciiLineBytes = null; private static ExtraInfoDescriptor createWithNonAsciiLineBytes( - byte[] lineBytes, boolean failUnrecognizedDescriptorLines) + byte[] lineBytes) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.nonAsciiLineBytes = lineBytes; - return db.buildDescriptor(failUnrecognizedDescriptorLines); + return db.buildDescriptor(); } private String routerSignatureLines = "router-signature\n" @@ -227,7 +227,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.routerSignatureLines = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String identityEd25519Lines = null; @@ -243,7 +243,7 @@ public class ExtraInfoDescriptorImplTest { db.identityEd25519Lines = identityEd25519Lines; db.masterKeyEd25519Line = masterKeyEd25519Line; db.routerSigEd25519Line = routerSigEd25519Line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private byte[] buildDescriptorBytes() { @@ -331,13 +331,11 @@ public class ExtraInfoDescriptorImplTest { return sb.toString().getBytes(); } - private ExtraInfoDescriptor buildDescriptor( - boolean failUnrecognizedDescriptorLines) + private ExtraInfoDescriptor buildDescriptor() throws DescriptorParseException { byte[] descriptorBytes = this.buildDescriptorBytes(); return new RelayExtraInfoDescriptorImpl(descriptorBytes, - new int[] { 0, descriptorBytes.length}, null, - failUnrecognizedDescriptorLines); + new int[] { 0, descriptorBytes.length}, null); } } @@ -954,7 +952,7 @@ public class ExtraInfoDescriptorImplTest { @Test public void testSampleDescriptor() throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); - ExtraInfoDescriptor descriptor = db.buildDescriptor(true); + ExtraInfoDescriptor descriptor = db.buildDescriptor(); assertEquals("chaoscomputerclub5", descriptor.getNickname()); assertEquals("A9C039A5FD02FCA06303DCFAABE25C5912C63B26", descriptor.getFingerprint()); @@ -1229,7 +1227,7 @@ public class ExtraInfoDescriptorImplTest { + "yblgrXtEgF3glaKv5ZvHhRREUI1t1c37SxparXSmYR4Q1yiK5zg4HE8eT9ILPRW9" + "3I5W/pZGQxL8Bu42dGjnAQ"); assertEquals("916A3CA8B7DF61473D5AE5B21711F35F301CE9E8", - descriptor.getGeoipDbDigest()); + descriptor.getGeoipDbDigestSha1Hex()); } @Test @@ -1249,7 +1247,7 @@ public class ExtraInfoDescriptorImplTest { + "yblgrXtEgF3glaKv5ZvHhRREUI1t1c37SxparXSmYR4Q1yiK5zg4HE8eT9ILPRW9" + "3I5W/pZGQxL8Bu42dGjnAQ"); assertEquals("916A3CA8B7DF61473D5AE5B21711F35F301CE9E8", - descriptor.getGeoip6DbDigest()); + descriptor.getGeoip6DbDigestSha1Hex()); } @Test @@ -1886,7 +1884,7 @@ public class ExtraInfoDescriptorImplTest { DescriptorBuilder.createWithNonAsciiLineBytes(new byte[] { 0x14, (byte) 0xfe, 0x18, // non-ascii chars 0x62, 0x72, 0x69, 0x64, 0x67, 0x65, 0x2d, // "bridge-" - 0x69, 0x70, 0x73 }, false); // "ips" (no newline) + 0x69, 0x70, 0x73 }); // "ips" (no newline) } @Test @@ -2167,7 +2165,7 @@ public class ExtraInfoDescriptorImplTest { this.thrown.expectMessage("Unrecognized line 'unrecognized-line 1' in " + "extra-info descriptor."); String unrecognizedLine = "unrecognized-line 1"; - DescriptorBuilder.createWithUnrecognizedLine(unrecognizedLine, true); + DescriptorBuilder.createWithUnrecognizedLine(unrecognizedLine); } @Test @@ -2175,7 +2173,7 @@ public class ExtraInfoDescriptorImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; ExtraInfoDescriptor descriptor = DescriptorBuilder - .createWithUnrecognizedLine(unrecognizedLine, false); + .createWithUnrecognizedLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, descriptor.getUnrecognizedLines()); @@ -2332,7 +2330,7 @@ public class ExtraInfoDescriptorImplTest { + "").getBytes(); RelayExtraInfoDescriptor descriptor = new RelayExtraInfoDescriptorImpl(descriptorBytes, - new int[] { 0, descriptorBytes.length }, null, true); + new int[] { 0, descriptorBytes.length }, null); assertEquals("Pt1BtzfRwhYqGCDo8jjchS8nJP3ovrDyHGn+dqPbMgw", descriptor.getDigestSha256Base64()); } @@ -2357,7 +2355,7 @@ public class ExtraInfoDescriptorImplTest { .getBytes(); BridgeExtraInfoDescriptor descriptor = new BridgeExtraInfoDescriptorImpl(descriptorBytes, - new int[] { 0, descriptorBytes.length }, null, true); + new int[] { 0, descriptorBytes.length }, null); assertEquals("TvrqpjI7OmCtwGwair/NHUxg5ROVVQYz6/EDyXsDHR4", descriptor.getDigestSha256Base64()); } @@ -2379,7 +2377,7 @@ public class ExtraInfoDescriptorImplTest { .getBytes(); BridgeExtraInfoDescriptor descriptor = new BridgeExtraInfoDescriptorImpl(descriptorBytes, - new int[] { 0, descriptorBytes.length }, null, true); + new int[] { 0, descriptorBytes.length }, null); assertNull(descriptor.getDigestSha1Hex()); assertNull(descriptor.getDigestSha256Base64()); } diff --git a/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java index f5c9131..a99895f 100644 --- a/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java @@ -32,7 +32,7 @@ public class MicrodescriptorImplTest { private static Microdescriptor createWithDefaultLines() throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String ntorOnionKeyLine = @@ -44,7 +44,7 @@ public class MicrodescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.idLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private byte[] buildDescriptorBytes() { @@ -61,13 +61,11 @@ public class MicrodescriptorImplTest { return sb.toString().getBytes(); } - private Microdescriptor buildDescriptor( - boolean failUnrecognizedDescriptorLines) + private Microdescriptor buildDescriptor() throws DescriptorParseException { byte[] descriptorBytes = this.buildDescriptorBytes(); return new MicrodescriptorImpl(descriptorBytes, - new int[] { 0, descriptorBytes.length }, null, - failUnrecognizedDescriptorLines); + new int[] { 0, descriptorBytes.length }, null); } } diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java index 4793e76..02b89e1 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java @@ -43,7 +43,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.dirSources.add(dirSourceString); - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String nickname = "gabelmoo"; @@ -156,7 +156,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(statusEntryString); - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String nickname = "right2privassy3"; @@ -314,7 +314,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.addDirectorySignature(directorySignatureString); - return cb.buildConsensus(true); + return cb.buildConsensus(); } private String identity = "ED03BB616EB2F60BEC80151114BB25CEF515B226"; @@ -355,7 +355,7 @@ public class RelayNetworkStatusConsensusImplTest { @Test public void testSampleConsensus() throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); - RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(); assertEquals(3, consensus.getNetworkStatusVersion()); assertEquals(11, consensus.getConsensusMethod()); assertEquals(1322643600000L, consensus.getValidAfterMillis()); @@ -742,7 +742,7 @@ public class RelayNetworkStatusConsensusImplTest { ConsensusBuilder cb = new ConsensusBuilder(); cb.clientVersionsLine = null; cb.serverVersionsLine = null; - RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(); assertNull(consensus.getRecommendedClientVersions()); assertNull(consensus.getRecommendedServerVersions()); } @@ -1362,7 +1362,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.sLine = sb.sLine + "\n" + sb.sLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - cb.buildConsensus(true); + cb.buildConsensus(); } @Test @@ -1373,7 +1373,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.prLine = sb.prLine + "\n" + sb.prLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - cb.buildConsensus(true); + cb.buildConsensus(); } @Test @@ -1403,7 +1403,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.wLine = sb.wLine + "\n" + sb.wLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - cb.buildConsensus(true); + cb.buildConsensus(); } @Test @@ -1412,7 +1412,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.wLine = "w Bandwidth=42424242 Unmeasured=1"; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(); for (NetworkStatusEntry s : consensus.getStatusEntries().values()) { if (s.getBandwidth() == 42424242L) { assertTrue(s.getUnmeasured()); @@ -1465,14 +1465,14 @@ public class RelayNetworkStatusConsensusImplTest { sb.pLine = sb.pLine + "\n" + sb.pLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - cb.buildConsensus(true); + cb.buildConsensus(); } @Test public void testNoStatusEntries() throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.clear(); - RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(); assertFalse(consensus.containsStatusEntry( "00795A6E8D91C270FC23B30F388A495553E01894")); } @@ -1498,7 +1498,7 @@ public class RelayNetworkStatusConsensusImplTest { cb.setBandwidthWeightsLine(null); /* This does not break, because directory footers were optional before * consensus method 9. */ - RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(); assertNull(consensus.getBandwidthWeights()); } @@ -1599,7 +1599,7 @@ public class RelayNetworkStatusConsensusImplTest { byte[] consensusBytes = cb.buildConsensusBytes(); consensusBytes[20] = (byte) 200; new RelayNetworkStatusConsensusImpl(consensusBytes, - new int[] { 0, consensusBytes.length }, null, true); + new int[] { 0, consensusBytes.length }, null); } @Test @@ -1613,7 +1613,7 @@ public class RelayNetworkStatusConsensusImplTest { byte[] consensusBytes = cb.buildConsensusBytes(); consensusBytes[0] = (byte) 200; new RelayNetworkStatusConsensusImpl(consensusBytes, - new int[] { 0, consensusBytes.length }, null, true); + new int[] { 0, consensusBytes.length }, null); } @Test @@ -1623,8 +1623,7 @@ public class RelayNetworkStatusConsensusImplTest { this.thrown.expectMessage( "Unrecognized line 'unrecognized-line 1' in consensus."); String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedHeaderLine(unrecognizedLine, - true); + ConsensusBuilder.createWithUnrecognizedHeaderLine(unrecognizedLine); } @Test @@ -1632,7 +1631,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusConsensus consensus = ConsensusBuilder - .createWithUnrecognizedHeaderLine(unrecognizedLine, false); + .createWithUnrecognizedHeaderLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, consensus.getUnrecognizedLines()); @@ -1645,8 +1644,7 @@ public class RelayNetworkStatusConsensusImplTest { this.thrown.expectMessage( "Unrecognized line 'unrecognized-line 1' in dir-source entry."); String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedDirSourceLine(unrecognizedLine, - true); + ConsensusBuilder.createWithUnrecognizedDirSourceLine(unrecognizedLine); } @Test @@ -1654,7 +1652,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusConsensus consensus = ConsensusBuilder - .createWithUnrecognizedDirSourceLine(unrecognizedLine, false); + .createWithUnrecognizedDirSourceLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, consensus.getUnrecognizedLines()); @@ -1668,7 +1666,7 @@ public class RelayNetworkStatusConsensusImplTest { + "status entry."); String unrecognizedLine = "unrecognized-line 1"; ConsensusBuilder.createWithUnrecognizedStatusEntryLine( - unrecognizedLine, true); + unrecognizedLine); } @Test @@ -1676,7 +1674,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusConsensus consensus = ConsensusBuilder - .createWithUnrecognizedStatusEntryLine(unrecognizedLine, false); + .createWithUnrecognizedStatusEntryLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, consensus.getUnrecognizedLines()); @@ -1689,8 +1687,7 @@ public class RelayNetworkStatusConsensusImplTest { this.thrown.expectMessage( "Unrecognized line 'unrecognized-line 1' in consensus."); String unrecognizedLine = "unrecognized-line 1"; - ConsensusBuilder.createWithUnrecognizedFooterLine(unrecognizedLine, - true); + ConsensusBuilder.createWithUnrecognizedFooterLine(unrecognizedLine); } @Test @@ -1698,7 +1695,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusConsensus consensus = ConsensusBuilder - .createWithUnrecognizedFooterLine(unrecognizedLine, false); + .createWithUnrecognizedFooterLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, consensus.getUnrecognizedLines()); @@ -1712,7 +1709,7 @@ public class RelayNetworkStatusConsensusImplTest { "Unrecognized line 'unrecognized-line 1' in dir-source entry."); String unrecognizedLine = "unrecognized-line 1"; ConsensusBuilder.createWithUnrecognizedDirectorySignatureLine( - unrecognizedLine, true); + unrecognizedLine); } @Test @@ -1720,8 +1717,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; RelayNetworkStatusConsensus consensus = ConsensusBuilder - .createWithUnrecognizedDirectorySignatureLine(unrecognizedLine, - false); + .createWithUnrecognizedDirectorySignatureLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, consensus.getUnrecognizedLines()); diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java index 9727f50..060cbbb 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java @@ -43,7 +43,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.networkStatusVersionLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String voteStatusLine = "vote-status vote"; @@ -53,7 +53,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.voteStatusLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String consensusMethodsLine = @@ -64,7 +64,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.consensusMethodsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String publishedLine = "published 2011-11-30 08:50:01"; @@ -74,7 +74,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.publishedLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String validAfterLine = "valid-after 2011-11-30 09:00:00"; @@ -84,7 +84,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.validAfterLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String freshUntilLine = "fresh-until 2011-11-30 10:00:00"; @@ -94,7 +94,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.freshUntilLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String validUntilLine = "valid-until 2011-11-30 12:00:00"; @@ -104,7 +104,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.validUntilLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String votingDelayLine = "voting-delay 300 300"; @@ -114,7 +114,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.votingDelayLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String clientVersionsLine = "client-versions 0.2.1.31," @@ -125,7 +125,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.clientVersionsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String serverVersionsLine = "server-versions 0.2.1.31," @@ -136,7 +136,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.serverVersionsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String packageLines = null; @@ -146,7 +146,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.packageLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private String knownFlagsLine = "known-flags Authority BadExit Exit " @@ -157,7 +157,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.knownFlagsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String recommendedClientProtocolsLine = @@ -169,7 +169,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.recommendedClientProtocolsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String recommendedRelayProtocolsLine = @@ -181,7 +181,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.recommendedRelayProtocolsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String requiredClientProtocolsLine = @@ -193,7 +193,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.requiredClientProtocolsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String requiredRelayProtocolsLine = @@ -205,7 +205,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.requiredRelayProtocolsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String flagThresholdsLine = "flag-thresholds " @@ -218,7 +218,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.flagThresholdsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String paramsLine = "params " @@ -232,7 +232,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.paramsLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String dirSourceLine = "dir-source urras " @@ -244,7 +244,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirSourceLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String contactLine = "contact 4096R/E012B42D Jacob Appelbaum " @@ -255,7 +255,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.contactLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String sharedRandParticipateLine = "shared-rand-participate"; @@ -264,7 +264,7 @@ public class RelayNetworkStatusVoteImplTest { String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandParticipateLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private List<String> sharedRandCommitLines = Arrays.asList(new String[] { @@ -279,7 +279,7 @@ public class RelayNetworkStatusVoteImplTest { List<String> lines) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandCommitLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private String sharedRandPreviousValueLine = @@ -290,7 +290,7 @@ public class RelayNetworkStatusVoteImplTest { String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandPreviousValueLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String sharedRandCurrentValueLine = @@ -301,7 +301,7 @@ public class RelayNetworkStatusVoteImplTest { String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandCurrentValueLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String legacyDirKeyLine = null; @@ -311,7 +311,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.legacyDirKeyLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String dirKeyCertificateVersionLine = @@ -322,7 +322,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyCertificateVersionLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String fingerprintLine = "fingerprint " @@ -333,7 +333,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.fingerprintLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String dirKeyPublishedLine = "dir-key-published 2011-04-27 " @@ -344,7 +344,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyPublishedLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String dirKeyExpiresLine = "dir-key-expires 2012-04-27 " @@ -355,7 +355,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyExpiresLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String dirIdentityKeyLines = "dir-identity-key\n" @@ -376,7 +376,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirIdentityKeyLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private String dirSigningKeyLines = "dir-signing-key\n" @@ -392,7 +392,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirSigningKeyLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private String dirKeyCrosscertLines = "dir-key-crosscert\n" @@ -407,7 +407,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyCrosscertLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private String dirKeyCertificationLines = "dir-key-certification\n" @@ -428,7 +428,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyCertificationLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private List<String> statusEntries = null; @@ -437,7 +437,7 @@ public class RelayNetworkStatusVoteImplTest { List<String> statusEntries) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.statusEntries = statusEntries; - return vb.buildVote(true); + return vb.buildVote(); } private String directoryFooterLine = "directory-footer"; @@ -447,7 +447,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.directoryFooterLine = line; - return vb.buildVote(true); + return vb.buildVote(); } private String directorySignatureLines = "directory-signature " @@ -464,7 +464,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.directorySignatureLines = lines; - return vb.buildVote(true); + return vb.buildVote(); } private String unrecognizedHeaderLine = null; @@ -475,7 +475,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedHeaderLine = line; - return vb.buildVote(failUnrecognizedDescriptorLines); + return vb.buildVote(); } private String unrecognizedDirSourceLine = null; @@ -486,7 +486,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedDirSourceLine = line; - return vb.buildVote(failUnrecognizedDescriptorLines); + return vb.buildVote(); } private String unrecognizedStatusEntryLine = null; @@ -497,7 +497,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedStatusEntryLine = line; - return vb.buildVote(failUnrecognizedDescriptorLines); + return vb.buildVote(); } private String unrecognizedFooterLine = null; @@ -508,7 +508,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedFooterLine = line; - return vb.buildVote(failUnrecognizedDescriptorLines); + return vb.buildVote(); } private String unrecognizedDirectorySignatureLine = null; @@ -519,7 +519,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedDirectorySignatureLine = line; - return vb.buildVote(failUnrecognizedDescriptorLines); + return vb.buildVote(); } private VoteBuilder() { @@ -552,13 +552,11 @@ public class RelayNetworkStatusVoteImplTest { return sb.toString().getBytes(); } - private RelayNetworkStatusVoteImpl buildVote( - boolean failUnrecognizedDescriptorLines) + private RelayNetworkStatusVoteImpl buildVote() throws DescriptorParseException { byte[] voteBytes = this.buildVoteBytes(); return new RelayNetworkStatusVoteImpl(voteBytes, - new int[] { 0, voteBytes.length }, null, - failUnrecognizedDescriptorLines); + new int[] { 0, voteBytes.length }, null); } private void appendHeader(StringBuilder sb) { @@ -705,7 +703,7 @@ public class RelayNetworkStatusVoteImplTest { @Test public void testSampleVote() throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); - RelayNetworkStatusVote vote = vb.buildVote(true); + RelayNetworkStatusVote vote = vb.buildVote(); assertEquals(3, vote.getNetworkStatusVersion()); List<Integer> consensusMethods = Arrays.asList( new Integer[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}); @@ -1220,7 +1218,7 @@ public class RelayNetworkStatusVoteImplTest { @Test public void testFlagThresholdsLine() throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); - RelayNetworkStatusVote vote = vb.buildVote(true); + RelayNetworkStatusVote vote = vb.buildVote(); assertEquals(693369L, vote.getStableUptime()); assertEquals(153249L, vote.getStableMtbf()); assertEquals(40960L, vote.getFastBandwidth()); @@ -1288,7 +1286,7 @@ public class RelayNetworkStatusVoteImplTest { VoteBuilder vb = new VoteBuilder(); vb.flagThresholdsLine = vb.flagThresholdsLine + "\n" + vb.flagThresholdsLine; - vb.buildVote(true); + vb.buildVote(); } @Test @@ -1785,7 +1783,8 @@ public class RelayNetworkStatusVoteImplTest { assertEquals(signingKeyDigestSha1, secondSignature.getSigningKeyDigestSha1Hex()); assertEquals(signatureSha1 + "\n", secondSignature.getSignature()); - assertEquals(signingKeyDigestSha1, vote.getSigningKeyDigest()); + assertEquals(signingKeyDigestSha1, vote.getSignatures().get(0) + .getSigningKeyDigestSha1Hex()); assertEquals("c0d58c8d3c3695526f6eb5c0d9f8452b2234d303", vote.getDigestSha1Hex()); } diff --git a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java index d0c2561..5f369ad 100644 --- a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java @@ -43,7 +43,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.routerLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String bandwidthLine = "bandwidth 51200 51200 53470"; @@ -52,7 +52,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.bandwidthLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String platformLine = "platform Tor 0.2.2.35 " @@ -62,7 +62,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.platformLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String publishedLine = "published 2012-01-01 04:03:19"; @@ -71,7 +71,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.publishedLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String fingerprintLine = "opt fingerprint D873 3048 FC8E " @@ -81,7 +81,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.fingerprintLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String hibernatingLine = null; @@ -90,7 +90,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.hibernatingLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String uptimeLine = "uptime 48"; @@ -99,7 +99,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.uptimeLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String onionKeyLines = "onion-key\n" @@ -114,7 +114,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.onionKeyLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String signingKeyLines = "signing-key\n" @@ -129,7 +129,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.signingKeyLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String onionKeyCrosscertLines = null; @@ -138,7 +138,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.onionKeyCrosscertLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String ntorOnionKeyCrosscertLines = null; @@ -147,7 +147,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.ntorOnionKeyCrosscertLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String exitPolicyLines = "reject *:*"; @@ -156,7 +156,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.exitPolicyLines = lines; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String contactLine = "contact Random Person <nobody AT " @@ -166,7 +166,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.contactLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String familyLine = null; @@ -175,7 +175,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.familyLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String readHistoryLine = null; @@ -184,7 +184,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.readHistoryLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String writeHistoryLine = null; @@ -193,7 +193,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.writeHistoryLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String eventdnsLine = null; @@ -202,7 +202,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.eventdnsLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String cachesExtraInfoLine = null; @@ -211,7 +211,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.cachesExtraInfoLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String extraInfoDigestLine = "opt extra-info-digest " @@ -221,7 +221,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.extraInfoDigestLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String hiddenServiceDirLine = "opt hidden-service-dir"; @@ -230,7 +230,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.hiddenServiceDirLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String protocolsLine = null; @@ -239,7 +239,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.protocolsLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String protoLine = "proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 " @@ -249,7 +249,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.protoLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String allowSingleHopExitsLine = null; @@ -259,7 +259,7 @@ public class ServerDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.allowSingleHopExitsLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String ipv6PolicyLine = null; @@ -268,7 +268,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.ipv6PolicyLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String ntorOnionKeyLine = null; @@ -277,7 +277,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.ntorOnionKeyLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String tunnelledDirServerLine = null; @@ -286,7 +286,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.tunnelledDirServerLine = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String routerSignatureLines = "router-signature\n" @@ -300,17 +300,16 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.routerSignatureLines = line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private String unrecognizedLine = null; - private static ServerDescriptor createWithUnrecognizedLine( - String line, boolean failUnrecognizedDescriptorLines) + private static ServerDescriptor createWithUnrecognizedLine(String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.unrecognizedLine = line; - return db.buildDescriptor(failUnrecognizedDescriptorLines); + return db.buildDescriptor(); } private byte[] nonAsciiLineBytes = null; @@ -320,7 +319,7 @@ public class ServerDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.nonAsciiLineBytes = lineBytes; - return db.buildDescriptor(failUnrecognizedDescriptorLines); + return db.buildDescriptor(); } private String identityEd25519Lines = null; @@ -336,7 +335,7 @@ public class ServerDescriptorImplTest { db.identityEd25519Lines = identityEd25519Lines; db.masterKeyEd25519Line = masterKeyEd25519Line; db.routerSigEd25519Line = routerSigEd25519Line; - return db.buildDescriptor(true); + return db.buildDescriptor(); } private byte[] buildDescriptorBytes() { @@ -451,20 +450,18 @@ public class ServerDescriptorImplTest { return sb.toString().getBytes(); } - private ServerDescriptorImpl buildDescriptor( - boolean failUnrecognizedDescriptorLines) + private ServerDescriptorImpl buildDescriptor() throws DescriptorParseException { byte[] descriptorBytes = this.buildDescriptorBytes(); return new RelayServerDescriptorImpl(descriptorBytes, - new int[] { 0, descriptorBytes.length }, null, - failUnrecognizedDescriptorLines); + new int[] { 0, descriptorBytes.length }, null); } } @Test public void testSampleDescriptor() throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); - ServerDescriptor descriptor = db.buildDescriptor(true); + ServerDescriptor descriptor = db.buildDescriptor(); assertEquals("saberrider2008", descriptor.getNickname()); assertEquals("94.134.192.243", descriptor.getAddress()); assertEquals(9001, (int) descriptor.getOrPort()); @@ -1715,7 +1712,7 @@ public class ServerDescriptorImplTest { this.thrown.expect(DescriptorParseException.class); this.thrown.expectMessage( "Unrecognized line 'unrecognized-line 1' in server descriptor."); - DescriptorBuilder.createWithUnrecognizedLine(unrecognizedLine, true); + DescriptorBuilder.createWithUnrecognizedLine(unrecognizedLine); } @Test @@ -1723,7 +1720,7 @@ public class ServerDescriptorImplTest { throws DescriptorParseException { String unrecognizedLine = "unrecognized-line 1"; ServerDescriptor descriptor = DescriptorBuilder - .createWithUnrecognizedLine(unrecognizedLine, false); + .createWithUnrecognizedLine(unrecognizedLine); List<String> unrecognizedLines = new ArrayList<>(); unrecognizedLines.add(unrecognizedLine); assertEquals(unrecognizedLines, descriptor.getUnrecognizedLines()); @@ -1746,7 +1743,7 @@ public class ServerDescriptorImplTest { sb.append("\n").append(line); } ServerDescriptor descriptor = DescriptorBuilder - .createWithUnrecognizedLine(sb.toString().substring(1), false); + .createWithUnrecognizedLine(sb.toString().substring(1)); assertEquals(unrecognizedLines, descriptor.getUnrecognizedLines()); } @@ -1767,7 +1764,7 @@ public class ServerDescriptorImplTest { sb.append("\n").append(line); } ServerDescriptor descriptor = DescriptorBuilder - .createWithUnrecognizedLine(sb.toString().substring(1), false); + .createWithUnrecognizedLine(sb.toString().substring(1)); assertEquals(unrecognizedLines, descriptor.getUnrecognizedLines()); } @@ -1963,7 +1960,7 @@ public class ServerDescriptorImplTest { + "\nrouter-digest " + digestSha1Hex; byte[] descriptorBytes = db.buildDescriptorBytes(); BridgeServerDescriptor descriptor = new BridgeServerDescriptorImpl( - descriptorBytes, new int[] { 0, descriptorBytes.length }, null, true); + descriptorBytes, new int[] { 0, descriptorBytes.length }, null); assertEquals(digestSha1Hex, descriptor.getDigestSha1Hex()); assertEquals(digestSha256Base64, descriptor.getDigestSha256Base64()); } @@ -1974,7 +1971,7 @@ public class ServerDescriptorImplTest { DescriptorBuilder db = new DescriptorBuilder(); byte[] descriptorBytes = db.buildDescriptorBytes(); BridgeServerDescriptor descriptor = new BridgeServerDescriptorImpl( - descriptorBytes, new int[] { 0, descriptorBytes.length }, null, true); + descriptorBytes, new int[] { 0, descriptorBytes.length }, null); assertNull(descriptor.getDigestSha1Hex()); assertNull(descriptor.getDigestSha256Base64()); } diff --git a/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java b/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java index e3a0c7c..cf5efd5 100644 --- a/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java +++ b/src/test/java/org/torproject/descriptor/impl/TestDescriptor.java @@ -8,10 +8,9 @@ import org.torproject.descriptor.DescriptorParseException; public class TestDescriptor extends DescriptorImpl { protected TestDescriptor(byte[] rawDescriptorBytes, int[] offsetAndLength, - boolean failUnrecognizedDescriptorLines, boolean blankLinesAllowed) + boolean blankLinesAllowed) throws DescriptorParseException { - super(rawDescriptorBytes, offsetAndLength, null, - failUnrecognizedDescriptorLines, blankLinesAllowed); + super(rawDescriptorBytes, offsetAndLength, null, blankLinesAllowed); } } diff --git a/src/test/java/org/torproject/descriptor/impl/TestServerDescriptor.java b/src/test/java/org/torproject/descriptor/impl/TestServerDescriptor.java index 0afd7b4..406e9c0 100644 --- a/src/test/java/org/torproject/descriptor/impl/TestServerDescriptor.java +++ b/src/test/java/org/torproject/descriptor/impl/TestServerDescriptor.java @@ -3,16 +3,16 @@ package org.torproject.descriptor.impl; +import org.torproject.descriptor.DescriptorParseException; + import java.io.File; public class TestServerDescriptor extends ServerDescriptorImpl { protected TestServerDescriptor(byte[] rawDescriptorBytes, - int[] offsetAndLength, File descriptorFile, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLength, File descriptorFile) throws DescriptorParseException { - super(rawDescriptorBytes, offsetAndLength, descriptorFile, - failUnrecognizedDescriptorLines); + super(rawDescriptorBytes, offsetAndLength, descriptorFile); } } diff --git a/src/test/java/org/torproject/descriptor/impl/TorperfResultImplTest.java b/src/test/java/org/torproject/descriptor/impl/TorperfResultImplTest.java index 7df4269..f648c7b 100644 --- a/src/test/java/org/torproject/descriptor/impl/TorperfResultImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/TorperfResultImplTest.java @@ -20,7 +20,7 @@ public class TorperfResultImplTest { public void testAnnotatedInput() throws Exception { TorperfResultImpl result = (TorperfResultImpl) (TorperfResultImpl.parseTorperfResults((torperfAnnotation + input) - .getBytes("US-ASCII"), null, false).get(0)); + .getBytes("US-ASCII"), null).get(0)); assertEquals("Expected one annotation.", 1, result.getAnnotations().size()); assertEquals(torperfAnnotation.substring(0, 17), @@ -37,7 +37,7 @@ public class TorperfResultImplTest { byte[] asciiBytes = (torperfAnnotation + input + input + input).getBytes("US-ASCII"); List<Descriptor> result = TorperfResultImpl.parseTorperfResults( - asciiBytes, null, false); + asciiBytes, null); assertEquals("Expected one annotation.", 1, ((TorperfResultImpl)(result.get(0))).getAnnotations().size()); assertEquals(3, result.size()); @@ -53,7 +53,7 @@ public class TorperfResultImplTest { + torperfAnnotation + input + torperfAnnotation + input).getBytes("US-ASCII"); List<Descriptor> result = TorperfResultImpl.parseTorperfResults( - asciiBytes, null, false); + asciiBytes, null); assertEquals("Expected one annotation.", 1, ((TorperfResultImpl)(result.get(0))).getAnnotations().size()); assertEquals(3, result.size()); @@ -90,7 +90,7 @@ public class TorperfResultImplTest { @Test public void testDatapercNonNumeric() throws Exception { List<Descriptor> result = TorperfResultImpl.parseTorperfResults( - ("DATAPERMILLE=2.0 " + input).getBytes(), null, false); + ("DATAPERMILLE=2.0 " + input).getBytes(), null); assertEquals(1, result.size()); TorperfResultImpl torperfResult = (TorperfResultImpl) result.get(0); assertEquals(1, torperfResult.getUnrecognizedKeys().size()); @@ -125,7 +125,7 @@ public class TorperfResultImplTest { public void testEndpointsHostnamesSourceAddress() throws DescriptorParseException { List<Descriptor> result = TorperfResultImpl.parseTorperfResults( - input2.getBytes(), null, true); + input2.getBytes(), null); assertEquals(1, result.size()); TorperfResultImpl torperfResult = (TorperfResultImpl) result.get(0); assertNull(torperfResult.getUnrecognizedKeys());

1 0

[tor/maint-0.3.1] changes file for bug 22719
by nickm＠torproject.org 27 Jun '17

27 Jun '17

commit 50be4600ae14687b9f4901b4d38a5dce644f5648 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jun 27 10:31:03 2017 -0400 changes file for bug 22719 --- changes/bug22719 | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/changes/bug22719 b/changes/bug22719 new file mode 100644 index 0000000..bfcda0a --- /dev/null +++ b/changes/bug22719 @@ -0,0 +1,7 @@ + o Minor bugfixes (compression): + - When spooling compressed data to an output buffer, don't try to + spool more data when there is no more data to spool and we are + not trying to flush the input. Previously, we would sometimes + launch compression requests with nothing to do, which interferes + with our 22672 checks. Fixes bug 22719; bugfix on 0.2.0.16-alpha. +

1 0

[tor/master] Try a little harder to make sure we never call tor_compress_process wrong.
by nickm＠torproject.org 27 Jun '17

27 Jun '17

commit b546d8bc2bfa0e20bdd14181a29a91ba31df2f0e Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 26 09:39:59 2017 -0400 Try a little harder to make sure we never call tor_compress_process wrong. --- src/or/buffers.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/or/buffers.c b/src/or/buffers.c index 3692ed4..12a6c02 100644 --- a/src/or/buffers.c +++ b/src/or/buffers.c @@ -2092,7 +2092,7 @@ fetch_from_buf_line(buf_t *buf, char *data_out, size_t *data_len) int write_to_buf_compress(buf_t *buf, tor_compress_state_t *state, const char *data, size_t data_len, - int done) + const int done) { char *next; size_t old_avail, avail; @@ -2114,8 +2114,10 @@ write_to_buf_compress(buf_t *buf, tor_compress_state_t *state, case TOR_COMPRESS_ERROR: return -1; case TOR_COMPRESS_OK: - if (data_len == 0) + if (data_len == 0) { + tor_assert_nonfatal(!done); over = 1; + } break; case TOR_COMPRESS_BUFFER_FULL: if (avail) { @@ -2124,6 +2126,11 @@ write_to_buf_compress(buf_t *buf, tor_compress_state_t *state, * whether were going to or not. */ need_new_chunk = 1; } + if (data_len == 0 && !done) { + /* We've consumed all the input data, though, so there's no + * point in forging ahead right now. */ + over = 1; + } break; } buf->datalen += old_avail - avail;

1 0

[tor/maint-0.3.1] Try a little harder to make sure we never call tor_compress_process wrong.
by nickm＠torproject.org 27 Jun '17

27 Jun '17

commit b546d8bc2bfa0e20bdd14181a29a91ba31df2f0e Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Jun 26 09:39:59 2017 -0400 Try a little harder to make sure we never call tor_compress_process wrong. --- src/or/buffers.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/or/buffers.c b/src/or/buffers.c index 3692ed4..12a6c02 100644 --- a/src/or/buffers.c +++ b/src/or/buffers.c @@ -2092,7 +2092,7 @@ fetch_from_buf_line(buf_t *buf, char *data_out, size_t *data_len) int write_to_buf_compress(buf_t *buf, tor_compress_state_t *state, const char *data, size_t data_len, - int done) + const int done) { char *next; size_t old_avail, avail; @@ -2114,8 +2114,10 @@ write_to_buf_compress(buf_t *buf, tor_compress_state_t *state, case TOR_COMPRESS_ERROR: return -1; case TOR_COMPRESS_OK: - if (data_len == 0) + if (data_len == 0) { + tor_assert_nonfatal(!done); over = 1; + } break; case TOR_COMPRESS_BUFFER_FULL: if (avail) { @@ -2124,6 +2126,11 @@ write_to_buf_compress(buf_t *buf, tor_compress_state_t *state, * whether were going to or not. */ need_new_chunk = 1; } + if (data_len == 0 && !done) { + /* We've consumed all the input data, though, so there's no + * point in forging ahead right now. */ + over = 1; + } break; } buf->datalen += old_avail - avail;

1 0