[tor-commits] [metrics-lib/master] Store raw descriptors as byte[], offset, and length.

commit a4d184bf949fe1d1524bf639c0b6c4e9d09fad73 Author: Karsten Loesing karsten.loesing@gmx.net Date: Thu Jun 1 00:35:15 2017 +0200

Store raw descriptors as byte[], offset, and length.

Prior to this commit we read raw descriptor bytes from disk, split them into serveral byte[] for each contained descriptor, and stored those copies together with descriptors. We further copied descriptor parts, like signatures or status entries, and stored those copies as well.

Overall, we temporarily required up to 3 times the size of descriptor files just to store raw descriptor contents: 1) the entire descriptor file read to memory, 2) copies of all contained descriptors, and 3) copies of contained descriptor parts. After moving on to the next descriptor file, 1) was freed, but 2) and 3) remained in memory. This was rather wasteful.

With this commit we store raw descriptors as reference to the byte[] containing the entire descriptor file plus offset and length of the part containing one descriptor. Similarly we store raw descriptor parts as a reference to the full descriptor plus offset and length of the descriptor part. This saves a lot of memory, and it avoids unnecessary array copying.

This change is also a step towards not storing raw descriptor contents in memory at all, but instead leaving contents on disk and accessing parts as needed. However, this commit does not take that step yet.

The original purpose of this commit was to prepare switching from the platform's default charset to UTF-8 for #21932. The idea was to reduce access to DescriptorImpl#rawDescriptorBytes and add all methods working on those bytes, including converting them to a String, to DescriptorImpl. This commit achieves this purpose by preparing that switch, yet it does not take that step, either. Switching to UTF-8 is midly backward-incompatible, so it'll have to wait until 2.0.0. However, switching will be much easier based on the changes in this commit.

Many of these changes in this commit are interdependent which makes it difficult to split up this commit with reasonable effort. Still, in order to facilitate reviews, here is an explanation of changes made in this commit from top to bottom:

Move all code for processing raw descriptor bytes from a) detecting the descriptor type, b) finding descriptor starts and ends, up to c) invoking the right DescriptorImpl subclass constructors from DescriptorImpl and its subclasses over to DescriptorParserImpl.

Include offset and limit in the constructors of DescriptorImpl and most of its subclasses.

Refer to directory and network status parts in RelayDirectoryImpl and NetworkStatusImpl and its subclasses by offset and length rather than passing copies of raw descriptors.

Provide two overloaded methods DescriptorImpl#newScanner() that internally handle the byte[]-to-String conversion rather than leaving this task to all DescriptorImpl subclasses.

In DescriptorImpl, rather than storing a copy of raw descriptor bytes per descriptor, store a reference to a potentially larger byte[], containing all descriptors read from a given file, together with offset and length.

Provide various methods in DescriptorImpl that provide access to raw descriptor bytes and that internally handle issues like unified character encoding.

Include an XXX21932 tag in all places where byte[] is currently converted to String using the platform's default charset.

Update existing methods in DescriptorImpl to only access rawDescriptorBytes within offset and offset + length.

In classes referenced from DescriptorImpl subclasses, like DirSourceEntryImpl and NetworkStatusEntryImpl, rather than storing a copy of raw descriptor bytes, store a reference to the parent DescriptorImpl instance together with offset and length.

Change raw descriptor bytes in ExitListEntryImpl into a String, because the byte[] we stored there was never read from disk but generated by ourselves using String#getBytes() using the platform's default charset. We also never used raw bytes in ExitListEntryImpl anyway. Admittedly, we could use offset and length there, too, but the amount of saved memory is likely not worth the necessary code changes.

Remove redundant zero-length checks from DescriptorImpl subclasses including ExitListImpl, NetworkStatusImpl, and RelayDirectoryImpl. These checks are redundant, because we already performed the same checks in DescriptorImpl#countKeys().

Move commonly used helper methods for finding the first index of a keyword or splitting descriptory by keyword from DescriptorImpl subclasses, like NetworkStatusImpl and RelayDirectoryImpl, to DescriptorImpl.

In test classes, replace the numerous invocations of DescriptorImpl subclass constructors with local buildSomething() methods, so that future changes to constructor signatures won't produce a diff as long as this one. --- CHANGELOG.md | 5 + .../impl/BridgeExtraInfoDescriptorImpl.java | 24 +- .../descriptor/impl/BridgeNetworkStatusImpl.java | 19 +- .../descriptor/impl/BridgePoolAssignmentImpl.java | 28 +- .../impl/BridgeServerDescriptorImpl.java | 26 +- .../torproject/descriptor/impl/DescriptorImpl.java | 344 ++++++++++----------- .../descriptor/impl/DescriptorParserImpl.java | 183 ++++++++++- .../descriptor/impl/DirSourceEntryImpl.java | 22 +- .../impl/DirectoryKeyCertificateImpl.java | 25 +- .../descriptor/impl/DirectorySignatureImpl.java | 16 +- .../descriptor/impl/ExitListEntryImpl.java | 12 +- .../torproject/descriptor/impl/ExitListImpl.java | 21 +- .../descriptor/impl/ExtraInfoDescriptorImpl.java | 8 +- .../descriptor/impl/MicrodescriptorImpl.java | 24 +- .../descriptor/impl/NetworkStatusEntryImpl.java | 26 +- .../descriptor/impl/NetworkStatusImpl.java | 156 +++------- .../descriptor/impl/RelayDirectoryImpl.java | 140 ++------- .../impl/RelayExtraInfoDescriptorImpl.java | 24 +- .../impl/RelayNetworkStatusConsensusImpl.java | 36 +-- .../descriptor/impl/RelayNetworkStatusImpl.java | 31 +- .../impl/RelayNetworkStatusVoteImpl.java | 29 +- .../descriptor/impl/RelayServerDescriptorImpl.java | 24 +- .../descriptor/impl/ServerDescriptorImpl.java | 8 +- .../descriptor/impl/TorperfResultImpl.java | 15 +- .../descriptor/impl/BridgeNetworkStatusTest.java | 29 +- .../descriptor/impl/ConsensusBuilder.java | 66 ++-- .../impl/ExtraInfoDescriptorImplTest.java | 64 ++-- .../descriptor/impl/MicrodescriptorImplTest.java | 15 +- .../impl/RelayNetworkStatusConsensusImplTest.java | 42 ++- .../impl/RelayNetworkStatusImplTest.java | 18 +- .../impl/RelayNetworkStatusVoteImplTest.java | 106 +++---- .../descriptor/impl/ServerDescriptorImplTest.java | 76 ++--- 32 files changed, 784 insertions(+), 878 deletions(-)

diff --git a/CHANGELOG.md b/CHANGELOG.md index 143494e..cf64603 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changes in version 1.8.0 - ??

+ * Medium changes + - Store raw descriptor contents as offset and length into a + referenced byte[], rather than copying contents into a separate + byte[] per descriptor. + * Minor changes - Turn keyword strings into enums and use the appropriate enum sets and maps to avoid repeating string literals and to use more speedy diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java index 0518392..7d0af52 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeExtraInfoDescriptorImpl.java @@ -5,34 +5,14 @@ package org.torproject.descriptor.impl;

import org.torproject.descriptor.BridgeExtraInfoDescriptor; import org.torproject.descriptor.DescriptorParseException; -import org.torproject.descriptor.ExtraInfoDescriptor; - -import java.util.ArrayList; -import java.util.List;

public class BridgeExtraInfoDescriptorImpl extends ExtraInfoDescriptorImpl implements BridgeExtraInfoDescriptor {

- protected static List<ExtraInfoDescriptor> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<ExtraInfoDescriptor> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DescriptorImpl.splitRawDescriptorBytes(descriptorsBytes, - Key.EXTRA_INFO.keyword + SP); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - ExtraInfoDescriptor parsedDescriptor = - new BridgeExtraInfoDescriptorImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - protected BridgeExtraInfoDescriptorImpl(byte[] descriptorBytes, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLimit, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines); + super(descriptorBytes, offsetAndLimit, failUnrecognizedDescriptorLines); } }

diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java index 6036e50..83a3a8a 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeNetworkStatusImpl.java @@ -18,10 +18,11 @@ import java.util.TimeZone; public class BridgeNetworkStatusImpl extends NetworkStatusImpl implements BridgeNetworkStatus {

- protected BridgeNetworkStatusImpl(byte[] statusBytes, - String fileName, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - super(statusBytes, failUnrecognizedDescriptorLines, false, false); + protected BridgeNetworkStatusImpl(byte[] rawDescriptorBytes, + int[] offsetAndLength, String fileName, + boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + super(rawDescriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines, + false, false); this.setPublishedMillisFromFileName(fileName); }

@@ -55,7 +56,7 @@ public class BridgeNetworkStatusImpl extends NetworkStatusImpl } }

- protected void parseHeader(byte[] headerBytes) + protected void parseHeader(int offset, int length) throws DescriptorParseException { /* Initialize flag-thresholds values here for the case that the status * doesn't contain those values. Initializing them in the constructor @@ -71,7 +72,7 @@ public class BridgeNetworkStatusImpl extends NetworkStatusImpl this.enoughMtbfInfo = -1; this.ignoringAdvertisedBws = -1;

- Scanner scanner = new Scanner(new String(headerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); while (scanner.hasNext()) { String line = scanner.next(); String[] parts = line.split("[ \t]+"); @@ -154,19 +155,19 @@ public class BridgeNetworkStatusImpl extends NetworkStatusImpl } }

- protected void parseDirSource(byte[] dirSourceBytes) + protected void parseDirSource(int offset, int length) throws DescriptorParseException { throw new DescriptorParseException("No directory source expected in " + "bridge network status."); }

- protected void parseFooter(byte[] footerBytes) + protected void parseFooter(int offset, int length) throws DescriptorParseException { throw new DescriptorParseException("No directory footer expected in " + "bridge network status."); }

- protected void parseDirectorySignature(byte[] directorySignatureBytes) + protected void parseDirectorySignature(int offset, int length) throws DescriptorParseException { throw new DescriptorParseException("No directory signature expected " + "in bridge network status."); diff --git a/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java index 609797e..ca07c29 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgePoolAssignmentImpl.java @@ -6,9 +6,7 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.BridgePoolAssignment; import org.torproject.descriptor.DescriptorParseException;

-import java.util.ArrayList; import java.util.EnumSet; -import java.util.List; import java.util.Scanner; import java.util.SortedMap; import java.util.TreeMap; @@ -16,26 +14,11 @@ import java.util.TreeMap; public class BridgePoolAssignmentImpl extends DescriptorImpl implements BridgePoolAssignment {

- protected static List<BridgePoolAssignment> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) + protected BridgePoolAssignmentImpl(byte[] rawDescriptorBytes, + int[] offsetAndlength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - List<BridgePoolAssignment> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DescriptorImpl.splitRawDescriptorBytes(descriptorsBytes, - Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - BridgePoolAssignment parsedDescriptor = - new BridgePoolAssignmentImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - - protected BridgePoolAssignmentImpl(byte[] descriptorBytes, - boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines, false); + super(rawDescriptorBytes, offsetAndlength, failUnrecognizedDescriptorLines, + false); this.parseDescriptorBytes(); this.checkExactlyOnceKeys(EnumSet.of(Key.BRIDGE_POOL_ASSIGNMENT)); this.checkFirstKey(Key.BRIDGE_POOL_ASSIGNMENT); @@ -44,8 +27,7 @@ public class BridgePoolAssignmentImpl extends DescriptorImpl }

private void parseDescriptorBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.rawDescriptorBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner().useDelimiter(NL); while (scanner.hasNext()) { String line = scanner.next(); if (line.startsWith(Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP)) { diff --git a/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java index 900f6cd..ac45591 100644 --- a/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/BridgeServerDescriptorImpl.java @@ -5,34 +5,14 @@ package org.torproject.descriptor.impl;

import org.torproject.descriptor.BridgeServerDescriptor; import org.torproject.descriptor.DescriptorParseException; -import org.torproject.descriptor.ServerDescriptor; - -import java.util.ArrayList; -import java.util.List;

public class BridgeServerDescriptorImpl extends ServerDescriptorImpl implements BridgeServerDescriptor {

- protected static List<ServerDescriptor> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<ServerDescriptor> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DescriptorImpl.splitRawDescriptorBytes(descriptorsBytes, - Key.ROUTER.keyword + SP); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - ServerDescriptor parsedDescriptor = - new BridgeServerDescriptorImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - - protected BridgeServerDescriptorImpl(byte[] descriptorBytes, - boolean failUnrecognizedDescriptorLines) + protected BridgeServerDescriptorImpl(byte[] rawDescriptorBytes, + int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines); + super(rawDescriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines); } }

diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java index 8c9c315..79905c0 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorImpl.java @@ -6,7 +6,7 @@ package org.torproject.descriptor.impl; import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.DescriptorParseException;

-import java.io.UnsupportedEncodingException; +import java.io.ByteArrayInputStream; import java.nio.charset.StandardCharsets; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; @@ -25,162 +25,146 @@ public abstract class DescriptorImpl implements Descriptor {

public static final String SP = " ";

- protected static List<Descriptor> parseDescriptors( - byte[] rawDescriptorBytes, String fileName, - boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<Descriptor> parsedDescriptors = new ArrayList<>(); - if (rawDescriptorBytes == null) { - return parsedDescriptors; + protected byte[] rawDescriptorBytes; + + /** + * The index of the first byte of this descriptor in + * {@link #rawDescriptorBytes} which may contain more than just one + * descriptor. + */ + protected int offset; + + /** + * The number of bytes of this descriptor in {@link #rawDescriptorBytes} which + * may contain more than just one descriptor. + */ + protected int length; + + /** + * Returns a <emph>copy</emph> of the full raw descriptor bytes. + * + * <p>If possible, subclasses should avoid retrieving raw descriptor bytes and + * converting them to a String themselves and instead rely on + * {@link #newScanner()} and related methods to parse the descriptor.</p> + * + * @return Copy of the full raw descriptor bytes. + */ + @Override + public byte[] getRawDescriptorBytes() { + return this.getRawDescriptorBytes(this.offset, this.length); + } + + /** + * Returns a <emph>copy</emph> of raw descriptor bytes starting at + * <code>offset</code> and containing <code>length</code> bytes. + * + * <p>If possible, subclasses should avoid retrieving raw descriptor bytes and + * converting them to a String themselves and instead rely on + * {@link #newScanner()} and related methods to parse the descriptor.</p> + * + * @param offset The index of the first byte to include. + * @param length The number of bytes to include. + * @return Copy of the given raw descriptor bytes. + */ + protected byte[] getRawDescriptorBytes(int offset, int length) { + if (offset < this.offset || offset + length > this.offset + this.length + || length < 0) { + throw new IndexOutOfBoundsException("offset=" + offset + " length=" + + length + " this.offset=" + this.offset + " this.length=" + + this.length); + } + byte[] result = new byte[length]; + System.arraycopy(this.rawDescriptorBytes, offset, result, 0, length); + return result; + } + + /** + * Returns a new {@link Scanner} for parsing the full raw descriptor starting + * using the platform's default charset. + * + * @return Scanner for the full raw descriptor bytes. + */ + protected Scanner newScanner() { + return this.newScanner(this.offset, this.length); + } + + /** + * Returns a new {@link Scanner} for parsing the raw descriptor starting at + * byte <code>offset</code> containing <code>length</code> bytes using the + * platform's default charset. + * + * @param offset The index of the first byte to parse. + * @param length The number of bytes to parse. + * @return Scanner for the given raw descriptor bytes. + */ + protected Scanner newScanner(int offset, int length) { + /* XXX21932 */ + return new Scanner(new ByteArrayInputStream(this.rawDescriptorBytes, offset, + length)); + } + + /** + * Returns the index within the raw descriptor of the first occurrence of the + * given <code>key</code>, or <code>-1</code> if the key is not contained. + * + * @param key Key to search for. + * @return Index of the first occurrence, or -1. + */ + protected int findFirstIndexOfKey(Key key) { + String ascii = new String(this.rawDescriptorBytes, this.offset, this.length, + StandardCharsets.US_ASCII); + if (ascii.startsWith(key.keyword + SP) + || ascii.startsWith(key.keyword + NL)) { + return this.offset; } - byte[] first100Chars = new byte[Math.min(100, - rawDescriptorBytes.length)]; - System.arraycopy(rawDescriptorBytes, 0, first100Chars, 0, - first100Chars.length); - String firstLines = new String(first100Chars); - if (firstLines.startsWith("@type network-status-consensus-3 1.") - || firstLines.startsWith( - "@type network-status-microdesc-consensus-3 1.") - || ((firstLines.startsWith( - Key.NETWORK_STATUS_VERSION.keyword + SP + "3") - || firstLines.contains( - NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "3")) - && firstLines.contains( - NL + Key.VOTE_STATUS.keyword + SP + "consensus" + NL))) { - parsedDescriptors.addAll(RelayNetworkStatusConsensusImpl - .parseConsensuses(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type network-status-vote-3 1.") - || ((firstLines.startsWith( - Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL) - || firstLines.contains( - NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL)) - && firstLines.contains( - NL + Key.VOTE_STATUS.keyword + SP + "vote" + NL))) { - parsedDescriptors.addAll(RelayNetworkStatusVoteImpl - .parseVotes(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type bridge-network-status 1.") - || firstLines.startsWith(Key.R.keyword + SP)) { - parsedDescriptors.add(new BridgeNetworkStatusImpl( - rawDescriptorBytes, fileName, failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type bridge-server-descriptor 1.")) { - parsedDescriptors.addAll(BridgeServerDescriptorImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type server-descriptor 1.") - || firstLines.startsWith(Key.ROUTER.keyword + SP) - || firstLines.contains(NL + Key.ROUTER.keyword + SP)) { - parsedDescriptors.addAll(RelayServerDescriptorImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type bridge-extra-info 1.")) { - parsedDescriptors.addAll(BridgeExtraInfoDescriptorImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type extra-info 1.") - || firstLines.startsWith(Key.EXTRA_INFO.keyword + SP) - || firstLines.contains(NL + Key.EXTRA_INFO.keyword + SP)) { - parsedDescriptors.addAll(RelayExtraInfoDescriptorImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type microdescriptor 1.") - || firstLines.startsWith(Key.ONION_KEY.keyword + NL) - || firstLines.contains(NL + Key.ONION_KEY.keyword + NL)) { - parsedDescriptors.addAll(MicrodescriptorImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type bridge-pool-assignment 1.") - || firstLines.startsWith(Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP) - || firstLines.contains(NL + Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP)) { - parsedDescriptors.addAll(BridgePoolAssignmentImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type dir-key-certificate-3 1.") - || firstLines.startsWith(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP) - || firstLines.contains( - NL + Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP)) { - parsedDescriptors.addAll(DirectoryKeyCertificateImpl - .parseDescriptors(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type tordnsel 1.") - || firstLines.startsWith("ExitNode" + SP) - || firstLines.contains(NL + "ExitNode" + SP)) { - parsedDescriptors.add(new ExitListImpl(rawDescriptorBytes, fileName, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type network-status-2 1.") - || firstLines.startsWith( - Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL) - || firstLines.contains( - NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL)) { - parsedDescriptors.add(new RelayNetworkStatusImpl(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type directory 1.") - || firstLines.startsWith(Key.SIGNED_DIRECTORY.keyword + NL) - || firstLines.contains(NL + Key.SIGNED_DIRECTORY.keyword + NL)) { - parsedDescriptors.add(new RelayDirectoryImpl(rawDescriptorBytes, - failUnrecognizedDescriptorLines)); - } else if (firstLines.startsWith("@type torperf 1.")) { - parsedDescriptors.addAll(TorperfResultImpl.parseTorperfResults( - rawDescriptorBytes, failUnrecognizedDescriptorLines)); + int keywordIndex = ascii.indexOf(NL + key.keyword + SP); + if (keywordIndex < 0) { + keywordIndex = ascii.indexOf(NL + key.keyword + NL); + } + if (keywordIndex < 0) { + return -1; } else { - throw new DescriptorParseException("Could not detect descriptor " - + "type in descriptor starting with '" + firstLines + "'."); + return this.offset + keywordIndex + 1; } - return parsedDescriptors; }

- protected static List<byte[]> splitRawDescriptorBytes( - byte[] rawDescriptorBytes, String startToken) { - List<byte[]> rawDescriptors = new ArrayList<>(); - String splitToken = NL + startToken; - String ascii; - try { - ascii = new String(rawDescriptorBytes, "US-ASCII"); - } catch (UnsupportedEncodingException e) { - return rawDescriptors; - } - int endAllDescriptors = rawDescriptorBytes.length; - int startAnnotations = 0; - boolean containsAnnotations = ascii.startsWith("@") - || ascii.contains(NL + "@"); - while (startAnnotations < endAllDescriptors) { - int startDescriptor; - if (ascii.indexOf(startToken, startAnnotations) == 0) { - startDescriptor = startAnnotations; - } else { - startDescriptor = ascii.indexOf(splitToken, startAnnotations - 1); - if (startDescriptor < 0) { - break; - } else { - startDescriptor += 1; - } - } - int endDescriptor = -1; - if (containsAnnotations) { - endDescriptor = ascii.indexOf(NL + "@", startDescriptor); + /** + * Returns a list of two-element arrays containing offsets and lengths of + * descriptors starting with the given <code>key</code> in the raw descriptor + * starting at byte <code>offset</code> containing <code>length</code> bytes. + * + * @param key Key to search for. + * @param offset The index of the first byte to split. + * @param length The number of bytes to split. + * @param truncateTrailingNewlines Whether trailing newlines shall be + * truncated. + * @return List of two-element arrays containing offsets and lengths. + */ + protected List<int[]> splitByKey(Key key, int offset, int length, + boolean truncateTrailingNewlines) { + List<int[]> splitParts = new ArrayList<>(); + String ascii = new String(this.rawDescriptorBytes, offset, length, + StandardCharsets.US_ASCII); + int from = 0; + while (from < length) { + int to = ascii.indexOf(NL + key.keyword + SP, from); + if (to < 0) { + to = ascii.indexOf(NL + key.keyword + NL, from); } - if (endDescriptor < 0) { - endDescriptor = ascii.indexOf(splitToken, startDescriptor); + if (to < 0) { + to = length; + } else { + to += 1; } - if (endDescriptor < 0) { - endDescriptor = endAllDescriptors - 1; + int toNoNewline = to; + while (truncateTrailingNewlines && toNoNewline > from + && ascii.charAt(toNoNewline - 1) == '\n') { + toNoNewline--; } - endDescriptor += 1; - byte[] rawDescriptor = new byte[endDescriptor - startAnnotations]; - System.arraycopy(rawDescriptorBytes, startAnnotations, - rawDescriptor, 0, endDescriptor - startAnnotations); - startAnnotations = endDescriptor; - rawDescriptors.add(rawDescriptor); + splitParts.add(new int[] { offset + from, toNoNewline - from }); + from = to; } - return rawDescriptors; - } - - protected byte[] rawDescriptorBytes; - - @Override - public byte[] getRawDescriptorBytes() { - return this.rawDescriptorBytes; + return splitParts; }

protected boolean failUnrecognizedDescriptorLines = false; @@ -193,23 +177,33 @@ public abstract class DescriptorImpl implements Descriptor { : new ArrayList<>(this.unrecognizedLines); }

- protected DescriptorImpl(byte[] rawDescriptorBytes, + protected DescriptorImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines, boolean blankLinesAllowed) throws DescriptorParseException { + int offset = offsetAndLength[0]; + int length = offsetAndLength[1]; + if (offset < 0 || offset + length > rawDescriptorBytes.length + || length < 0) { + throw new IndexOutOfBoundsException("Invalid bounds: " + + "rawDescriptorBytes.length=" + rawDescriptorBytes.length + + " offset=" + offset + " length=" + length); + } this.rawDescriptorBytes = rawDescriptorBytes; + this.offset = offset; + this.length = length; this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; - this.cutOffAnnotations(rawDescriptorBytes); + this.cutOffAnnotations(); this.countKeys(rawDescriptorBytes, blankLinesAllowed); }

/* Parse annotation lines from the descriptor bytes. */ private List<String> annotations = new ArrayList<>();

- private void cutOffAnnotations(byte[] rawDescriptorBytes) - throws DescriptorParseException { - String ascii = new String(rawDescriptorBytes); + private void cutOffAnnotations() throws DescriptorParseException { int start = 0; + String ascii = new String(this.getRawDescriptorBytes(), + StandardCharsets.US_ASCII); while ((start == 0 && ascii.startsWith("@")) || (start > 0 && ascii.indexOf(NL + "@", start - 1) >= 0)) { int end = ascii.indexOf(NL, start); @@ -220,13 +214,8 @@ public abstract class DescriptorImpl implements Descriptor { this.annotations.add(ascii.substring(start, end)); start = end + 1; } - if (start > 0) { - int length = rawDescriptorBytes.length; - byte[] rawDescriptor = new byte[length - start]; - System.arraycopy(rawDescriptorBytes, start, rawDescriptor, 0, - length - start); - this.rawDescriptorBytes = rawDescriptor; - } + this.offset += start; + this.length -= start; }

@Override @@ -246,16 +235,13 @@ public abstract class DescriptorImpl implements Descriptor { if (rawDescriptorBytes.length == 0) { throw new DescriptorParseException("Descriptor is empty."); } - String descriptorString = new String(rawDescriptorBytes); - if (!blankLinesAllowed && (descriptorString.startsWith(NL) - || descriptorString.contains(NL + NL))) { - throw new DescriptorParseException("Blank lines are not allowed."); - } boolean skipCrypto = false; - Scanner scanner = new Scanner(descriptorString).useDelimiter(NL); + Scanner scanner = this.newScanner().useDelimiter(NL); while (scanner.hasNext()) { String line = scanner.next(); - if (line.startsWith(Key.CRYPTO_BEGIN.keyword)) { + if (line.isEmpty() && !blankLinesAllowed) { + throw new DescriptorParseException("Blank lines are not allowed."); + } else if (line.startsWith(Key.CRYPTO_BEGIN.keyword)) { skipCrypto = true; } else if (line.startsWith(Key.CRYPTO_END.keyword)) { skipCrypto = false; @@ -368,8 +354,8 @@ public abstract class DescriptorImpl implements Descriptor { protected void calculateDigestSha1Hex(String startToken, String endToken) throws DescriptorParseException { if (null == this.digestSha1Hex) { - String ascii = new String(this.rawDescriptorBytes, - StandardCharsets.US_ASCII); + String ascii = new String(this.rawDescriptorBytes, this.offset, + this.length, StandardCharsets.US_ASCII); int start = ascii.indexOf(startToken); int end = -1; if (null == endToken) { @@ -378,12 +364,10 @@ public abstract class DescriptorImpl implements Descriptor { end = ascii.indexOf(endToken) + endToken.length(); } if (start >= 0 && end >= 0 && end > start) { - byte[] forDigest = new byte[end - start]; - System.arraycopy(this.rawDescriptorBytes, start, forDigest, 0, - end - start); try { - this.digestSha1Hex = DatatypeConverter.printHexBinary( - MessageDigest.getInstance("SHA-1").digest(forDigest)) + MessageDigest md = MessageDigest.getInstance("SHA-1"); + md.update(this.rawDescriptorBytes, this.offset + start, end - start); + this.digestSha1Hex = DatatypeConverter.printHexBinary(md.digest()) .toLowerCase(); } catch (NoSuchAlgorithmException e) { /* Handle below. */ @@ -409,8 +393,8 @@ public abstract class DescriptorImpl implements Descriptor { protected void calculateDigestSha256Base64(String startToken, String endToken) throws DescriptorParseException { if (null == this.digestSha256Base64) { - String ascii = new String(this.rawDescriptorBytes, - StandardCharsets.US_ASCII); + String ascii = new String(this.rawDescriptorBytes, this.offset, + this.length, StandardCharsets.US_ASCII); int start = ascii.indexOf(startToken); int end = -1; if (null == endToken) { @@ -419,13 +403,11 @@ public abstract class DescriptorImpl implements Descriptor { end = ascii.indexOf(endToken) + endToken.length(); } if (start >= 0 && end >= 0 && end > start) { - byte[] forDigest = new byte[end - start]; - System.arraycopy(this.rawDescriptorBytes, start, forDigest, 0, - end - start); try { + MessageDigest md = MessageDigest.getInstance("SHA-256"); + md.update(this.rawDescriptorBytes, this.offset + start, end - start); this.digestSha256Base64 = DatatypeConverter.printBase64Binary( - MessageDigest.getInstance("SHA-256").digest(forDigest)) - .replaceAll("=", ""); + md.digest()).replaceAll("=", ""); } catch (NoSuchAlgorithmException e) { /* Handle below. */ } diff --git a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java index 518f1a2..6db2883 100644 --- a/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DescriptorParserImpl.java @@ -3,10 +3,17 @@

package org.torproject.descriptor.impl;

+import static org.torproject.descriptor.impl.DescriptorImpl.NL; +import static org.torproject.descriptor.impl.DescriptorImpl.SP; + import org.torproject.descriptor.Descriptor; import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.DescriptorParser;

+import java.lang.reflect.Constructor; +import java.lang.reflect.InvocationTargetException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; import java.util.List;

public class DescriptorParserImpl implements DescriptorParser { @@ -21,9 +28,177 @@ public class DescriptorParserImpl implements DescriptorParser { }

@Override - public List<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, - String fileName) throws DescriptorParseException { - return DescriptorImpl.parseDescriptors(rawDescriptorBytes, fileName, - this.failUnrecognizedDescriptorLines); + public List<Descriptor> parseDescriptors( + byte[] rawDescriptorBytes, String fileName) + throws DescriptorParseException { + byte[] first100Chars = new byte[Math.min(100, + rawDescriptorBytes.length)]; + System.arraycopy(rawDescriptorBytes, 0, first100Chars, 0, + first100Chars.length); + String firstLines = new String(first100Chars); + if (firstLines.startsWith("@type network-status-consensus-3 1.") + || firstLines.startsWith( + "@type network-status-microdesc-consensus-3 1.") + || ((firstLines.startsWith( + Key.NETWORK_STATUS_VERSION.keyword + SP + "3") + || firstLines.contains( + NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "3")) + && firstLines.contains( + NL + Key.VOTE_STATUS.keyword + SP + "consensus" + NL))) { + return parseDescriptors(rawDescriptorBytes, Key.NETWORK_STATUS_VERSION, + RelayNetworkStatusConsensusImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type network-status-vote-3 1.") + || ((firstLines.startsWith( + Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL) + || firstLines.contains( + NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "3" + NL)) + && firstLines.contains( + NL + Key.VOTE_STATUS.keyword + SP + "vote" + NL))) { + return parseDescriptors(rawDescriptorBytes, Key.NETWORK_STATUS_VERSION, + RelayNetworkStatusVoteImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type bridge-network-status 1.") + || firstLines.startsWith(Key.R.keyword + SP)) { + List<Descriptor> parsedDescriptors = new ArrayList<>(); + parsedDescriptors.add(new BridgeNetworkStatusImpl( + rawDescriptorBytes, new int[] { 0, rawDescriptorBytes.length }, + fileName, this.failUnrecognizedDescriptorLines)); + return parsedDescriptors; + } else if (firstLines.startsWith("@type bridge-server-descriptor 1.")) { + return parseDescriptors(rawDescriptorBytes, Key.ROUTER, + BridgeServerDescriptorImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type server-descriptor 1.") + || firstLines.startsWith(Key.ROUTER.keyword + SP) + || firstLines.contains(NL + Key.ROUTER.keyword + SP)) { + return parseDescriptors(rawDescriptorBytes, Key.ROUTER, + RelayServerDescriptorImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type bridge-extra-info 1.")) { + return parseDescriptors(rawDescriptorBytes, Key.EXTRA_INFO, + BridgeExtraInfoDescriptorImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type extra-info 1.") + || firstLines.startsWith(Key.EXTRA_INFO.keyword + SP) + || firstLines.contains(NL + Key.EXTRA_INFO.keyword + SP)) { + return parseDescriptors(rawDescriptorBytes, Key.EXTRA_INFO, + RelayExtraInfoDescriptorImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type microdescriptor 1.") + || firstLines.startsWith(Key.ONION_KEY.keyword + NL) + || firstLines.contains(NL + Key.ONION_KEY.keyword + NL)) { + return parseDescriptors(rawDescriptorBytes, Key.ONION_KEY, + MicrodescriptorImpl.class, this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type bridge-pool-assignment 1.") + || firstLines.startsWith(Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP) + || firstLines.contains(NL + Key.BRIDGE_POOL_ASSIGNMENT.keyword + SP)) { + return parseDescriptors(rawDescriptorBytes, Key.BRIDGE_POOL_ASSIGNMENT, + BridgePoolAssignmentImpl.class, this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type dir-key-certificate-3 1.") + || firstLines.startsWith(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP) + || firstLines.contains( + NL + Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP)) { + return parseDescriptors(rawDescriptorBytes, + Key.DIR_KEY_CERTIFICATE_VERSION, DirectoryKeyCertificateImpl.class, + this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type tordnsel 1.") + || firstLines.startsWith("ExitNode" + SP) + || firstLines.contains(NL + "ExitNode" + SP)) { + List<Descriptor> parsedDescriptors = new ArrayList<>(); + parsedDescriptors.add(new ExitListImpl(rawDescriptorBytes, fileName, + this.failUnrecognizedDescriptorLines)); + return parsedDescriptors; + } else if (firstLines.startsWith("@type network-status-2 1.") + || firstLines.startsWith( + Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL) + || firstLines.contains( + NL + Key.NETWORK_STATUS_VERSION.keyword + SP + "2" + NL)) { + return parseDescriptors(rawDescriptorBytes, Key.NETWORK_STATUS_VERSION, + RelayNetworkStatusImpl.class, this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type directory 1.") + || firstLines.startsWith(Key.SIGNED_DIRECTORY.keyword + NL) + || firstLines.contains(NL + Key.SIGNED_DIRECTORY.keyword + NL)) { + return parseDescriptors(rawDescriptorBytes, Key.SIGNED_DIRECTORY, + RelayDirectoryImpl.class, this.failUnrecognizedDescriptorLines); + } else if (firstLines.startsWith("@type torperf 1.")) { + return TorperfResultImpl.parseTorperfResults( + rawDescriptorBytes, this.failUnrecognizedDescriptorLines); + } else { + throw new DescriptorParseException("Could not detect descriptor " + + "type in descriptor starting with '" + firstLines + "'."); + } + } + + private static List<Descriptor> parseDescriptors(byte[] rawDescriptorBytes, + Key key, Class<? extends DescriptorImpl> descriptorClass, + boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + List<Descriptor> parsedDescriptors = new ArrayList<>(); + Constructor<? extends DescriptorImpl> constructor; + try { + constructor = descriptorClass.getDeclaredConstructor(byte[].class, + int[].class, boolean.class); + } catch (NoSuchMethodException e) { + throw new RuntimeException(e); + } + int startAnnotations = 0; + int endAllDescriptors = rawDescriptorBytes.length; + String ascii = new String(rawDescriptorBytes, StandardCharsets.US_ASCII); + boolean containsAnnotations = ascii.startsWith("@") + || ascii.contains(NL + "@"); + while (startAnnotations < endAllDescriptors) { + int startDescriptor; + if (startAnnotations == ascii.indexOf(key.keyword + SP, + startAnnotations) || startAnnotations == ascii.indexOf( + key.keyword + SP)) { + startDescriptor = startAnnotations; + } else { + startDescriptor = ascii.indexOf(NL + key.keyword + SP, + startAnnotations - 1); + if (startDescriptor < 0) { + startDescriptor = ascii.indexOf(NL + key.keyword + NL, + startAnnotations - 1); + } + if (startDescriptor < 0) { + break; + } else { + startDescriptor += 1; + } + } + int endDescriptor = -1; + if (containsAnnotations) { + endDescriptor = ascii.indexOf(NL + "@", startDescriptor); + } + if (endDescriptor < 0) { + endDescriptor = ascii.indexOf(NL + key.keyword + SP, startDescriptor); + } + if (endDescriptor < 0) { + endDescriptor = ascii.indexOf(NL + key.keyword + NL, startDescriptor); + } + if (endDescriptor < 0) { + endDescriptor = endAllDescriptors - 1; + } + endDescriptor += 1; + int[] offsetAndLength = new int[] { startAnnotations, + endDescriptor - startAnnotations }; + parsedDescriptors.add(parseDescriptor(rawDescriptorBytes, + offsetAndLength, constructor, failUnrecognizedDescriptorLines)); + startAnnotations = endDescriptor; + } + return parsedDescriptors; } + + private static Descriptor parseDescriptor(byte[] rawDescriptorBytes, + int[] offsetAndLength, Constructor<? extends DescriptorImpl> constructor, + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + try { + return constructor.newInstance(rawDescriptorBytes, + offsetAndLength, failUnrecognizedDescriptorLines); + } catch (InstantiationException | IllegalAccessException + | InvocationTargetException e) { + throw new RuntimeException(); + } + } + } diff --git a/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java b/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java index fd2c783..770ca61 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirSourceEntryImpl.java @@ -17,11 +17,18 @@ import java.util.Set;

public class DirSourceEntryImpl implements DirSourceEntry {

- private byte[] dirSourceEntryBytes; + private DescriptorImpl parent; + + private int offset; + + private int length;

@Override public byte[] getDirSourceEntryBytes() { - return this.dirSourceEntryBytes; + /* We need to pass this.offset and this.length, because the overloaded + * method without arguments would use this.parent.offset and + * this.parent.length as bounds, which is not what we want! */ + return this.parent.getRawDescriptorBytes(this.offset, this.length); }

private boolean failUnrecognizedDescriptorLines; @@ -34,10 +41,12 @@ public class DirSourceEntryImpl implements DirSourceEntry { return lines; }

- protected DirSourceEntryImpl(byte[] dirSourceEntryBytes, + protected DirSourceEntryImpl(DescriptorImpl parent, int offset, int length, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - this.dirSourceEntryBytes = dirSourceEntryBytes; + this.parent = parent; + this.offset = offset; + this.length = length; this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; this.parseDirSourceEntryBytes(); @@ -80,7 +89,10 @@ public class DirSourceEntryImpl implements DirSourceEntry {

private void parseDirSourceEntryBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.dirSourceEntryBytes)) + /* We need to pass this.offset and this.length, because the overloaded + * method without arguments would use this.parent.offset and + * this.parent.length as bounds, which is not what we want! */ + Scanner scanner = this.parent.newScanner(this.offset, this.length) .useDelimiter(NL); boolean skipCrypto = false; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java b/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java index 0ade2c9..9a06a93 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirectoryKeyCertificateImpl.java @@ -8,33 +8,17 @@ import org.torproject.descriptor.DirectoryKeyCertificate;

import java.util.ArrayList; import java.util.EnumSet; -import java.util.List; import java.util.Scanner; import java.util.Set;

public class DirectoryKeyCertificateImpl extends DescriptorImpl implements DirectoryKeyCertificate {

- protected static List<DirectoryKeyCertificate> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<DirectoryKeyCertificate> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DirectoryKeyCertificateImpl.splitRawDescriptorBytes( - descriptorsBytes, Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - DirectoryKeyCertificate parsedDescriptor = - new DirectoryKeyCertificateImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - protected DirectoryKeyCertificateImpl(byte[] rawDescriptorBytes, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(rawDescriptorBytes, failUnrecognizedDescriptorLines, false); + super(rawDescriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines, + false); this.parseDescriptorBytes(); this.calculateDigestSha1Hex(Key.DIR_KEY_CERTIFICATE_VERSION.keyword + SP, NL + Key.DIR_KEY_CERTIFICATION.keyword + NL); @@ -52,8 +36,7 @@ public class DirectoryKeyCertificateImpl extends DescriptorImpl }

private void parseDescriptorBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.rawDescriptorBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner().useDelimiter(NL); Key nextCrypto = Key.EMPTY; StringBuilder crypto = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java b/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java index 674b634..be0abf8 100644 --- a/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/DirectorySignatureImpl.java @@ -15,7 +15,11 @@ import java.util.Scanner;

public class DirectorySignatureImpl implements DirectorySignature {

- private byte[] directorySignatureBytes; + private DescriptorImpl parent; + + private int offset; + + private int length;

private boolean failUnrecognizedDescriptorLines;

@@ -27,10 +31,12 @@ public class DirectorySignatureImpl implements DirectorySignature { return lines; }

- protected DirectorySignatureImpl(byte[] directorySignatureBytes, - boolean failUnrecognizedDescriptorLines) + protected DirectorySignatureImpl(DescriptorImpl parent, int offset, + int length, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - this.directorySignatureBytes = directorySignatureBytes; + this.parent = parent; + this.offset = offset; + this.length = length; this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; this.parseDirectorySignatureBytes(); @@ -38,7 +44,7 @@ public class DirectorySignatureImpl implements DirectorySignature {

private void parseDirectorySignatureBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.directorySignatureBytes)) + Scanner scanner = this.parent.newScanner(this.offset, this.length) .useDelimiter(NL); StringBuilder crypto = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java b/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java index 947cf9c..c0d8b52 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExitListEntryImpl.java @@ -17,7 +17,7 @@ import java.util.TreeSet;

public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry {

- private byte[] exitListEntryBytes; + private String exitListEntryString;

private boolean failUnrecognizedDescriptorLines;

@@ -54,14 +54,14 @@ public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry { return result; }

- protected ExitListEntryImpl(byte[] exitListEntryBytes, + protected ExitListEntryImpl(String exitListEntryString, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - this.exitListEntryBytes = exitListEntryBytes; + this.exitListEntryString = exitListEntryString; this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; this.initializeKeywords(); - this.parseExitListEntryBytes(); + this.parseExitListEntry(); this.checkAndClearKeywords(); }

@@ -92,9 +92,9 @@ public class ExitListEntryImpl implements ExitListEntry, ExitList.Entry { this.keywordCountingSet = null; }

- private void parseExitListEntryBytes() + private void parseExitListEntry() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.exitListEntryBytes)) + Scanner scanner = new Scanner(this.exitListEntryString) .useDelimiter(ExitList.EOL); while (scanner.hasNext()) { String line = scanner.next(); diff --git a/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java b/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java index 75d3a04..a87a55f 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExitListImpl.java @@ -21,8 +21,9 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { protected ExitListImpl(byte[] rawDescriptorBytes, String fileName, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(rawDescriptorBytes, failUnrecognizedDescriptorLines, false); - this.splitAndParseExitListEntries(rawDescriptorBytes); + super(rawDescriptorBytes, new int[] { 0, rawDescriptorBytes.length }, + failUnrecognizedDescriptorLines, false); + this.splitAndParseExitListEntries(); this.setPublishedMillisFromFileName(fileName); }

@@ -46,13 +47,9 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { } }

- private void splitAndParseExitListEntries(byte[] rawDescriptorBytes) + private void splitAndParseExitListEntries() throws DescriptorParseException { - if (this.rawDescriptorBytes.length == 0) { - throw new DescriptorParseException("Descriptor is empty."); - } - String descriptorString = new String(rawDescriptorBytes); - Scanner scanner = new Scanner(descriptorString).useDelimiter(EOL); + Scanner scanner = this.newScanner().useDelimiter(EOL); StringBuilder sb = new StringBuilder(); boolean firstEntry = true; while (scanner.hasNext()) { @@ -73,7 +70,7 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { break; case "ExitNode": if (!firstEntry) { - this.parseExitListEntry(sb.toString().getBytes()); + this.parseExitListEntry(sb.toString()); } else { firstEntry = false; } @@ -102,13 +99,13 @@ public class ExitListImpl extends DescriptorImpl implements ExitList { } } /* Parse the last entry. */ - this.parseExitListEntry(sb.toString().getBytes()); + this.parseExitListEntry(sb.toString()); }

- protected void parseExitListEntry(byte[] exitListEntryBytes) + protected void parseExitListEntry(String exitListEntryString) throws DescriptorParseException { ExitListEntryImpl exitListEntry = new ExitListEntryImpl( - exitListEntryBytes, this.failUnrecognizedDescriptorLines); + exitListEntryString, this.failUnrecognizedDescriptorLines); this.exitListEntries.add(exitListEntry); this.oldExitListEntries.addAll(exitListEntry.oldEntries()); List<String> unrecognizedExitListEntryLines = exitListEntry diff --git a/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java index 7b99114..b4c3a0b 100644 --- a/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImpl.java @@ -34,9 +34,10 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl Key.PADDING_COUNTS);

protected ExtraInfoDescriptorImpl(byte[] descriptorBytes, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLimit, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines, false); + super(descriptorBytes, offsetAndLimit, failUnrecognizedDescriptorLines, + false); this.parseDescriptorBytes(); this.calculateDigestSha1Hex(Key.EXTRA_INFO.keyword + SP, NL + Key.ROUTER_SIGNATURE.keyword + NL); @@ -78,8 +79,7 @@ public abstract class ExtraInfoDescriptorImpl extends DescriptorImpl }

private void parseDescriptorBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.rawDescriptorBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner().useDelimiter(NL); Key nextCrypto = Key.EMPTY; List<String> cryptoLines = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java index e54b939..65c20d4 100644 --- a/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/MicrodescriptorImpl.java @@ -17,26 +17,11 @@ import java.util.Set; public class MicrodescriptorImpl extends DescriptorImpl implements Microdescriptor {

- protected static List<Microdescriptor> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<Microdescriptor> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DescriptorImpl.splitRawDescriptorBytes(descriptorsBytes, - Key.ONION_KEY + NL); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - Microdescriptor parsedDescriptor = - new MicrodescriptorImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - - protected MicrodescriptorImpl(byte[] descriptorBytes, + protected MicrodescriptorImpl(byte[] descriptorBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines, false); + super(descriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines, + false); this.parseDescriptorBytes(); this.calculateDigestSha256Base64(Key.ONION_KEY.keyword + NL); this.checkExactlyOnceKeys(EnumSet.of(Key.ONION_KEY)); @@ -49,8 +34,7 @@ public class MicrodescriptorImpl extends DescriptorImpl }

private void parseDescriptorBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.rawDescriptorBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner().useDelimiter(NL); Key nextCrypto = Key.EMPTY; StringBuilder crypto = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java b/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java index fd4b3e3..987b530 100644 --- a/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/NetworkStatusEntryImpl.java @@ -24,11 +24,18 @@ import java.util.TreeSet;

public class NetworkStatusEntryImpl implements NetworkStatusEntry {

- private byte[] statusEntryBytes; + private DescriptorImpl parent; + + private int offset; + + private int length;

@Override public byte[] getStatusEntryBytes() { - return this.statusEntryBytes; + /* We need to pass this.offset and this.length, because the overloaded + * method without arguments would use this.parent.offset and + * this.parent.length as bounds, which is not what we want! */ + return this.parent.getRawDescriptorBytes(this.offset, this.length); }

private boolean microdescConsensus; @@ -43,10 +50,12 @@ public class NetworkStatusEntryImpl implements NetworkStatusEntry { return lines; }

- protected NetworkStatusEntryImpl(byte[] statusEntryBytes, - boolean microdescConsensus, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - this.statusEntryBytes = statusEntryBytes; + protected NetworkStatusEntryImpl(DescriptorImpl parent, int offset, + int length, boolean microdescConsensus, + boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { + this.parent = parent; + this.offset = offset; + this.length = length; this.microdescConsensus = microdescConsensus; this.failUnrecognizedDescriptorLines = failUnrecognizedDescriptorLines; @@ -67,7 +76,10 @@ public class NetworkStatusEntryImpl implements NetworkStatusEntry { }

private void parseStatusEntryBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.statusEntryBytes)) + /* We need to pass this.offset and this.length, because the overloaded + * method without arguments would use this.parent.offset and + * this.parent.length as bounds, which is not what we want! */ + Scanner scanner = this.parent.newScanner(this.offset, this.length) .useDelimiter(NL); String line = null; if (!scanner.hasNext() || !(line = scanner.next()).startsWith("r ")) { diff --git a/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java index 93acf7b..6765b6f 100644 --- a/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/NetworkStatusImpl.java @@ -8,7 +8,6 @@ import org.torproject.descriptor.DirSourceEntry; import org.torproject.descriptor.DirectorySignature; import org.torproject.descriptor.NetworkStatusEntry;

-import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.List; import java.util.SortedMap; @@ -19,33 +18,25 @@ import java.util.TreeMap; * delegate the specific parts to the subclasses. */ public abstract class NetworkStatusImpl extends DescriptorImpl {

- protected NetworkStatusImpl(byte[] rawDescriptorBytes, + protected NetworkStatusImpl(byte[] rawDescriptorBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines, boolean containsDirSourceEntries, boolean blankLinesAllowed) throws DescriptorParseException { - super(rawDescriptorBytes, failUnrecognizedDescriptorLines, + super(rawDescriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines, blankLinesAllowed); - this.splitAndParseParts(this.rawDescriptorBytes, - containsDirSourceEntries); + this.splitAndParseParts(containsDirSourceEntries); }

- private void splitAndParseParts(byte[] rawDescriptorBytes, - boolean containsDirSourceEntries) throws DescriptorParseException { - if (this.rawDescriptorBytes.length == 0) { - throw new DescriptorParseException("Descriptor is empty."); - } - String descriptorString = new String(rawDescriptorBytes, - StandardCharsets.US_ASCII); - int firstRIndex = this.findFirstIndexOfKeyword(descriptorString, - Key.R.keyword); - int endIndex = descriptorString.length(); - int firstDirectorySignatureIndex = this.findFirstIndexOfKeyword( - descriptorString, Key.DIRECTORY_SIGNATURE.keyword); + private void splitAndParseParts(boolean containsDirSourceEntries) + throws DescriptorParseException { + int firstRIndex = this.findFirstIndexOfKey(Key.R); + int firstDirectorySignatureIndex = this.findFirstIndexOfKey( + Key.DIRECTORY_SIGNATURE); + int endIndex = this.offset + this.length; if (firstDirectorySignatureIndex < 0) { firstDirectorySignatureIndex = endIndex; } - int directoryFooterIndex = this.findFirstIndexOfKeyword( - descriptorString, Key.DIRECTORY_FOOTER.keyword); + int directoryFooterIndex = this.findFirstIndexOfKey(Key.DIRECTORY_FOOTER); if (directoryFooterIndex < 0) { directoryFooterIndex = firstDirectorySignatureIndex; } @@ -53,119 +44,64 @@ public abstract class NetworkStatusImpl extends DescriptorImpl { firstRIndex = directoryFooterIndex; } int firstDirSourceIndex = !containsDirSourceEntries ? -1 - : this.findFirstIndexOfKeyword(descriptorString, - Key.DIR_SOURCE.keyword); + : this.findFirstIndexOfKey(Key.DIR_SOURCE); if (firstDirSourceIndex < 0) { firstDirSourceIndex = firstRIndex; } - if (firstDirSourceIndex > 0) { - this.parseHeaderBytes(descriptorString, 0, firstDirSourceIndex); + if (firstDirSourceIndex > this.offset) { + this.parseHeader(this.offset, firstDirSourceIndex - this.offset); } if (firstRIndex > firstDirSourceIndex) { - this.parseDirSourceBytes(descriptorString, firstDirSourceIndex, - firstRIndex); + this.parseDirSources(firstDirSourceIndex, firstRIndex + - firstDirSourceIndex); } if (directoryFooterIndex > firstRIndex) { - this.parseStatusEntryBytes(descriptorString, firstRIndex, - directoryFooterIndex); + this.parseStatusEntries(firstRIndex, directoryFooterIndex - firstRIndex); } if (firstDirectorySignatureIndex > directoryFooterIndex) { - this.parseDirectoryFooterBytes(descriptorString, - directoryFooterIndex, firstDirectorySignatureIndex); + this.parseFooter(directoryFooterIndex, firstDirectorySignatureIndex + - directoryFooterIndex); } if (endIndex > firstDirectorySignatureIndex) { - this.parseDirectorySignatureBytes(descriptorString, - firstDirectorySignatureIndex, endIndex); - } - } - - private int findFirstIndexOfKeyword(String descriptorString, - String keyword) { - if (descriptorString.startsWith(keyword)) { - return 0; - } else if (descriptorString.contains(NL + keyword + SP)) { - return descriptorString.indexOf(NL + keyword + SP) + 1; - } else if (descriptorString.contains(NL + keyword + NL)) { - return descriptorString.indexOf(NL + keyword + NL) + 1; - } else { - return -1; + this.parseDirectorySignatures(firstDirectorySignatureIndex, + endIndex - firstDirectorySignatureIndex); } }

- private void parseHeaderBytes(String descriptorString, int start, - int end) throws DescriptorParseException { - byte[] headerBytes = new byte[end - start]; - System.arraycopy(this.rawDescriptorBytes, start, - headerBytes, 0, end - start); - this.parseHeader(headerBytes); - } - - private void parseDirSourceBytes(String descriptorString, int start, - int end) throws DescriptorParseException { - List<byte[]> splitDirSourceBytes = - this.splitByKeyword( - descriptorString, Key.DIR_SOURCE.keyword, start, end); - for (byte[] dirSourceBytes : splitDirSourceBytes) { - this.parseDirSource(dirSourceBytes); - } - } - - private void parseStatusEntryBytes(String descriptorString, int start, - int end) throws DescriptorParseException { - List<byte[]> splitStatusEntryBytes = - this.splitByKeyword(descriptorString, Key.R.keyword, start, end); - for (byte[] statusEntryBytes : splitStatusEntryBytes) { - this.parseStatusEntry(statusEntryBytes); + private void parseDirSources(int offset, int length) + throws DescriptorParseException { + List<int[]> offsetsAndLengths = this.splitByKey(Key.DIR_SOURCE, offset, + length, false); + for (int[] offsetAndLength : offsetsAndLengths) { + this.parseDirSource(offsetAndLength[0], offsetAndLength[1]); } }

- private void parseDirectoryFooterBytes(String descriptorString, - int start, int end) throws DescriptorParseException { - byte[] directoryFooterBytes = new byte[end - start]; - System.arraycopy(this.rawDescriptorBytes, start, - directoryFooterBytes, 0, end - start); - this.parseFooter(directoryFooterBytes); - } - - private void parseDirectorySignatureBytes(String descriptorString, - int start, int end) throws DescriptorParseException { - List<byte[]> splitDirectorySignatureBytes = this.splitByKeyword( - descriptorString, Key.DIRECTORY_SIGNATURE.keyword, start, end); - for (byte[] directorySignatureBytes : splitDirectorySignatureBytes) { - this.parseDirectorySignature(directorySignatureBytes); + private void parseStatusEntries(int offset, int length) + throws DescriptorParseException { + List<int[]> offsetsAndLengths = this.splitByKey(Key.R, offset, length, + false); + for (int[] offsetAndLength : offsetsAndLengths) { + this.parseStatusEntry(offsetAndLength[0], offsetAndLength[1]); } }

- private List<byte[]> splitByKeyword(String descriptorString, - String keyword, int start, int end) { - List<byte[]> splitParts = new ArrayList<>(); - int from = start; - while (from < end) { - int to = descriptorString.indexOf(NL + keyword + SP, from); - if (to < 0) { - to = descriptorString.indexOf(NL + keyword + NL, from); - } - if (to < 0) { - to = end; - } else { - to += 1; - } - byte[] part = new byte[to - from]; - System.arraycopy(this.rawDescriptorBytes, from, part, 0, - to - from); - from = to; - splitParts.add(part); + private void parseDirectorySignatures(int offset, int length) + throws DescriptorParseException { + List<int[]> offsetsAndLengths = this.splitByKey(Key.DIRECTORY_SIGNATURE, + offset, length, false); + for (int[] offsetAndLength : offsetsAndLengths) { + this.parseDirectorySignature(offsetAndLength[0], offsetAndLength[1]); } - return splitParts; }

- protected abstract void parseHeader(byte[] headerBytes) + protected abstract void parseHeader(int offset, int length) throws DescriptorParseException;

- protected void parseDirSource(byte[] dirSourceBytes) + protected void parseDirSource(int offset, int length) throws DescriptorParseException { DirSourceEntryImpl dirSourceEntry = new DirSourceEntryImpl( - dirSourceBytes, this.failUnrecognizedDescriptorLines); + this, offset, length, this.failUnrecognizedDescriptorLines); this.dirSourceEntries.put(dirSourceEntry.getIdentity(), dirSourceEntry); List<String> unrecognizedDirSourceLines = dirSourceEntry @@ -201,10 +137,10 @@ public abstract class NetworkStatusImpl extends DescriptorImpl { return result; }

- protected void parseStatusEntry(byte[] statusEntryBytes) + protected void parseStatusEntry(int offset, int length) throws DescriptorParseException { NetworkStatusEntryImpl statusEntry = new NetworkStatusEntryImpl( - statusEntryBytes, false, this.failUnrecognizedDescriptorLines); + this, offset, length, false, this.failUnrecognizedDescriptorLines); this.statusEntries.put(statusEntry.getFingerprint(), statusEntry); List<String> unrecognizedStatusEntryLines = statusEntry .getAndClearUnrecognizedLines(); @@ -216,16 +152,16 @@ public abstract class NetworkStatusImpl extends DescriptorImpl { } }

- protected abstract void parseFooter(byte[] footerBytes) + protected abstract void parseFooter(int offset, int length) throws DescriptorParseException;

- protected void parseDirectorySignature(byte[] directorySignatureBytes) + protected void parseDirectorySignature(int offset, int length) throws DescriptorParseException { if (this.signatures == null) { this.signatures = new ArrayList<>(); } DirectorySignatureImpl signature = new DirectorySignatureImpl( - directorySignatureBytes, failUnrecognizedDescriptorLines); + this, offset, length, failUnrecognizedDescriptorLines); this.signatures.add(signature); List<String> unrecognizedStatusEntryLines = signature .getAndClearUnrecognizedLines(); diff --git a/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java index 27887e6..d5108ca 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayDirectoryImpl.java @@ -8,7 +8,6 @@ import org.torproject.descriptor.RelayDirectory; import org.torproject.descriptor.RouterStatusEntry; import org.torproject.descriptor.ServerDescriptor;

-import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.EnumSet; import java.util.List; @@ -18,27 +17,12 @@ import java.util.Set; public class RelayDirectoryImpl extends DescriptorImpl implements RelayDirectory {

- protected static List<RelayDirectory> parseDirectories( - byte[] directoriesBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<RelayDirectory> parsedDirectories = new ArrayList<>(); - List<byte[]> splitDirectoriesBytes = - DescriptorImpl.splitRawDescriptorBytes(directoriesBytes, - Key.SIGNED_DIRECTORY.keyword + NL); - for (byte[] directoryBytes : splitDirectoriesBytes) { - RelayDirectory parsedDirectory = - new RelayDirectoryImpl(directoryBytes, - failUnrecognizedDescriptorLines); - parsedDirectories.add(parsedDirectory); - } - return parsedDirectories; - } - - protected RelayDirectoryImpl(byte[] directoryBytes, + protected RelayDirectoryImpl(byte[] directoryBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(directoryBytes, failUnrecognizedDescriptorLines, true); - this.splitAndParseParts(rawDescriptorBytes); + super(directoryBytes, offsetAndLength, failUnrecognizedDescriptorLines, + true); + this.splitAndParseParts(); this.calculateDigestSha1Hex(Key.SIGNED_DIRECTORY.keyword + NL, NL + Key.DIRECTORY_SIGNATURE.keyword + SP); Set<Key> exactlyOnceKeys = EnumSet.of( @@ -52,19 +36,12 @@ public class RelayDirectoryImpl extends DescriptorImpl this.clearParsedKeys(); }

- private void splitAndParseParts(byte[] rawDescriptorBytes) - throws DescriptorParseException { - if (this.rawDescriptorBytes.length == 0) { - throw new DescriptorParseException("Descriptor is empty."); - } - String descriptorString = new String(rawDescriptorBytes, - StandardCharsets.US_ASCII); + private void splitAndParseParts() throws DescriptorParseException { int startIndex = 0; - int firstRouterIndex = this.findFirstIndexOfKeyword(descriptorString, - Key.ROUTER.keyword); - int directorySignatureIndex = this.findFirstIndexOfKeyword( - descriptorString, Key.DIRECTORY_SIGNATURE.keyword); - int endIndex = descriptorString.length(); + int firstRouterIndex = this.findFirstIndexOfKey(Key.ROUTER); + int directorySignatureIndex = this.findFirstIndexOfKey( + Key.DIRECTORY_SIGNATURE); + int endIndex = this.offset + this.length; if (directorySignatureIndex < 0) { directorySignatureIndex = endIndex; } @@ -72,89 +49,39 @@ public class RelayDirectoryImpl extends DescriptorImpl firstRouterIndex = directorySignatureIndex; } if (firstRouterIndex > startIndex) { - this.parseHeaderBytes(descriptorString, startIndex, - firstRouterIndex); + this.parseHeader(startIndex, firstRouterIndex - startIndex); } if (directorySignatureIndex > firstRouterIndex) { - this.parseServerDescriptorBytes(descriptorString, firstRouterIndex, - directorySignatureIndex); + this.parseServerDescriptors(firstRouterIndex, + directorySignatureIndex - firstRouterIndex); } if (endIndex > directorySignatureIndex) { - this.parseDirectorySignatureBytes(descriptorString, - directorySignatureIndex, endIndex); - } - } - - private int findFirstIndexOfKeyword(String descriptorString, - String keyword) { - if (descriptorString.startsWith(keyword)) { - return 0; - } else if (descriptorString.contains(NL + keyword + SP)) { - return descriptorString.indexOf(NL + keyword + SP) + 1; - } else if (descriptorString.contains(NL + keyword + NL)) { - return descriptorString.indexOf(NL + keyword + NL) + 1; - } else { - return -1; + this.parseDirectorySignatures(directorySignatureIndex, + endIndex - directorySignatureIndex); } }

- private void parseHeaderBytes(String descriptorString, int start, - int end) throws DescriptorParseException { - byte[] headerBytes = new byte[end - start]; - System.arraycopy(this.rawDescriptorBytes, start, - headerBytes, 0, end - start); - this.parseHeader(headerBytes); - } - - private void parseServerDescriptorBytes(String descriptorString, - int start, int end) throws DescriptorParseException { - List<byte[]> splitServerDescriptorBytes = - this.splitByKeyword(descriptorString, Key.ROUTER.keyword, start, end); - for (byte[] statusEntryBytes : splitServerDescriptorBytes) { - this.parseServerDescriptor(statusEntryBytes); - } - } - - private void parseDirectorySignatureBytes(String descriptorString, - int start, int end) throws DescriptorParseException { - List<byte[]> splitDirectorySignatureBytes = this.splitByKeyword( - descriptorString, "directory-signature", start, end); - for (byte[] directorySignatureBytes : splitDirectorySignatureBytes) { - this.parseDirectorySignature(directorySignatureBytes); + private void parseServerDescriptors(int offset, int length) + throws DescriptorParseException { + List<int[]> offsetsAndLengths = this.splitByKey(Key.ROUTER, offset, length, + true); + for (int[] offsetAndLength : offsetsAndLengths) { + this.parseServerDescriptor(offsetAndLength[0], offsetAndLength[1]); } }

- private List<byte[]> splitByKeyword(String descriptorString, - String keyword, int start, int end) { - List<byte[]> splitParts = new ArrayList<>(); - int from = start; - while (from < end) { - int to = descriptorString.indexOf(NL + keyword + SP, from); - if (to < 0) { - to = descriptorString.indexOf(NL + keyword + NL, from); - } - if (to < 0) { - to = end; - } else { - to += 1; - } - int toNoNewline = to; - while (toNoNewline > from - && descriptorString.charAt(toNoNewline - 1) == '\n') { - toNoNewline--; - } - byte[] part = new byte[toNoNewline - from]; - System.arraycopy(this.rawDescriptorBytes, from, part, 0, - toNoNewline - from); - from = to; - splitParts.add(part); + private void parseDirectorySignatures(int offset, int length) + throws DescriptorParseException { + List<int[]> offsetsAndLengths = this.splitByKey(Key.DIRECTORY_SIGNATURE, + offset, length, false); + for (int[] offsetAndLength : offsetsAndLengths) { + this.parseDirectorySignature(offsetAndLength[0], offsetAndLength[1]); } - return splitParts; }

- private void parseHeader(byte[] headerBytes) + private void parseHeader(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(headerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); String publishedLine = null; Key nextCrypto = Key.EMPTY; String runningRoutersLine = null; @@ -263,21 +190,20 @@ public class RelayDirectoryImpl extends DescriptorImpl } }

- protected void parseServerDescriptor(byte[] serverDescriptorBytes) { + protected void parseServerDescriptor(int offset, int length) { try { ServerDescriptorImpl serverDescriptor = - new RelayServerDescriptorImpl(serverDescriptorBytes, - this.failUnrecognizedDescriptorLines); + new RelayServerDescriptorImpl(this.rawDescriptorBytes, + new int[] { offset, length }, this.failUnrecognizedDescriptorLines); this.serverDescriptors.add(serverDescriptor); } catch (DescriptorParseException e) { this.serverDescriptorParseExceptions.add(e); } }

- private void parseDirectorySignature(byte[] directorySignatureBytes) + private void parseDirectorySignature(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(directorySignatureBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); Key nextCrypto = Key.EMPTY; StringBuilder crypto = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java index 6ee86b1..37bdc21 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayExtraInfoDescriptorImpl.java @@ -4,35 +4,15 @@ package org.torproject.descriptor.impl;

import org.torproject.descriptor.DescriptorParseException; -import org.torproject.descriptor.ExtraInfoDescriptor; import org.torproject.descriptor.RelayExtraInfoDescriptor;

-import java.util.ArrayList; -import java.util.List; - public class RelayExtraInfoDescriptorImpl extends ExtraInfoDescriptorImpl implements RelayExtraInfoDescriptor {

- protected static List<ExtraInfoDescriptor> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<ExtraInfoDescriptor> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DescriptorImpl.splitRawDescriptorBytes(descriptorsBytes, - Key.EXTRA_INFO.keyword + SP); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - ExtraInfoDescriptor parsedDescriptor = - new RelayExtraInfoDescriptorImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - protected RelayExtraInfoDescriptorImpl(byte[] descriptorBytes, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLimit, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines); + super(descriptorBytes, offsetAndLimit, failUnrecognizedDescriptorLines); } }

diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java index a90ae3e..12a9dc3 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImpl.java @@ -21,27 +21,11 @@ import java.util.TreeSet; public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl implements RelayNetworkStatusConsensus {

- protected static List<RelayNetworkStatusConsensus> parseConsensuses( - byte[] consensusesBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<RelayNetworkStatusConsensus> parsedConsensuses = - new ArrayList<>(); - List<byte[]> splitConsensusBytes = - DescriptorImpl.splitRawDescriptorBytes(consensusesBytes, - Key.NETWORK_STATUS_VERSION.keyword + SP + "3"); - for (byte[] consensusBytes : splitConsensusBytes) { - RelayNetworkStatusConsensus parsedConsensus = - new RelayNetworkStatusConsensusImpl(consensusBytes, - failUnrecognizedDescriptorLines); - parsedConsensuses.add(parsedConsensus); - } - return parsedConsensuses; - } - protected RelayNetworkStatusConsensusImpl(byte[] consensusBytes, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLimit, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(consensusBytes, failUnrecognizedDescriptorLines, true, false); + super(consensusBytes, offsetAndLimit, failUnrecognizedDescriptorLines, true, + false); Set<Key> exactlyOnceKeys = EnumSet.of( Key.VOTE_STATUS, Key.CONSENSUS_METHOD, Key.VALID_AFTER, Key.FRESH_UNTIL, Key.VALID_UNTIL, Key.VOTING_DELAY, Key.KNOWN_FLAGS); @@ -59,9 +43,9 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl NL + Key.DIRECTORY_SIGNATURE.keyword + SP); }

- protected void parseHeader(byte[] headerBytes) + protected void parseHeader(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(headerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); while (scanner.hasNext()) { String line = scanner.next(); String[] parts = line.split("[ \t]+"); @@ -137,10 +121,10 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl

private boolean microdescConsensus = false;

- protected void parseStatusEntry(byte[] statusEntryBytes) + protected void parseStatusEntry(int offset, int length) throws DescriptorParseException { - NetworkStatusEntryImpl statusEntry = new NetworkStatusEntryImpl( - statusEntryBytes, this.microdescConsensus, + NetworkStatusEntryImpl statusEntry = new NetworkStatusEntryImpl(this, + offset, length, this.microdescConsensus, this.failUnrecognizedDescriptorLines); this.statusEntries.put(statusEntry.getFingerprint(), statusEntry); List<String> unrecognizedStatusEntryLines = statusEntry @@ -153,9 +137,9 @@ public class RelayNetworkStatusConsensusImpl extends NetworkStatusImpl } }

- protected void parseFooter(byte[] footerBytes) + protected void parseFooter(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(footerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); while (scanner.hasNext()) { String line = scanner.next(); String[] parts = line.split("[ \t]+"); diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java index cc84ebd..edc483a 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusImpl.java @@ -18,25 +18,11 @@ import java.util.TreeSet; public class RelayNetworkStatusImpl extends NetworkStatusImpl implements RelayNetworkStatus {

- protected static List<RelayNetworkStatus> parseStatuses( - byte[] statusesBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<RelayNetworkStatus> parsedStatuses = new ArrayList<>(); - List<byte[]> splitStatusBytes = - DescriptorImpl.splitRawDescriptorBytes(statusesBytes, - "network-status-version 2"); - for (byte[] statusBytes : splitStatusBytes) { - RelayNetworkStatus parsedStatus = new RelayNetworkStatusImpl( - statusBytes, failUnrecognizedDescriptorLines); - parsedStatuses.add(parsedStatus); - } - return parsedStatuses; - } - - protected RelayNetworkStatusImpl(byte[] statusBytes, + protected RelayNetworkStatusImpl(byte[] statusBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(statusBytes, failUnrecognizedDescriptorLines, false, true); + super(statusBytes, offsetAndLength, failUnrecognizedDescriptorLines, false, + true); Set<Key> exactlyOnceKeys = EnumSet.of( Key.NETWORK_STATUS_VERSION, Key.DIR_SOURCE, Key.FINGERPRINT, Key.CONTACT, Key.DIR_SIGNING_KEY, Key.PUBLISHED); @@ -50,9 +36,9 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl NL + Key.DIRECTORY_SIGNATURE.keyword + SP); }

- protected void parseHeader(byte[] headerBytes) + protected void parseHeader(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(headerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); Key nextCrypto = Key.EMPTY; StringBuilder crypto = null; while (scanner.hasNext()) { @@ -123,16 +109,15 @@ public class RelayNetworkStatusImpl extends NetworkStatusImpl } }

- protected void parseFooter(byte[] footerBytes) + protected void parseFooter(int offset, int length) throws DescriptorParseException { throw new DescriptorParseException("No directory footer expected in " + "v2 network status."); }

- protected void parseDirectorySignature(byte[] directorySignatureBytes) + protected void parseDirectorySignature(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(directorySignatureBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); Key nextCrypto = Key.EMPTY; StringBuilder crypto = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java index b694b5a..cad5c15 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImpl.java @@ -23,26 +23,11 @@ import java.util.TreeSet; public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl implements RelayNetworkStatusVote {

- protected static List<RelayNetworkStatusVote> parseVotes( - byte[] votesBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<RelayNetworkStatusVote> parsedVotes = new ArrayList<>(); - List<byte[]> splitVotesBytes = - DescriptorImpl.splitRawDescriptorBytes(votesBytes, - Key.NETWORK_STATUS_VERSION.keyword + SP + "3"); - for (byte[] voteBytes : splitVotesBytes) { - RelayNetworkStatusVote parsedVote = - new RelayNetworkStatusVoteImpl(voteBytes, - failUnrecognizedDescriptorLines); - parsedVotes.add(parsedVote); - } - return parsedVotes; - } - - protected RelayNetworkStatusVoteImpl(byte[] voteBytes, + protected RelayNetworkStatusVoteImpl(byte[] voteBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(voteBytes, failUnrecognizedDescriptorLines, false, false); + super(voteBytes, offsetAndLength, failUnrecognizedDescriptorLines, false, + false); Set<Key> exactlyOnceKeys = EnumSet.of( Key.VOTE_STATUS, Key.PUBLISHED, Key.VALID_AFTER, Key.FRESH_UNTIL, Key.VALID_UNTIL, Key.VOTING_DELAY, Key.KNOWN_FLAGS, Key.DIR_SOURCE, @@ -66,7 +51,7 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl NL + Key.DIRECTORY_SIGNATURE.keyword + SP); }

- protected void parseHeader(byte[] headerBytes) + protected void parseHeader(int offset, int length) throws DescriptorParseException { /* Initialize flag-thresholds values here for the case that the vote * doesn't contain those values. Initializing them in the constructor @@ -82,7 +67,7 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl this.enoughMtbfInfo = -1; this.ignoringAdvertisedBws = -1;

- Scanner scanner = new Scanner(new String(headerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); Key nextCrypto = Key.EMPTY; StringBuilder crypto = null; while (scanner.hasNext()) { @@ -596,9 +581,9 @@ public class RelayNetworkStatusVoteImpl extends NetworkStatusImpl } }

- protected void parseFooter(byte[] footerBytes) + protected void parseFooter(int offset, int length) throws DescriptorParseException { - Scanner scanner = new Scanner(new String(footerBytes)).useDelimiter(NL); + Scanner scanner = this.newScanner(offset, length).useDelimiter(NL); while (scanner.hasNext()) { String line = scanner.next(); if (!line.equals(Key.DIRECTORY_FOOTER.keyword)) { diff --git a/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java index eefa24f..b1d3f47 100644 --- a/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/RelayServerDescriptorImpl.java @@ -5,34 +5,14 @@ package org.torproject.descriptor.impl;

import org.torproject.descriptor.DescriptorParseException; import org.torproject.descriptor.RelayServerDescriptor; -import org.torproject.descriptor.ServerDescriptor; - -import java.util.ArrayList; -import java.util.List;

public class RelayServerDescriptorImpl extends ServerDescriptorImpl implements RelayServerDescriptor {

- protected static List<ServerDescriptor> parseDescriptors( - byte[] descriptorsBytes, boolean failUnrecognizedDescriptorLines) - throws DescriptorParseException { - List<ServerDescriptor> parsedDescriptors = new ArrayList<>(); - List<byte[]> splitDescriptorsBytes = - DescriptorImpl.splitRawDescriptorBytes(descriptorsBytes, - Key.ROUTER.keyword + SP); - for (byte[] descriptorBytes : splitDescriptorsBytes) { - ServerDescriptor parsedDescriptor = - new RelayServerDescriptorImpl(descriptorBytes, - failUnrecognizedDescriptorLines); - parsedDescriptors.add(parsedDescriptor); - } - return parsedDescriptors; - } - protected RelayServerDescriptorImpl(byte[] descriptorBytes, - boolean failUnrecognizedDescriptorLines) + int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines); + super(descriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines); } }

diff --git a/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java b/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java index 9241fcf..a4cf25e 100644 --- a/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/ServerDescriptorImpl.java @@ -34,10 +34,11 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl private static final Set<Key> exactlyOnce = EnumSet.of( Key.ROUTER, Key.BANDWIDTH, Key.PUBLISHED);

- protected ServerDescriptorImpl(byte[] descriptorBytes, + protected ServerDescriptorImpl(byte[] descriptorBytes, int[] offsetAndLength, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(descriptorBytes, failUnrecognizedDescriptorLines, false); + super(descriptorBytes, offsetAndLength, failUnrecognizedDescriptorLines, + false); this.parseDescriptorBytes(); this.calculateDigestSha1Hex(Key.ROUTER.keyword + SP, NL + Key.ROUTER_SIGNATURE.keyword + NL); @@ -56,8 +57,7 @@ public abstract class ServerDescriptorImpl extends DescriptorImpl }

private void parseDescriptorBytes() throws DescriptorParseException { - Scanner scanner = new Scanner(new String(this.rawDescriptorBytes)) - .useDelimiter(NL); + Scanner scanner = this.newScanner().useDelimiter(NL); Key nextCrypto = Key.EMPTY; List<String> cryptoLines = null; while (scanner.hasNext()) { diff --git a/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java b/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java index 16274c7..fb00a37 100644 --- a/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java +++ b/src/main/java/org/torproject/descriptor/impl/TorperfResultImpl.java @@ -26,6 +26,7 @@ public class TorperfResultImpl extends DescriptorImpl throw new DescriptorParseException("Descriptor is empty."); } List<Descriptor> parsedDescriptors = new ArrayList<>(); + /* XXX21932 */ String descriptorString = new String(rawDescriptorBytes); Scanner scanner = new Scanner(descriptorString).useDelimiter("\r?\n"); String typeAnnotation = ""; @@ -44,6 +45,7 @@ public class TorperfResultImpl extends DescriptorImpl } typeAnnotation = line + "\n"; } else { + /* XXX21932 */ parsedDescriptors.add(new TorperfResultImpl( (typeAnnotation + line).getBytes(), failUnrecognizedDescriptorLines)); @@ -56,17 +58,18 @@ public class TorperfResultImpl extends DescriptorImpl protected TorperfResultImpl(byte[] rawDescriptorBytes, boolean failUnrecognizedDescriptorLines) throws DescriptorParseException { - super(rawDescriptorBytes, failUnrecognizedDescriptorLines, false); - this.parseTorperfResultLine(new String(rawDescriptorBytes)); + super(rawDescriptorBytes, new int[] { 0, rawDescriptorBytes.length }, + failUnrecognizedDescriptorLines, false); + this.parseTorperfResultLine(); }

- private void parseTorperfResultLine(String inputLine) + private void parseTorperfResultLine() throws DescriptorParseException { - String line = inputLine; - while (line.startsWith("@") && line.contains("\n")) { + String line = this.newScanner().nextLine(); + while (null != line && line.startsWith("@") && line.contains("\n")) { line = line.split("\n")[1]; } - if (line.isEmpty()) { + if (null == line || line.isEmpty()) { throw new DescriptorParseException("Blank lines are not allowed."); } String[] parts = line.split(" "); diff --git a/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java b/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java index 8720d7b..25db3d0 100644 --- a/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java +++ b/src/test/java/org/torproject/descriptor/impl/BridgeNetworkStatusTest.java @@ -30,8 +30,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.fileName = fileName; - return new BridgeNetworkStatusImpl(sb.buildStatus(), sb.fileName, - true); + return sb.buildStatus(true); }

private String publishedLine = "published 2015-11-21 17:39:36"; @@ -41,8 +40,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.publishedLine = line; - return new BridgeNetworkStatusImpl(sb.buildStatus(), sb.fileName, - true); + return sb.buildStatus(true); }

private String flagThresholdsLine = "flag-thresholds " @@ -56,8 +54,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.flagThresholdsLine = line; - return new BridgeNetworkStatusImpl(sb.buildStatus(), sb.fileName, - true); + return sb.buildStatus(true); }

private List<String> statusEntries = new ArrayList<>(); @@ -70,8 +67,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.unrecognizedHeaderLine = line; - return new BridgeNetworkStatusImpl(sb.buildStatus(), sb.fileName, - failUnrecognizedDescriptorLines); + return sb.buildStatus(failUnrecognizedDescriptorLines); }

private String unrecognizedStatusEntryLine = null; @@ -82,8 +78,7 @@ public class BridgeNetworkStatusTest { throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); sb.unrecognizedStatusEntryLine = line; - return new BridgeNetworkStatusImpl(sb.buildStatus(), sb.fileName, - failUnrecognizedDescriptorLines); + return sb.buildStatus(failUnrecognizedDescriptorLines); }

private StatusBuilder() { @@ -93,13 +88,22 @@ public class BridgeNetworkStatusTest { + "w Bandwidth=264\np reject 1-65535"); }

- private byte[] buildStatus() { + private byte[] buildStatusBytes() { StringBuilder sb = new StringBuilder(); this.appendHeader(sb); this.appendStatusEntries(sb); return sb.toString().getBytes(); }

+ private BridgeNetworkStatus buildStatus( + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + byte[] statusBytes = this.buildStatusBytes(); + return new BridgeNetworkStatusImpl(statusBytes, + new int[] { 0, statusBytes.length }, this.fileName, + failUnrecognizedDescriptorLines); + } + private void appendHeader(StringBuilder sb) { if (this.publishedLine != null) { sb.append(this.publishedLine).append("\n"); @@ -125,8 +129,7 @@ public class BridgeNetworkStatusTest { @Test() public void testSampleStatus() throws DescriptorParseException { StatusBuilder sb = new StatusBuilder(); - BridgeNetworkStatus status = - new BridgeNetworkStatusImpl(sb.buildStatus(), sb.fileName, true); + BridgeNetworkStatus status = sb.buildStatus(true); assertEquals(1448127576000L, status.getPublishedMillis()); assertEquals(3105080L, status.getStableUptime()); assertEquals(2450615L, status.getStableMtbf()); diff --git a/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java b/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java index d1cacc0..e0d0c99 100644 --- a/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java +++ b/src/test/java/org/torproject/descriptor/impl/ConsensusBuilder.java @@ -20,7 +20,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.networkStatusVersionLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String voteStatusLine = "vote-status consensus"; @@ -30,7 +30,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.voteStatusLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String consensusMethodLine = "consensus-method 11"; @@ -40,7 +40,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.consensusMethodLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String validAfterLine = "valid-after 2011-11-30 09:00:00"; @@ -50,7 +50,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.validAfterLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String freshUntilLine = "fresh-until 2011-11-30 10:00:00"; @@ -60,7 +60,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.freshUntilLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String validUntilLine = "valid-until 2011-11-30 12:00:00"; @@ -70,7 +70,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.validUntilLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String votingDelayLine = "voting-delay 300 300"; @@ -80,7 +80,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.votingDelayLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

String clientVersionsLine = "client-versions 0.2.1.31," @@ -91,7 +91,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.clientVersionsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

String serverVersionsLine = "server-versions 0.2.1.31," @@ -102,7 +102,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.serverVersionsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String packageLines = null; @@ -112,7 +112,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.packageLines = lines; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String knownFlagsLine = "known-flags Authority BadExit Exit " @@ -123,7 +123,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.knownFlagsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String recommendedClientProtocolsLine = @@ -135,7 +135,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.recommendedClientProtocolsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String recommendedRelayProtocolsLine = @@ -147,7 +147,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.recommendedRelayProtocolsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String requiredClientProtocolsLine = @@ -159,7 +159,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.requiredClientProtocolsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String requiredRelayProtocolsLine = @@ -171,7 +171,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.requiredRelayProtocolsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String paramsLine = "params " @@ -185,7 +185,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.paramsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String sharedRandPreviousValueLine = @@ -197,7 +197,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.sharedRandPreviousValueLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String sharedRandCurrentValueLine = @@ -209,7 +209,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.sharedRandCurrentValueLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

List<String> dirSources = new ArrayList<>(); @@ -227,7 +227,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.directoryFooterLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private String bandwidthWeightsLine = "bandwidth-weights Wbd=285 " @@ -244,7 +244,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.bandwidthWeightsLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + return cb.buildConsensus(true); }

private List<String> directorySignatures = new ArrayList<>(); @@ -261,8 +261,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedHeaderLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - failUnrecognizedDescriptorLines); + return cb.buildConsensus(failUnrecognizedDescriptorLines); }

private String unrecognizedDirSourceLine = null; @@ -273,8 +272,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedDirSourceLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - failUnrecognizedDescriptorLines); + return cb.buildConsensus(failUnrecognizedDescriptorLines); }

private String unrecognizedStatusEntryLine = null; @@ -285,8 +283,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedStatusEntryLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - failUnrecognizedDescriptorLines); + return cb.buildConsensus(failUnrecognizedDescriptorLines); }

private String unrecognizedFooterLine = null; @@ -297,8 +294,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedFooterLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - failUnrecognizedDescriptorLines); + return cb.buildConsensus(failUnrecognizedDescriptorLines); }

private String unrecognizedDirectorySignatureLine = null; @@ -309,8 +305,7 @@ public class ConsensusBuilder { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.unrecognizedDirectorySignatureLine = line; - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - failUnrecognizedDescriptorLines); + return cb.buildConsensus(failUnrecognizedDescriptorLines); }

protected ConsensusBuilder() { @@ -350,7 +345,7 @@ public class ConsensusBuilder { + "-----END SIGNATURE-----"); }

- protected byte[] buildConsensus() { + protected byte[] buildConsensusBytes() { StringBuilder sb = new StringBuilder(); this.appendHeader(sb); this.appendDirSources(sb); @@ -360,6 +355,15 @@ public class ConsensusBuilder { return sb.toString().getBytes(); }

+ protected RelayNetworkStatusConsensus buildConsensus( + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + byte[] consensusBytes = this.buildConsensusBytes(); + return new RelayNetworkStatusConsensusImpl(consensusBytes, + new int[] { 0, consensusBytes.length }, + failUnrecognizedDescriptorLines); + } + private void appendHeader(StringBuilder sb) { if (this.networkStatusVersionLine != null) { sb.append(this.networkStatusVersionLine).append("\n"); diff --git a/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java index d1e38d6..c47c873 100644 --- a/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ExtraInfoDescriptorImplTest.java @@ -43,7 +43,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.extraInfoLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String publishedLine = "published 2012-02-11 09:08:36"; @@ -52,7 +52,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.publishedLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String writeHistoryLine = "write-history 2012-02-11 09:03:39 " @@ -62,7 +62,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.writeHistoryLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String readHistoryLine = "read-history 2012-02-11 09:03:39 " @@ -72,7 +72,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.readHistoryLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String dirreqWriteHistoryLine = "dirreq-write-history " @@ -83,7 +83,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.dirreqWriteHistoryLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String dirreqReadHistoryLine = "dirreq-read-history " @@ -94,7 +94,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.dirreqReadHistoryLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String geoipDbDigestLine = null; @@ -103,7 +103,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.geoipDbDigestLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String geoip6DbDigestLine = null; @@ -112,7 +112,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.geoip6DbDigestLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String geoipStatsLines = null; @@ -121,7 +121,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.geoipStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String dirreqStatsLines = null; @@ -130,7 +130,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.dirreqStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String entryStatsLines = null; @@ -139,7 +139,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.entryStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String cellStatsLines = null; @@ -148,7 +148,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.cellStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String connBiDirectLine = null; @@ -157,7 +157,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.connBiDirectLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String exitStatsLines = null; @@ -166,7 +166,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.exitStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String bridgeStatsLines = null; @@ -175,7 +175,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.bridgeStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String hidservStatsLines = null; @@ -184,7 +184,7 @@ public class ExtraInfoDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.hidservStatsLines = lines; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String paddingCountsLine = null; @@ -193,7 +193,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.paddingCountsLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String unrecognizedLine = null; @@ -203,8 +203,7 @@ public class ExtraInfoDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.unrecognizedLine = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), - failUnrecognizedDescriptorLines); + return db.buildDescriptor(failUnrecognizedDescriptorLines); }

private byte[] nonAsciiLineBytes = null; @@ -214,8 +213,7 @@ public class ExtraInfoDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.nonAsciiLineBytes = lineBytes; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), - failUnrecognizedDescriptorLines); + return db.buildDescriptor(failUnrecognizedDescriptorLines); }

private String routerSignatureLines = "router-signature\n" @@ -229,7 +227,7 @@ public class ExtraInfoDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.routerSignatureLines = line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String identityEd25519Lines = null; @@ -245,10 +243,10 @@ public class ExtraInfoDescriptorImplTest { db.identityEd25519Lines = identityEd25519Lines; db.masterKeyEd25519Line = masterKeyEd25519Line; db.routerSigEd25519Line = routerSigEd25519Line; - return new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

- private byte[] buildDescriptor() { + private byte[] buildDescriptorBytes() { StringBuilder sb = new StringBuilder(); if (this.extraInfoLine != null) { sb.append(this.extraInfoLine).append("\n"); @@ -332,6 +330,15 @@ public class ExtraInfoDescriptorImplTest { } return sb.toString().getBytes(); } + + private ExtraInfoDescriptor buildDescriptor( + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + byte[] descriptorBytes = this.buildDescriptorBytes(); + return new RelayExtraInfoDescriptorImpl(descriptorBytes, + new int[] { 0, descriptorBytes.length}, + failUnrecognizedDescriptorLines); + } }

/* Helper class to build a set of geoip-stats lines based on default @@ -947,8 +954,7 @@ public class ExtraInfoDescriptorImplTest { @Test() public void testSampleDescriptor() throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); - ExtraInfoDescriptor descriptor = - new RelayExtraInfoDescriptorImpl(db.buildDescriptor(), true); + ExtraInfoDescriptor descriptor = db.buildDescriptor(true); assertEquals("chaoscomputerclub5", descriptor.getNickname()); assertEquals("A9C039A5FD02FCA06303DCFAABE25C5912C63B26", descriptor.getFingerprint()); @@ -2130,7 +2136,8 @@ public class ExtraInfoDescriptorImplTest { + "-----END SIGNATURE-----\n" + "").getBytes(); RelayExtraInfoDescriptor descriptor = - new RelayExtraInfoDescriptorImpl(descriptorBytes, true); + new RelayExtraInfoDescriptorImpl(descriptorBytes, + new int[] { 0, descriptorBytes.length }, true); assertEquals("Pt1BtzfRwhYqGCDo8jjchS8nJP3ovrDyHGn+dqPbMgw", descriptor.getDigestSha256Base64()); } @@ -2154,7 +2161,8 @@ public class ExtraInfoDescriptorImplTest { + "router-digest 00B98F076B586272C3172B7F3DA29ADEE75F2ED8\n") .getBytes(); BridgeExtraInfoDescriptor descriptor = - new BridgeExtraInfoDescriptorImpl(descriptorBytes, true); + new BridgeExtraInfoDescriptorImpl(descriptorBytes, + new int[] { 0, descriptorBytes.length }, true); assertEquals("TvrqpjI7OmCtwGwair/NHUxg5ROVVQYz6/EDyXsDHR4", descriptor.getDigestSha256Base64()); } diff --git a/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java index d9040f3..140974c 100644 --- a/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/MicrodescriptorImplTest.java @@ -27,7 +27,7 @@ public class MicrodescriptorImplTest { private static Microdescriptor createWithDefaultLines() throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); - return new MicrodescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String ntorOnionKeyLine = @@ -39,10 +39,10 @@ public class MicrodescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.idLine = line; - return new MicrodescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

- private byte[] buildDescriptor() { + private byte[] buildDescriptorBytes() { StringBuilder sb = new StringBuilder(); if (this.onionKeyLines != null) { sb.append(this.onionKeyLines).append("\n"); @@ -55,6 +55,15 @@ public class MicrodescriptorImplTest { } return sb.toString().getBytes(); } + + private Microdescriptor buildDescriptor( + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + byte[] descriptorBytes = this.buildDescriptorBytes(); + return new MicrodescriptorImpl(descriptorBytes, + new int[] { 0, descriptorBytes.length }, + failUnrecognizedDescriptorLines); + } }

@Test() diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java index 6298668..d6bf4cf 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusConsensusImplTest.java @@ -38,8 +38,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.dirSources.add(dirSourceString); - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - true); + return cb.buildConsensus(true); }

private String nickname = "gabelmoo"; @@ -152,8 +151,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(statusEntryString); - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - true); + return cb.buildConsensus(true); }

private String nickname = "right2privassy3"; @@ -311,8 +309,7 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.addDirectorySignature(directorySignatureString); - return new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), - true); + return cb.buildConsensus(true); }

private String identity = "ED03BB616EB2F60BEC80151114BB25CEF515B226"; @@ -353,8 +350,7 @@ public class RelayNetworkStatusConsensusImplTest { @Test() public void testSampleConsensus() throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); - RelayNetworkStatusConsensus consensus = - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); assertEquals(3, consensus.getNetworkStatusVersion()); assertEquals(11, consensus.getConsensusMethod()); assertEquals(1322643600000L, consensus.getValidAfterMillis()); @@ -636,8 +632,7 @@ public class RelayNetworkStatusConsensusImplTest { ConsensusBuilder cb = new ConsensusBuilder(); cb.clientVersionsLine = null; cb.serverVersionsLine = null; - RelayNetworkStatusConsensus consensus = - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); assertNull(consensus.getRecommendedClientVersions()); assertNull(consensus.getRecommendedServerVersions()); } @@ -1103,7 +1098,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.sLine = sb.sLine + "\n" + sb.sLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + cb.buildConsensus(true); }

@Test(expected = DescriptorParseException.class) @@ -1112,7 +1107,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.prLine = sb.prLine + "\n" + sb.prLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + cb.buildConsensus(true); }

@Test(expected = DescriptorParseException.class) @@ -1136,7 +1131,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.wLine = sb.wLine + "\n" + sb.wLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + cb.buildConsensus(true); }

@Test() @@ -1145,8 +1140,7 @@ public class RelayNetworkStatusConsensusImplTest { sb.wLine = "w Bandwidth=42424242 Unmeasured=1"; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - RelayNetworkStatusConsensus consensus = - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); for (NetworkStatusEntry s : consensus.getStatusEntries().values()) { if (s.getBandwidth() == 42424242L) { assertTrue(s.getUnmeasured()); @@ -1189,15 +1183,14 @@ public class RelayNetworkStatusConsensusImplTest { sb.pLine = sb.pLine + "\n" + sb.pLine; ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.add(sb.buildStatusEntry()); - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + cb.buildConsensus(true); }

@Test() public void testNoStatusEntries() throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.statusEntries.clear(); - RelayNetworkStatusConsensus consensus = - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); assertFalse(consensus.containsStatusEntry( "00795A6E8D91C270FC23B30F388A495553E01894")); } @@ -1218,8 +1211,7 @@ public class RelayNetworkStatusConsensusImplTest { cb.setBandwidthWeightsLine(null); /* This does not break, because directory footers were optional before * consensus method 9. */ - RelayNetworkStatusConsensus consensus = - new RelayNetworkStatusConsensusImpl(cb.buildConsensus(), true); + RelayNetworkStatusConsensus consensus = cb.buildConsensus(true); assertNull(consensus.getBandwidthWeights()); }

@@ -1305,9 +1297,10 @@ public class RelayNetworkStatusConsensusImplTest { @Test(expected = DescriptorParseException.class) public void testNonAsciiByte20() throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); - byte[] consensusBytes = cb.buildConsensus(); + byte[] consensusBytes = cb.buildConsensusBytes(); consensusBytes[20] = (byte) 200; - new RelayNetworkStatusConsensusImpl(consensusBytes, true); + new RelayNetworkStatusConsensusImpl(consensusBytes, + new int[] { 0, consensusBytes.length }, true); }

@Test(expected = DescriptorParseException.class) @@ -1315,9 +1308,10 @@ public class RelayNetworkStatusConsensusImplTest { throws DescriptorParseException { ConsensusBuilder cb = new ConsensusBuilder(); cb.networkStatusVersionLine = "Xnetwork-status-version 3"; - byte[] consensusBytes = cb.buildConsensus(); + byte[] consensusBytes = cb.buildConsensusBytes(); consensusBytes[0] = (byte) 200; - new RelayNetworkStatusConsensusImpl(consensusBytes, true); + new RelayNetworkStatusConsensusImpl(consensusBytes, + new int[] { 0, consensusBytes.length }, true); }

@Test(expected = DescriptorParseException.class) diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java index cccc6f7..d0ff7fb 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusImplTest.java @@ -11,6 +11,8 @@ import org.junit.Test;

public class RelayNetworkStatusImplTest {

+ private static final String validAnnotation = "@type network-status-2 1.0\n"; + private static final String validHeader = "network-status-version 2\n" + "dir-source 194.109.206.212 194.109.206.212 80\n" + "fingerprint 7EA6EAD6FD83083C538F44038BBFA077587DD755\n" @@ -32,20 +34,22 @@ public class RelayNetworkStatusImplTest { + "-----END SIGNATURE-----\n";

private static final String validStatus = - "@type network-status-2 1.0\n" + validHeader + validFooter; + validAnnotation + validHeader + validFooter;

@Test(expected = DescriptorParseException.class) public void testParseBrokenHeader() throws DescriptorParseException { - RelayNetworkStatusImpl rnsi - = new RelayNetworkStatusImpl(validStatus.getBytes(), true); - rnsi.parseHeader("network-status-version 2\nxyx\nabc".getBytes()); + String invalidHeader = "network-status-version 2\nxyx\nabc"; + byte[] statusBytes = (validAnnotation + invalidHeader + validFooter) + .getBytes(); + new RelayNetworkStatusImpl(statusBytes, new int[] { 0, statusBytes.length }, + true); }

@Test() public void testValidHeader() throws DescriptorParseException { - RelayNetworkStatusImpl rnsi = - new RelayNetworkStatusImpl(validStatus.getBytes(), true); - rnsi.parseHeader(validHeader.getBytes()); + byte[] statusBytes = validStatus.getBytes(); + RelayNetworkStatusImpl rnsi = new RelayNetworkStatusImpl(statusBytes, + new int[] { 0, statusBytes.length }, true); assertEquals(rnsi.getContactLine(), "1024R/8D56913D Alex de Joode adejoode@sabotage.org"); } diff --git a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java index 4a61f1c..aa7ff4a 100644 --- a/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/RelayNetworkStatusVoteImplTest.java @@ -37,7 +37,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.networkStatusVersionLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String voteStatusLine = "vote-status vote"; @@ -47,7 +47,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.voteStatusLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String consensusMethodsLine = @@ -58,7 +58,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.consensusMethodsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String publishedLine = "published 2011-11-30 08:50:01"; @@ -68,7 +68,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.publishedLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String validAfterLine = "valid-after 2011-11-30 09:00:00"; @@ -78,7 +78,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.validAfterLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String freshUntilLine = "fresh-until 2011-11-30 10:00:00"; @@ -88,7 +88,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.freshUntilLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String validUntilLine = "valid-until 2011-11-30 12:00:00"; @@ -98,7 +98,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.validUntilLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String votingDelayLine = "voting-delay 300 300"; @@ -108,7 +108,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.votingDelayLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String clientVersionsLine = "client-versions 0.2.1.31," @@ -119,7 +119,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.clientVersionsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String serverVersionsLine = "server-versions 0.2.1.31," @@ -130,7 +130,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.serverVersionsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String packageLines = null; @@ -140,7 +140,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.packageLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String knownFlagsLine = "known-flags Authority BadExit Exit " @@ -151,7 +151,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.knownFlagsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String recommendedClientProtocolsLine = @@ -163,7 +163,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.recommendedClientProtocolsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String recommendedRelayProtocolsLine = @@ -175,7 +175,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.recommendedRelayProtocolsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String requiredClientProtocolsLine = @@ -187,7 +187,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.requiredClientProtocolsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String requiredRelayProtocolsLine = @@ -199,7 +199,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.requiredRelayProtocolsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String flagThresholdsLine = "flag-thresholds " @@ -212,7 +212,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.flagThresholdsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String paramsLine = "params " @@ -226,7 +226,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.paramsLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirSourceLine = "dir-source urras " @@ -238,7 +238,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirSourceLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String contactLine = "contact 4096R/E012B42D Jacob Appelbaum " @@ -249,7 +249,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.contactLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String sharedRandParticipateLine = "shared-rand-participate"; @@ -258,7 +258,7 @@ public class RelayNetworkStatusVoteImplTest { String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandParticipateLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private List<String> sharedRandCommitLines = Arrays.asList(new String[] { @@ -273,7 +273,7 @@ public class RelayNetworkStatusVoteImplTest { List<String> lines) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandCommitLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String sharedRandPreviousValueLine = @@ -284,7 +284,7 @@ public class RelayNetworkStatusVoteImplTest { String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandPreviousValueLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String sharedRandCurrentValueLine = @@ -295,7 +295,7 @@ public class RelayNetworkStatusVoteImplTest { String line) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.sharedRandCurrentValueLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String legacyDirKeyLine = null; @@ -305,7 +305,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.legacyDirKeyLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirKeyCertificateVersionLine = @@ -316,7 +316,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyCertificateVersionLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String fingerprintLine = "fingerprint " @@ -327,7 +327,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.fingerprintLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirKeyPublishedLine = "dir-key-published 2011-04-27 " @@ -338,7 +338,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyPublishedLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirKeyExpiresLine = "dir-key-expires 2012-04-27 " @@ -349,7 +349,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyExpiresLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirIdentityKeyLines = "dir-identity-key\n" @@ -370,7 +370,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirIdentityKeyLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirSigningKeyLines = "dir-signing-key\n" @@ -386,7 +386,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirSigningKeyLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirKeyCrosscertLines = "dir-key-crosscert\n" @@ -401,7 +401,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyCrosscertLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String dirKeyCertificationLines = "dir-key-certification\n" @@ -422,7 +422,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.dirKeyCertificationLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private List<String> statusEntries = null; @@ -431,7 +431,7 @@ public class RelayNetworkStatusVoteImplTest { List<String> statusEntries) throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.statusEntries = statusEntries; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String directoryFooterLine = "directory-footer"; @@ -441,7 +441,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.directoryFooterLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String directorySignatureLines = "directory-signature " @@ -458,7 +458,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.directorySignatureLines = lines; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + return vb.buildVote(true); }

private String unrecognizedHeaderLine = null; @@ -469,8 +469,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedHeaderLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), - failUnrecognizedDescriptorLines); + return vb.buildVote(failUnrecognizedDescriptorLines); }

private String unrecognizedDirSourceLine = null; @@ -481,8 +480,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedDirSourceLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), - failUnrecognizedDescriptorLines); + return vb.buildVote(failUnrecognizedDescriptorLines); }

private String unrecognizedStatusEntryLine = null; @@ -493,8 +491,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedStatusEntryLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), - failUnrecognizedDescriptorLines); + return vb.buildVote(failUnrecognizedDescriptorLines); }

private String unrecognizedFooterLine = null; @@ -505,8 +502,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedFooterLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), - failUnrecognizedDescriptorLines); + return vb.buildVote(failUnrecognizedDescriptorLines); }

private String unrecognizedDirectorySignatureLine = null; @@ -517,8 +513,7 @@ public class RelayNetworkStatusVoteImplTest { throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); vb.unrecognizedDirectorySignatureLine = line; - return new RelayNetworkStatusVoteImpl(vb.buildVote(), - failUnrecognizedDescriptorLines); + return vb.buildVote(failUnrecognizedDescriptorLines); }

private VoteBuilder() { @@ -541,7 +536,7 @@ public class RelayNetworkStatusVoteImplTest { + "sha256=9ciEx9t0McXk9A06I7qwN7pxuNOdpCP64RV/6cx2Zkc"); }

- private byte[] buildVote() { + private byte[] buildVoteBytes() { StringBuilder sb = new StringBuilder(); this.appendHeader(sb); this.appendDirSource(sb); @@ -551,6 +546,14 @@ public class RelayNetworkStatusVoteImplTest { return sb.toString().getBytes(); }

+ private RelayNetworkStatusVoteImpl buildVote( + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + byte[] voteBytes = this.buildVoteBytes(); + return new RelayNetworkStatusVoteImpl(voteBytes, + new int[] { 0, voteBytes.length }, failUnrecognizedDescriptorLines); + } + private void appendHeader(StringBuilder sb) { if (this.networkStatusVersionLine != null) { sb.append(this.networkStatusVersionLine).append("\n"); @@ -695,8 +698,7 @@ public class RelayNetworkStatusVoteImplTest { @Test() public void testSampleVote() throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); - RelayNetworkStatusVote vote = - new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + RelayNetworkStatusVote vote = vb.buildVote(true); assertEquals(3, vote.getNetworkStatusVersion()); List<Integer> consensusMethods = Arrays.asList( new Integer[] {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}); @@ -1081,8 +1083,7 @@ public class RelayNetworkStatusVoteImplTest { @Test() public void testFlagThresholdsLine() throws DescriptorParseException { VoteBuilder vb = new VoteBuilder(); - RelayNetworkStatusVote vote = - new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + RelayNetworkStatusVote vote = vb.buildVote(true); assertEquals(693369L, vote.getStableUptime()); assertEquals(153249L, vote.getStableMtbf()); assertEquals(40960L, vote.getFastBandwidth()); @@ -1143,7 +1144,7 @@ public class RelayNetworkStatusVoteImplTest { VoteBuilder vb = new VoteBuilder(); vb.flagThresholdsLine = vb.flagThresholdsLine + "\n" + vb.flagThresholdsLine; - new RelayNetworkStatusVoteImpl(vb.buildVote(), true); + vb.buildVote(true); }

@Test(expected = DescriptorParseException.class) @@ -1521,7 +1522,6 @@ public class RelayNetworkStatusVoteImplTest { secondSignature.getSigningKeyDigestSha1Hex()); assertEquals(signatureSha1 + "\n", secondSignature.getSignature()); assertEquals(signingKeyDigestSha1, vote.getSigningKeyDigest()); - System.out.println(new String(vote.getRawDescriptorBytes())); assertEquals("c0d58c8d3c3695526f6eb5c0d9f8452b2234d303", vote.getDigestSha1Hex()); } diff --git a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java index 6cfb024..71ce810 100644 --- a/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java +++ b/src/test/java/org/torproject/descriptor/impl/ServerDescriptorImplTest.java @@ -37,7 +37,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.routerLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String bandwidthLine = "bandwidth 51200 51200 53470"; @@ -46,7 +46,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.bandwidthLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String platformLine = "platform Tor 0.2.2.35 " @@ -56,7 +56,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.platformLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String publishedLine = "published 2012-01-01 04:03:19"; @@ -65,7 +65,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.publishedLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String fingerprintLine = "opt fingerprint D873 3048 FC8E " @@ -75,7 +75,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.fingerprintLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String hibernatingLine = null; @@ -84,7 +84,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.hibernatingLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String uptimeLine = "uptime 48"; @@ -93,7 +93,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.uptimeLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String onionKeyLines = "onion-key\n" @@ -108,7 +108,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.onionKeyLines = lines; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String signingKeyLines = "signing-key\n" @@ -123,7 +123,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.signingKeyLines = lines; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String onionKeyCrosscertLines = null; @@ -132,7 +132,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.onionKeyCrosscertLines = lines; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String ntorOnionKeyCrosscertLines = null; @@ -141,7 +141,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.ntorOnionKeyCrosscertLines = lines; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String exitPolicyLines = "reject *:*"; @@ -150,7 +150,7 @@ public class ServerDescriptorImplTest { String lines) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.exitPolicyLines = lines; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String contactLine = "contact Random Person <nobody AT " @@ -160,7 +160,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.contactLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String familyLine = null; @@ -169,7 +169,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.familyLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String readHistoryLine = null; @@ -178,7 +178,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.readHistoryLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String writeHistoryLine = null; @@ -187,7 +187,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.writeHistoryLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String eventdnsLine = null; @@ -196,7 +196,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.eventdnsLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String cachesExtraInfoLine = null; @@ -205,7 +205,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.cachesExtraInfoLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String extraInfoDigestLine = "opt extra-info-digest " @@ -215,7 +215,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.extraInfoDigestLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String hiddenServiceDirLine = "opt hidden-service-dir"; @@ -224,7 +224,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.hiddenServiceDirLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String protocolsLine = null; @@ -233,7 +233,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.protocolsLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String protoLine = "proto Cons=1-2 Desc=1-2 DirCache=1 HSDir=1 " @@ -243,7 +243,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.protoLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String allowSingleHopExitsLine = null; @@ -253,7 +253,7 @@ public class ServerDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.allowSingleHopExitsLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String ipv6PolicyLine = null; @@ -262,7 +262,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.ipv6PolicyLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String ntorOnionKeyLine = null; @@ -271,7 +271,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.ntorOnionKeyLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String tunnelledDirServerLine = null; @@ -280,7 +280,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.tunnelledDirServerLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String routerSignatureLines = "router-signature\n" @@ -294,7 +294,7 @@ public class ServerDescriptorImplTest { String line) throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.routerSignatureLines = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

private String unrecognizedLine = null; @@ -304,8 +304,7 @@ public class ServerDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.unrecognizedLine = line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), - failUnrecognizedDescriptorLines); + return db.buildDescriptor(failUnrecognizedDescriptorLines); }

private byte[] nonAsciiLineBytes = null; @@ -315,8 +314,7 @@ public class ServerDescriptorImplTest { throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); db.nonAsciiLineBytes = lineBytes; - return new RelayServerDescriptorImpl(db.buildDescriptor(), - failUnrecognizedDescriptorLines); + return db.buildDescriptor(failUnrecognizedDescriptorLines); }

private String identityEd25519Lines = null; @@ -332,10 +330,10 @@ public class ServerDescriptorImplTest { db.identityEd25519Lines = identityEd25519Lines; db.masterKeyEd25519Line = masterKeyEd25519Line; db.routerSigEd25519Line = routerSigEd25519Line; - return new RelayServerDescriptorImpl(db.buildDescriptor(), true); + return db.buildDescriptor(true); }

- private byte[] buildDescriptor() { + private byte[] buildDescriptorBytes() { StringBuilder sb = new StringBuilder(); if (this.routerLine != null) { sb.append(this.routerLine).append("\n"); @@ -446,13 +444,21 @@ public class ServerDescriptorImplTest { } return sb.toString().getBytes(); } + + private ServerDescriptorImpl buildDescriptor( + boolean failUnrecognizedDescriptorLines) + throws DescriptorParseException { + byte[] descriptorBytes = this.buildDescriptorBytes(); + return new RelayServerDescriptorImpl(descriptorBytes, + new int[] { 0, descriptorBytes.length }, + failUnrecognizedDescriptorLines); + } }

@Test() public void testSampleDescriptor() throws DescriptorParseException { DescriptorBuilder db = new DescriptorBuilder(); - ServerDescriptor descriptor = - new RelayServerDescriptorImpl(db.buildDescriptor(), true); + ServerDescriptor descriptor = db.buildDescriptor(true); assertEquals("saberrider2008", descriptor.getNickname()); assertEquals("94.134.192.243", descriptor.getAddress()); assertEquals(9001, (int) descriptor.getOrPort());