tor-commits May 2018

tor-commits@lists.torproject.org

17 participants
1514 discussions

[collector/release] Bump version to 1.5.1-dev.
by karsten＠torproject.org 25 May '18

25 May '18

commit 88867bf3b4e880478f8392d7f593dbd5ec6c38a2 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Mar 19 16:26:13 2018 +0100 Bump version to 1.5.1-dev. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index d7372bc..780f24e 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.5.1" /> + <property name="release.version" value="1.5.1-dev" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.2.0" />

1 0

[collector/release] Include webstats in create-tarballs.sh.
by karsten＠torproject.org 25 May '18

25 May '18

commit 4e166b7832a8026690950ca8f8e971998b3a0bcb Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Mon Mar 19 20:59:34 2018 +0100 Include webstats in create-tarballs.sh. --- src/main/resources/create-tarballs.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources/create-tarballs.sh index c20303e..cd16b2d 100755 --- a/src/main/resources/create-tarballs.sh +++ b/src/main/resources/create-tarballs.sh @@ -105,8 +105,15 @@ for (( i = 0 ; i < ${#TARBALLS[@]} ; i++ )); do fi done +cd $OUTDIR/webstats/ +echo `date` "Creating webstats-"$YEARONE"-"$MONTHONE".tar" +find . -type f | grep "/"$YEARONE"/"$MONTHONE"/" | cut -c3- | tar cf $WORKDIR/webstats-$YEARONE-$MONTHONE.tar --transform 's,^,webstats-'$YEARONE'-'$MONTHONE'/,' -T - +echo `date` "Creating webstats-"$YEARTWO"-"$MONTHTWO".tar" +find . -type f | grep "/"$YEARTWO"/"$MONTHTWO"/" | cut -c3- | tar cf $WORKDIR/webstats-$YEARTWO-$MONTHTWO.tar --transform 's,^,webstats-'$YEARTWO'-'$MONTHTWO'/,' -T - +cd $WORKDIR + echo `date` "Moving tarballs into place" -mv *.tar.xz $TARBALLTARGETDIR +mv *.tar* $TARBALLTARGETDIR cd $CURRENTPATH @@ -154,4 +161,7 @@ ln -f -s -t $ARCHIVEDIR/relay-descriptors/votes/ $TARBALLTARGETDIR/votes-20??-?? mkdir -p $ARCHIVEDIR/torperf/ ln -f -s -t $ARCHIVEDIR/torperf/ $TARBALLTARGETDIR/torperf-20??-??.tar.xz +mkdir -p $ARCHIVEDIR/webstats/ +ln -f -s -t $ARCHIVEDIR/webstats/ $TARBALLTARGETDIR/webstats-20??-??.tar + echo `date` "Finished."

1 0

[collector/release] Corrected description of vote descriptor path.
by karsten＠torproject.org 25 May '18

25 May '18

commit 770f55cf82a1281e5081de7731adc9e36a916622 Author: iwakeh <iwakeh(a)torproject.org> Date: Mon Feb 26 15:24:50 2018 +0000 Corrected description of vote descriptor path. Implements task-20287. --- src/main/resources/docs/PROTOCOL | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/main/resources/docs/PROTOCOL b/src/main/resources/docs/PROTOCOL index c84c286..2c8b603 100644 --- a/src/main/resources/docs/PROTOCOL +++ b/src/main/resources/docs/PROTOCOL @@ -209,7 +209,8 @@ Where VOTE is the string "vote" and all time related values are derived from the valid-after dates. 'fingerprint' is the fingerprint of the authority and 'digest' is the SHA1 - digest of the authority's medium term signing key. + digest for the vote document, i.e., the digest of the descriptor + bytes from the start of the vote to the 'directory-signature'. 4.3.4 'microdescs' contains the subfolders

1 0

[collector/release] Describe 'contrib' path contents.
by karsten＠torproject.org 25 May '18

25 May '18

commit 970dd3d1e1757d01b485fe05b1869cfc244e82f2 Author: iwakeh <iwakeh(a)torproject.org> Date: Tue Mar 20 08:31:46 2018 +0000 Describe 'contrib' path contents. --- src/main/resources/docs/PROTOCOL | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/main/resources/docs/PROTOCOL b/src/main/resources/docs/PROTOCOL index 2c8b603..9f65e43 100644 --- a/src/main/resources/docs/PROTOCOL +++ b/src/main/resources/docs/PROTOCOL @@ -392,7 +392,19 @@ 'webstats' contains compressed log files structured and named according to the 'Tor web server logs' specification, section 4.3 [0]. +6.0 The 'contrib' Folder + + The 'contrib' folder contains third-party contributions. + The contributed data and documentation is provided in a subfolder per + contribution. There is no particular substructure for contributed data. + +6.1 'bgp' below 'contrib' + + 'bgp' contains the BGP data collected between June 2016 and August 2017. + For detailed documentation see [1]. + A.0 Appendix A.1 References [0] https://metrics.torproject.org/web-server-logs.html#n-re-assembling-log-fil… +[1] https://metrics.torproject.org/contrib/bgp.html

1 0

[collector/release] Add 'contrib' directory to index.json.
by karsten＠torproject.org 25 May '18

25 May '18

commit 5d2bdbb8ca615551264d21e801499f8720d2709b Author: iwakeh <iwakeh(a)torproject.org> Date: Tue Mar 20 08:31:46 2018 +0000 Add 'contrib' directory to index.json. --- src/main/java/org/torproject/collector/conf/Key.java | 1 + src/main/java/org/torproject/collector/index/CreateIndexJson.java | 1 + src/main/resources/collector.properties | 2 ++ src/test/java/org/torproject/collector/conf/ConfigurationTest.java | 2 +- 4 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/main/java/org/torproject/collector/conf/Key.java b/src/main/java/org/torproject/collector/conf/Key.java index 6454009..5e91e3e 100644 --- a/src/main/java/org/torproject/collector/conf/Key.java +++ b/src/main/java/org/torproject/collector/conf/Key.java @@ -19,6 +19,7 @@ public enum Key { ExitlistUrl(URL.class), InstanceBaseUrl(String.class), ArchivePath(Path.class), + ContribPath(Path.class), RecentPath(Path.class), OutputPath(Path.class), IndexPath(Path.class), diff --git a/src/main/java/org/torproject/collector/index/CreateIndexJson.java b/src/main/java/org/torproject/collector/index/CreateIndexJson.java index fb693a7..49bfb11 100644 --- a/src/main/java/org/torproject/collector/index/CreateIndexJson.java +++ b/src/main/java/org/torproject/collector/index/CreateIndexJson.java @@ -92,6 +92,7 @@ public class CreateIndexJson extends CollecTorMain { basePath = config.getProperty(Key.InstanceBaseUrl.name()); indexedDirectories = new File[] { config.getPath(Key.ArchivePath).toFile(), + config.getPath(Key.ContribPath).toFile(), config.getPath(Key.RecentPath).toFile() }; writeIndex(indexDirectories()); } catch (Exception e) { diff --git a/src/main/resources/collector.properties b/src/main/resources/collector.properties index 2aa556d..0fe6924 100644 --- a/src/main/resources/collector.properties +++ b/src/main/resources/collector.properties @@ -62,6 +62,8 @@ InstanceBaseUrl = https://collector.torproject.org IndexPath = index # The top-level directory for archived descriptors. ArchivePath = archive +# The top-level directory for third party data. +ContribPath = contrib # The top-level directory for the recent descriptors that were # published in the last 72 hours. RecentPath = recent diff --git a/src/test/java/org/torproject/collector/conf/ConfigurationTest.java b/src/test/java/org/torproject/collector/conf/ConfigurationTest.java index fcaa71f..e408109 100644 --- a/src/test/java/org/torproject/collector/conf/ConfigurationTest.java +++ b/src/test/java/org/torproject/collector/conf/ConfigurationTest.java @@ -40,7 +40,7 @@ public class ConfigurationTest { public void testKeyCount() throws Exception { assertEquals("The number of properties keys in enum Key changed." + "\n This test class should be adapted.", - 52, Key.values().length); + 53, Key.values().length); } @Test()

1 0

[collector/release] Adapt to metrics-lib 2.3.0 changes.
by karsten＠torproject.org 25 May '18

25 May '18

commit 574a3ec4c60bd205d97720308d21f1c2f3e1853c Author: iwakeh <iwakeh(a)torproject.org> Date: Mon Mar 26 11:01:56 2018 +0000 Adapt to metrics-lib 2.3.0 changes. This concerns #25523. --- CHANGELOG.md | 6 ++++++ build.xml | 2 +- .../java/org/torproject/collector/webstats/SanitizeWeblogs.java | 3 ++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94f7a96..d9cf7da 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,9 @@ +# Changes in version 1.6.0 - 2018-0?-?? + + * Medium changes + - Update and adapt to metrics-lib 2.3.0. + + # Changes in version 1.5.1 - 2018-03-19 * Medium changes diff --git a/build.xml b/build.xml index 780f24e..921ad33 100644 --- a/build.xml +++ b/build.xml @@ -11,7 +11,7 @@ <property name="release.version" value="1.5.1-dev" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> - <property name="metricslibversion" value="2.2.0" /> + <property name="metricslibversion" value="2.3.0" /> <property name="jarincludes" value="collector.properties logback.xml" /> <patternset id="runtime" > diff --git a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java index fd301b0..88d1456 100644 --- a/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java +++ b/src/main/java/org/torproject/collector/webstats/SanitizeWeblogs.java @@ -31,6 +31,7 @@ import org.slf4j.LoggerFactory; import java.io.BufferedReader; import java.io.ByteArrayOutputStream; +import java.io.File; import java.io.InputStreamReader; import java.io.OutputStream; import java.nio.file.Files; @@ -150,7 +151,7 @@ public class SanitizeWeblogs extends CollecTorMain { WebServerAccessLogPersistence walp = new WebServerAccessLogPersistence( new WebServerAccessLogImpl(toCompressedBytes(retainedLines), - name, false)); + new File(name), name, false)); log.debug("Storing {}.", name); walp.storeOut(this.outputPathName); walp.storeRecent(this.recentPathName);

1 0

[collector/release] Replace Gson with Jackson.
by karsten＠torproject.org 25 May '18

25 May '18

commit 818488836545aaa08bb8ef90e4a2b1d5accf535b Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue May 22 15:31:41 2018 +0200 Replace Gson with Jackson. Implements #26162. --- CHANGELOG.md | 3 ++- build.xml | 6 ++++-- .../collector/index/CreateIndexJson.java | 7 +------ .../collector/relaydescs/ReferenceChecker.java | 22 +++++++++++----------- .../collector/relaydescs/ReferenceCheckerTest.java | 21 +++++++++++++++++++++ 5 files changed, 39 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d9cf7da..9c6cb85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,8 @@ # Changes in version 1.6.0 - 2018-0?-?? * Medium changes - - Update and adapt to metrics-lib 2.3.0. + - Update and adapt to metrics-lib 2.4.0. + - Replace Gson with Jackson. # Changes in version 1.5.1 - 2018-03-19 diff --git a/build.xml b/build.xml index 921ad33..39a46e9 100644 --- a/build.xml +++ b/build.xml @@ -11,13 +11,15 @@ <property name="release.version" value="1.5.1-dev" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> - <property name="metricslibversion" value="2.3.0" /> + <property name="metricslibversion" value="2.4.0" /> <property name="jarincludes" value="collector.properties logback.xml" /> <patternset id="runtime" > <include name="commons-codec-1.10.jar"/> <include name="commons-compress-1.13.jar"/> - <include name="gson-2.4.jar"/> + <include name="jackson-annotations-2.8.6.jar"/> + <include name="jackson-core-2.8.6.jar"/> + <include name="jackson-databind-2.8.6.jar"/> <include name="xz-1.6.jar"/> <include name="metrics-lib-${metricslibversion}.jar"/> <include name="logback-core-1.1.9.jar" /> diff --git a/src/main/java/org/torproject/collector/index/CreateIndexJson.java b/src/main/java/org/torproject/collector/index/CreateIndexJson.java index 49bfb11..5f030eb 100644 --- a/src/main/java/org/torproject/collector/index/CreateIndexJson.java +++ b/src/main/java/org/torproject/collector/index/CreateIndexJson.java @@ -12,9 +12,6 @@ import org.torproject.descriptor.index.FileNode; import org.torproject.descriptor.index.IndexNode; import org.torproject.descriptor.internal.FileType; -import com.google.gson.Gson; -import com.google.gson.GsonBuilder; - import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -164,9 +161,7 @@ public class CreateIndexJson extends CollecTorMain { private void writeIndex(IndexNode indexNode) throws Exception { indexJsonFile.getParentFile().mkdirs(); - Gson gson = new GsonBuilder().excludeFieldsWithoutExposeAnnotation() - .create(); - String indexNodeString = gson.toJson(indexNode); + String indexNodeString = IndexNode.makeJsonString(indexNode); for (String filename : new String[] {indexJsonFile.toString(), indexJsonFile + ".gz", indexJsonFile + ".xz", indexJsonFile + ".bz2"}) { FileType type = FileType.valueOf( diff --git a/src/main/java/org/torproject/collector/relaydescs/ReferenceChecker.java b/src/main/java/org/torproject/collector/relaydescs/ReferenceChecker.java index 43706dd..df0508a 100644 --- a/src/main/java/org/torproject/collector/relaydescs/ReferenceChecker.java +++ b/src/main/java/org/torproject/collector/relaydescs/ReferenceChecker.java @@ -14,14 +14,14 @@ import org.torproject.descriptor.RelayNetworkStatusConsensus; import org.torproject.descriptor.RelayNetworkStatusVote; import org.torproject.descriptor.ServerDescriptor; -import com.google.gson.Gson; +import com.fasterxml.jackson.annotation.JsonAutoDetect; +import com.fasterxml.jackson.annotation.PropertyAccessor; +import com.fasterxml.jackson.databind.ObjectMapper; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.File; -import java.io.FileReader; -import java.io.FileWriter; import java.io.IOException; import java.nio.file.Files; import java.text.DateFormat; @@ -50,6 +50,10 @@ public class ReferenceChecker { private SortedSet<Reference> references = new TreeSet<>(); + private static ObjectMapper objectMapper = new ObjectMapper() + .setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.NONE) + .setVisibility(PropertyAccessor.FIELD, JsonAutoDetect.Visibility.ANY); + private static DateFormat dateTimeFormat; static { @@ -141,10 +145,9 @@ public class ReferenceChecker { if (!this.referencesFile.exists()) { return; } - Gson gson = new Gson(); - try (FileReader fr = new FileReader(this.referencesFile)) { - this.references.addAll(Arrays.asList(gson.fromJson(fr, - Reference[].class))); + try { + this.references.addAll(Arrays.asList(objectMapper.readValue( + this.referencesFile, Reference[].class))); } catch (IOException e) { logger.warn("Cannot read existing references file " + "from previous run.", e); @@ -321,11 +324,8 @@ public class ReferenceChecker { } private void writeReferencesFile() { - Gson gson = new Gson(); try { - FileWriter fw = new FileWriter(this.referencesFile); - gson.toJson(this.references, fw); - fw.close(); + objectMapper.writeValue(this.referencesFile, this.references); } catch (IOException e) { logger.warn("Cannot write references file for next " + "run.", e); diff --git a/src/test/java/org/torproject/collector/relaydescs/ReferenceCheckerTest.java b/src/test/java/org/torproject/collector/relaydescs/ReferenceCheckerTest.java index adb0b48..da2d9ad 100644 --- a/src/test/java/org/torproject/collector/relaydescs/ReferenceCheckerTest.java +++ b/src/test/java/org/torproject/collector/relaydescs/ReferenceCheckerTest.java @@ -74,5 +74,26 @@ public class ReferenceCheckerTest { .get(0)); } + @Test() + public void testEmptyReferencedString() throws Exception { + String validEmptyReferencedString + = validReferenceJson.substring(0, + validReferenceJson.indexOf("S-D8736")) + + validReferenceJson.substring( + validReferenceJson.indexOf("\",\"weight")); + File descDir = tmpf.newFolder(); + File refsFile = tmpf.newFile(); + File histFile = tmpf.newFile(); + Files.write(refsFile.toPath(), validEmptyReferencedString.getBytes()); + assertEquals(validEmptyReferencedString, + Files.readAllLines(refsFile.toPath(), + Charset.forName("US-ASCII")).get(0)); + ReferenceChecker rc = new ReferenceChecker(descDir, refsFile, histFile); + rc.check(); + assertTrue(refsFile.exists()); + assertEquals(validEmptyReferencedString, + Files.readAllLines(refsFile.toPath(), + Charset.forName("US-ASCII")).get(0)); + } }

1 0

[collector/release] Prepare for 1.6.0 release.
by karsten＠torproject.org 25 May '18

25 May '18

commit 78d01e5b01eaabfd1eb9ae604c4ad73e6c2cbe0d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed May 23 22:14:27 2018 +0200 Prepare for 1.6.0 release. --- CERT | 18 +++++++++--------- CHANGELOG.md | 2 +- build.xml | 2 +- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/CERT b/CERT index 43be56e..61fa7c6 100644 --- a/CERT +++ b/CERT @@ -1,8 +1,8 @@ -----BEGIN CERTIFICATE----- -MIIDaTCCAlGgAwIBAgIEIk6NnzANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV +MIIDaTCCAlGgAwIBAgIENNGkczANBgkqhkiG9w0BAQsFADBlMQswCQYDVQQGEwJV UzELMAkGA1UECBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBU b3IgUHJvamVjdCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwHhcNMTgw -MjI2MTQwMzUzWhcNMTgwNTI3MTQwMzUzWjBlMQswCQYDVQQGEwJVUzELMAkGA1UE +NTIzMTUxNDU2WhcNMTgwODIxMTUxNDU2WjBlMQswCQYDVQQGEwJVUzELMAkGA1UE CBMCV0ExEDAOBgNVBAcTB1NlYXR0bGUxHTAbBgNVBAoTFFRoZSBUb3IgUHJvamVj dCwgSW5jMRgwFgYDVQQDEw9LYXJzdGVuIExvZXNpbmcwggEiMA0GCSqGSIb3DQEB AQUAA4IBDwAwggEKAoIBAQChXn+IUp+o6G+k4ffxk3TkxZb3iXfiG7byNsG63olU @@ -11,11 +11,11 @@ Qw+VAhKTcEIv4yiR0BWapQyR07pgmKirYVjN6s6ef8NJzUptpxLlaYJ3ZfQfc4aE MXzScgaccwDFIWQ661lzLGCfeSxxa3Xy4wWsGwzNzLITYrrABcbg7yogLo2btNvD oEwGL3/baQdhl0dra6biVCZr9ydn3Hg57S55pUU0rBY25id78zUO8xrfNHw54wwX lOblGt75OOkahP/ZZSBxxoiknJ6y5VQV8y+noA4vigXFAgMBAAGjITAfMB0GA1Ud -DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAlUkU -qqf+4yfXwAWFr2q6iijr54NDDEQwybCblIzVnsuGHPUDuie3ZWSHirtblBs/uJ9x -RxmwkBrJr9IGMmGhN2GKXIPeUH0EZBYo7bsgo5d+E61OCnd/O+1JZzdG9dK+0kfq -MLfo6ltFZZouHIIXfvOm8sLLRrdkXPrLQ/E8fTHB7dL6T8Hqg6pHRrRZDtuSM9CO -zSYropxqlFzzlzciOdTU05D8Cnx2j/RtaycxHxFS7QtriDB0uOfqvyiVeqpr72wG -qetlu3h46fXj3ALGVSXy+YZpYxcRNZsQyiBXdlXbgY0OfOVPFOH3HiZuv3zhfRJW -2DiJiA8BLxZToe2XDA== +DgQWBBSeh60M+/wMYyYhlxtuff2Hk9n7bzANBgkqhkiG9w0BAQsFAAOCAQEAJtEQ +B2AVpVtjTGU7uujUtEjSn1ICv7QLW/37TmHn+m1SWvXOVlSG81eb5JwX449Pn3w5 +K1Bx0JJmJ6Kec2kHtoR1r5/DUlTMtRqKQ5ZHQbvYMx+Ifq8TRALHjcw8p5Vw+gep +0zYbSQycklzFaFqLB8Cus0ICb+UY54HTddpezQ3IXS/vc4Vy07ocm8rUCz1s8L/r +ehTOxSym7G7+kcRzNplFOLL5iO8o6uHPyLR1TIAIXa0XZ41ogNl2q+6CvXG+UI/j +qkzSXD6FJK1Px2nxNFIe/w9NL+chSytIGnV3CImyiORuV+1OxMglyQspSGEbl1XP +OfiTEYHnp12BYMeRyw== -----END CERTIFICATE----- diff --git a/CHANGELOG.md b/CHANGELOG.md index 9c6cb85..3a88792 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -# Changes in version 1.6.0 - 2018-0?-?? +# Changes in version 1.6.0 - 2018-05-23 * Medium changes - Update and adapt to metrics-lib 2.4.0. diff --git a/build.xml b/build.xml index 39a46e9..7240db6 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.5.1-dev" /> + <property name="release.version" value="1.6.0" /> <property name="project-main-class" value="org.torproject.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.4.0" />

1 0

[collector/master] Prepare for 1.6.0 release.
by karsten＠torproject.org 25 May '18

25 May '18

1 0

[collector/master] Replace Gson with Jackson.
by karsten＠torproject.org 25 May '18

25 May '18

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits May 2018