September 2019 - tor-commits - lists.torproject.org

[collector/release] Update to Debian buster libraries.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit e65269dfe70312e50472de807dc0714b0aed5ec1 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 24 10:43:52 2019 +0200 Update to Debian buster libraries. Also upgrade to latest metrics-base. Checkstyle is excluded from this update. --- build.xml | 18 +++++++++--------- src/build | 2 +- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/build.xml b/build.xml index d0fd248..d35fd58 100644 --- a/build.xml +++ b/build.xml @@ -15,16 +15,16 @@ <property name="jarincludes" value="collector.properties logback.xml" /> <patternset id="runtime" > - <include name="commons-codec-1.10.jar"/> - <include name="commons-compress-1.13.jar"/> - <include name="jackson-annotations-2.8.6.jar"/> - <include name="jackson-core-2.8.6.jar"/> - <include name="jackson-databind-2.8.6.jar"/> - <include name="xz-1.6.jar"/> + <include name="commons-codec-1.11.jar"/> + <include name="commons-compress-1.18.jar"/> + <include name="jackson-annotations-2.9.8.jar"/> + <include name="jackson-core-2.9.8.jar"/> + <include name="jackson-databind-2.9.8.jar"/> + <include name="xz-1.8.jar"/> <include name="metrics-lib-${metricslibversion}-thin.jar"/> - <include name="logback-core-1.1.9.jar" /> - <include name="logback-classic-1.1.9.jar" /> - <include name="slf4j-api-1.7.22.jar" /> + <include name="logback-core-1.2.3.jar" /> + <include name="logback-classic-1.2.3.jar" /> + <include name="slf4j-api-1.7.25.jar" /> </patternset> <target name="coverage-check"> diff --git a/src/build b/src/build index e639c69..e12ba96 160000 --- a/src/build +++ b/src/build @@ -1 +1 @@ -Subproject commit e639c697e9e94c6dbb26e946e5247c20a62c0661 +Subproject commit e12ba96aa0708b20375cf7cf5d20dfa19940d18a

1 0

[collector/release] Bump version to 1.9.1-dev.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit d992d8719d507fa8fffee24faa18546da848e16e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu May 30 15:32:07 2019 +0200 Bump version to 1.9.1-dev. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index d2d1f73..d0fd248 100644 --- a/build.xml +++ b/build.xml @@ -8,7 +8,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.9.1" /> + <property name="release.version" value="1.9.1-dev" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.6.2" />

1 0

[collector/release] Remove Cobertura from the build process.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit 801008431115951fe3b0f967c7d4ca0469610ae8 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Aug 14 11:38:40 2019 +0200 Remove Cobertura from the build process. --- CHANGELOG.md | 1 + src/build | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 66e53ed..4fd0401 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,7 @@ copied to the `lib/` directory manually. Current dependency versions resolved by Ivy are the same as in Debian stretch with few exceptions. + - Remove Cobertura from the build process. # Changes in version 1.9.1 - 2019-05-29 diff --git a/src/build b/src/build index 256e619..75cda35 160000 --- a/src/build +++ b/src/build @@ -1 +1 @@ -Subproject commit 256e6192ad0066e48c6abb0ee9ac71714c7a3a0f +Subproject commit 75cda35ba7375ab0f5b12fa214620c3a29917097

1 0

[collector/release] Adds GitLab CI for tests and checks
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit a8a4961a1b49a3104dd37ac93cc41e3f908e25c5 Author: Iain R. Learmonth <irl(a)fsfe.org> Date: Thu Sep 5 12:21:43 2019 +0100 Adds GitLab CI for tests and checks Configuration is held in the .gitlab-ci.yml file. Fixes: #31399 --- .gitlab-ci.yml | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..b67b5a4 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,25 @@ +variables: + GIT_STRATEGY: clone + METRICS_LIB_VERSION: "2.6.2" + JAVA_TOOL_OPTIONS: "-Dfile.encoding=UTF-8" + +stages: + - test + +test: + stage: test + image: debian:buster + script: + - apt update + - apt -y install default-jdk ant ivy git curl + - git submodule init + - git submodule update + - mkdir lib + - mkdir tmp + - pushd tmp + - curl https://dist.torproject.org/metrics-lib/$METRICS_LIB_VERSION/metrics-lib-$M… | tar xzf - + - popd + - mv tmp/metrics-lib-$METRICS_LIB_VERSION/generated/dist/metrics-lib-$METRICS_LIB_VERSION-thin.jar lib + - ant -lib /usr/share/java resolve + - ant test + - ant checks

1 0

[collector/release] Use Ivy for resolving external dependencies.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit 027a54bfe2c970a72db2afc485d2fa12f77cb247 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Aug 6 17:04:41 2019 +0200 Use Ivy for resolving external dependencies. Includes an upgrade to the latest metrics-base commit 256e619. Implements part of #31326. --- CHANGELOG.md | 9 +++++++++ build.xml | 3 ++- src/build | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 613ef71..66e53ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,15 @@ * Medium changes - Changed local import of bandwidth files to include the parent directory name as @source annotation and to the filename. + - Use Ivy for resolving external dependencies rather than relying + on files found in Debian stable packages. Requires installing Ivy + (using `apt-get install ivy`, `brew install ivy`, or similar) and + running `ant resolve` (or `ant -lib /usr/share/java resolve`). + Retrieved files are then copied to the `lib/` directory, except + for dependencies on other metrics libraries that still need to be + copied to the `lib/` directory manually. Current dependency + versions resolved by Ivy are the same as in Debian stretch with + few exceptions. # Changes in version 1.9.1 - 2019-05-29 diff --git a/build.xml b/build.xml index d0fd248..39180c7 100644 --- a/build.xml +++ b/build.xml @@ -4,7 +4,8 @@ <!ENTITY base SYSTEM "src/build/java/base.xml"> ]> -<project default="usage" name="collector" basedir="."> +<project default="usage" name="collector" basedir="." + xmlns:ivy="antlib:org.apache.ivy.ant"> <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> diff --git a/src/build b/src/build index ed48192..256e619 160000 --- a/src/build +++ b/src/build @@ -1 +1 @@ -Subproject commit ed4819215aa60cc8935f955d9404f30e83dd892b +Subproject commit 256e6192ad0066e48c6abb0ee9ac71714c7a3a0f

1 0

[collector/release] Add source name to locally imported bandwidth files.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit abde3f7591117060ccf1afc9b26531d79619301f Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jul 10 10:34:09 2019 +0200 Add source name to locally imported bandwidth files. Implements #30219. --- CHANGELOG.md | 7 +++++++ .../metrics/collector/relaydescs/ArchiveReader.java | 2 +- .../metrics/collector/relaydescs/ArchiveWriter.java | 12 +++++++++--- .../collector/relaydescs/CachedRelayDescriptorReader.java | 6 +++--- .../collector/relaydescs/RelayDescriptorDownloader.java | 4 ++-- .../metrics/collector/relaydescs/RelayDescriptorParser.java | 7 +++++-- src/main/resources/collector.properties | 5 ++++- 7 files changed, 31 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4682fef..613ef71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +# Changes in version 1.10.0 - 2019-??-?? + + * Medium changes + - Changed local import of bandwidth files to include the parent + directory name as @source annotation and to the filename. + + # Changes in version 1.9.1 - 2019-05-29 * Medium changes diff --git a/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveReader.java b/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveReader.java index 27010a9..fee46c9 100644 --- a/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveReader.java +++ b/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveReader.java @@ -141,7 +141,7 @@ public class ArchiveReader { } bis.close(); byte[] allData = baos.toByteArray(); - boolean stored = this.rdp.parse(allData); + boolean stored = this.rdp.parse(allData, pop); if (!stored) { filesToRetry.add(pop); continue; diff --git a/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveWriter.java b/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveWriter.java index e1279ee..911ac2a 100644 --- a/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveWriter.java +++ b/src/main/java/org/torproject/metrics/collector/relaydescs/ArchiveWriter.java @@ -745,18 +745,24 @@ public class ArchiveWriter extends CollecTorMain { /** Stores a bandwidth file to disk. */ void storeBandwidthFile(byte[] data, LocalDateTime fileCreatedOrTimestamp, - String bandwidthFileDigest) { + String sourceName, String bandwidthFileDigest) { DateTimeFormatter printFormat = DateTimeFormatter .ofPattern("uuuu/MM/dd/uuuu-MM-dd-HH-mm-ss").withZone(ZoneOffset.UTC); File tarballFile = Paths.get(this.outputDirectory, "bandwidth", fileCreatedOrTimestamp.format(printFormat) + "-bandwidth-" + + (null == sourceName ? "" : (sourceName + "-")) + bandwidthFileDigest).toFile(); + StringBuilder sb = new StringBuilder(); + sb.append(Annotation.BandwidthFile.toString()); + if (null != sourceName) { + sb.append("@source ").append(sourceName).append('\n'); + } boolean tarballFileExistedBefore = tarballFile.exists(); File rsyncFile = Paths.get(recentPathName, RELAY_DESCRIPTORS, "bandwidths", tarballFile.getName()).toFile(); File[] outputFiles = new File[] { tarballFile, rsyncFile }; - if (this.store(Annotation.BandwidthFile.bytes(), data, outputFiles, null)) { - this.storedVotesCounter++; + if (this.store(sb.toString().getBytes(), data, outputFiles, null)) { + this.storedBandwidthsCounter++; } if (!tarballFileExistedBefore && this.nowLocalDateTime.isAfter(fileCreatedOrTimestamp.plusDays(3L))) { diff --git a/src/main/java/org/torproject/metrics/collector/relaydescs/CachedRelayDescriptorReader.java b/src/main/java/org/torproject/metrics/collector/relaydescs/CachedRelayDescriptorReader.java index 3e1bba4..0308f05 100644 --- a/src/main/java/org/torproject/metrics/collector/relaydescs/CachedRelayDescriptorReader.java +++ b/src/main/java/org/torproject/metrics/collector/relaydescs/CachedRelayDescriptorReader.java @@ -158,7 +158,7 @@ public class CachedRelayDescriptorReader { allData)); if (!this.lastImportHistory.contains(digest) && !this.currentImportHistory.contains(digest)) { - this.rdp.parse(allData); + this.rdp.parse(allData, null); } else { this.dumpStats.append(" (skipped)"); } @@ -183,7 +183,7 @@ public class CachedRelayDescriptorReader { rawNetworkStatusBytes)); if (!this.lastImportHistory.contains(digest) && !this.currentImportHistory.contains(digest)) { - this.rdp.parse(rawNetworkStatusBytes); + this.rdp.parse(rawNetworkStatusBytes, null); parsedNum++; } else { skippedNum++; @@ -229,7 +229,7 @@ public class CachedRelayDescriptorReader { descBytes)); if (!this.lastImportHistory.contains(digest) && !this.currentImportHistory.contains(digest)) { - this.rdp.parse(descBytes); + this.rdp.parse(descBytes, null); parsedNum++; } else { skippedNum++; diff --git a/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorDownloader.java b/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorDownloader.java index 5a241f4..f1179d8 100644 --- a/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorDownloader.java +++ b/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorDownloader.java @@ -899,7 +899,7 @@ public class RelayDescriptorDownloader { int receivedDescriptors = 0; if (allData != null) { if (resource.startsWith("/tor/status-vote/")) { - this.rdp.parse(allData); + this.rdp.parse(allData, null); receivedDescriptors = 1; } else if (resource.startsWith("/tor/server/") || resource.startsWith("/tor/extra/")) { @@ -933,7 +933,7 @@ public class RelayDescriptorDownloader { end += endToken.length(); byte[] descBytes = new byte[end - start]; System.arraycopy(allData, start, descBytes, 0, end - start); - this.rdp.parse(descBytes); + this.rdp.parse(descBytes, null); receivedDescriptors++; } } else if (resource.startsWith("/tor/micro/")) { diff --git a/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorParser.java b/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorParser.java index 113ac77..a610ed9 100644 --- a/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorParser.java +++ b/src/main/java/org/torproject/metrics/collector/relaydescs/RelayDescriptorParser.java @@ -10,6 +10,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.BufferedReader; +import java.io.File; import java.io.IOException; import java.io.StringReader; import java.text.ParseException; @@ -75,7 +76,7 @@ public class RelayDescriptorParser { * forwards them to the archive writer to store them to disk, and tells * the relay descriptor downloader and archive reader about the * contained descriptor and all referenced descriptors. */ - public boolean parse(byte[] data) { + public boolean parse(byte[] data, File containingFile) { boolean stored = false; try { /* Convert descriptor to ASCII for parsing. This means we'll lose @@ -357,7 +358,9 @@ public class RelayDescriptorParser { fileCreatedOrTimestamp = LocalDateTime.of(2000, 1, 1, 0, 0, 0); } } - this.aw.storeBandwidthFile(data, fileCreatedOrTimestamp, + String sourceName = containingFile == null ? null + : containingFile.getParentFile().getName(); + this.aw.storeBandwidthFile(data, fileCreatedOrTimestamp, sourceName, DigestUtils.sha256Hex(data).toUpperCase()); stored = true; } diff --git a/src/main/resources/collector.properties b/src/main/resources/collector.properties index 787f188..292e876 100644 --- a/src/main/resources/collector.properties +++ b/src/main/resources/collector.properties @@ -86,7 +86,10 @@ RelaySyncOrigins = https://collector.torproject.org ## Path to Tor data directory to read cached-* files from ## the listed path(s). If there is more that one separated by comma. RelayCacheOrigins = in/relay-descriptors/cacheddesc/ -## Relative path to directory to import directory archives from +## Relative path to directory to import directory archives from. +## Note that when importing bandwidth files, the parent directory name is +## included in the @source annotation and in the file name. Recommended +## source names are nicknames of directory authorities using these files. RelayLocalOrigins = in/relay-descriptors/archives/ # ## Keep a history of imported directory archive files to know which files

1 0

[collector/release] Archive snowflake statistics.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit 569bb83ff5e50843079ebd1150801a8bd8efa747 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Aug 14 08:51:29 2019 +0200 Archive snowflake statistics. Implements #29461. --- CHANGELOG.md | 2 + build.xml | 2 +- .../org/torproject/metrics/collector/Main.java | 3 + .../metrics/collector/conf/Annotation.java | 3 +- .../metrics/collector/conf/Configuration.java | 1 + .../org/torproject/metrics/collector/conf/Key.java | 8 +- .../persist/SnowflakeStatsPersistence.java | 37 ++++ .../snowflake/SnowflakeStatsDownloader.java | 191 +++++++++++++++++++++ .../metrics/collector/sync/SyncPersistence.java | 5 + src/main/resources/collector.properties | 20 +++ src/main/resources/create-tarballs.sh | 7 + src/main/resources/docs/PROTOCOL | 33 +++- .../metrics/collector/conf/ConfigurationTest.java | 2 +- .../metrics/collector/cron/CollecTorMainTest.java | 1 + .../metrics/collector/cron/SchedulerTest.java | 9 +- 15 files changed, 317 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fd0401..c537d22 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,8 @@ versions resolved by Ivy are the same as in Debian stretch with few exceptions. - Remove Cobertura from the build process. + - Archive snowflake statistics. + - Update to metrics-lib 2.7.0. # Changes in version 1.9.1 - 2019-05-29 diff --git a/build.xml b/build.xml index 39180c7..d1d5b70 100644 --- a/build.xml +++ b/build.xml @@ -12,7 +12,7 @@ <property name="release.version" value="1.9.1-dev" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> - <property name="metricslibversion" value="2.6.2" /> + <property name="metricslibversion" value="2.7.0" /> <property name="jarincludes" value="collector.properties logback.xml" /> <patternset id="runtime" > diff --git a/src/main/java/org/torproject/metrics/collector/Main.java b/src/main/java/org/torproject/metrics/collector/Main.java index 46e93af..6907e93 100644 --- a/src/main/java/org/torproject/metrics/collector/Main.java +++ b/src/main/java/org/torproject/metrics/collector/Main.java @@ -14,6 +14,7 @@ import org.torproject.metrics.collector.exitlists.ExitListDownloader; import org.torproject.metrics.collector.indexer.CreateIndexJson; import org.torproject.metrics.collector.onionperf.OnionPerfDownloader; import org.torproject.metrics.collector.relaydescs.ArchiveWriter; +import org.torproject.metrics.collector.snowflake.SnowflakeStatsDownloader; import org.torproject.metrics.collector.webstats.SanitizeWeblogs; import org.slf4j.Logger; @@ -53,6 +54,8 @@ public class Main { collecTorMains.put(Key.RelaydescsActivated, ArchiveWriter.class); collecTorMains.put(Key.OnionPerfActivated, OnionPerfDownloader.class); collecTorMains.put(Key.WebstatsActivated, SanitizeWeblogs.class); + collecTorMains.put(Key.SnowflakeStatsActivated, + SnowflakeStatsDownloader.class); } private static Configuration conf = new Configuration(); diff --git a/src/main/java/org/torproject/metrics/collector/conf/Annotation.java b/src/main/java/org/torproject/metrics/collector/conf/Annotation.java index 2e47df0..8cd3324 100644 --- a/src/main/java/org/torproject/metrics/collector/conf/Annotation.java +++ b/src/main/java/org/torproject/metrics/collector/conf/Annotation.java @@ -18,7 +18,8 @@ public enum Annotation { Server("@type server-descriptor 1.0\n"), Status("@type bridge-network-status 1.2\n"), OnionPerf("@type torperf 1.1\n"), - Vote("@type network-status-vote-3 1.0\n"); + Vote("@type network-status-vote-3 1.0\n"), + SnowflakeStats("@type snowflake-stats 1.0\n"); private final String annotation; private final byte[] bytes; diff --git a/src/main/java/org/torproject/metrics/collector/conf/Configuration.java b/src/main/java/org/torproject/metrics/collector/conf/Configuration.java index 69d3bcd..27f5125 100644 --- a/src/main/java/org/torproject/metrics/collector/conf/Configuration.java +++ b/src/main/java/org/torproject/metrics/collector/conf/Configuration.java @@ -91,6 +91,7 @@ public class Configuration extends Observable implements Cloneable { || this.getBool(Key.ExitlistsActivated) || this.getBool(Key.UpdateindexActivated) || this.getBool(Key.OnionPerfActivated) + || this.getBool(Key.SnowflakeStatsActivated) || this.getBool(Key.WebstatsActivated))) { throw new ConfigurationException("Nothing is activated!\n" + "Please edit collector.properties. Exiting."); diff --git a/src/main/java/org/torproject/metrics/collector/conf/Key.java b/src/main/java/org/torproject/metrics/collector/conf/Key.java index ba4bcd9..e683fe2 100644 --- a/src/main/java/org/torproject/metrics/collector/conf/Key.java +++ b/src/main/java/org/torproject/metrics/collector/conf/Key.java @@ -66,7 +66,13 @@ public enum Key { WebstatsActivated(Boolean.class), WebstatsLimits(Boolean.class), WebstatsOffsetMinutes(Integer.class), - WebstatsPeriodMinutes(Integer.class); + WebstatsPeriodMinutes(Integer.class), + SnowflakeStatsActivated(Boolean.class), + SnowflakeStatsOffsetMinutes(Integer.class), + SnowflakeStatsPeriodMinutes(Integer.class), + SnowflakeStatsUrl(URL.class), + SnowflakeStatsSources(SourceType[].class), + SnowflakeStatsSyncOrigins(URL[].class); private Class clazz; private static Set<String> keys; diff --git a/src/main/java/org/torproject/metrics/collector/persist/SnowflakeStatsPersistence.java b/src/main/java/org/torproject/metrics/collector/persist/SnowflakeStatsPersistence.java new file mode 100644 index 0000000..ee6e029 --- /dev/null +++ b/src/main/java/org/torproject/metrics/collector/persist/SnowflakeStatsPersistence.java @@ -0,0 +1,37 @@ +/* Copyright 2019 The Tor Project + * See LICENSE for licensing information */ + +package org.torproject.metrics.collector.persist; + +import org.torproject.descriptor.SnowflakeStats; +import org.torproject.metrics.collector.conf.Annotation; + +import java.nio.file.Paths; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; + +public class SnowflakeStatsPersistence + extends DescriptorPersistence<SnowflakeStats> { + + private static final String SNOWFLAKES = "snowflakes"; + + public SnowflakeStatsPersistence(SnowflakeStats desc) { + super(desc, Annotation.SnowflakeStats.bytes()); + calculatePaths(); + } + + private void calculatePaths() { + DateTimeFormatter directoriesFormatter = DateTimeFormatter + .ofPattern("uuuu/MM/dd").withZone(ZoneOffset.UTC); + String[] directories = this.desc.snowflakeStatsEnd() + .format(directoriesFormatter).split("/"); + DateTimeFormatter fileFormatter = DateTimeFormatter + .ofPattern("uuuu-MM-dd-HH-mm-ss").withZone(ZoneOffset.UTC); + String fileOut = this.desc.snowflakeStatsEnd().format(fileFormatter) + + "-snowflake-stats"; + this.recentPath = Paths.get(SNOWFLAKES, fileOut).toString(); + this.storagePath = Paths.get(SNOWFLAKES, directories[0], directories[1], + directories[2], fileOut).toString(); + } +} + diff --git a/src/main/java/org/torproject/metrics/collector/snowflake/SnowflakeStatsDownloader.java b/src/main/java/org/torproject/metrics/collector/snowflake/SnowflakeStatsDownloader.java new file mode 100644 index 0000000..4f7994e --- /dev/null +++ b/src/main/java/org/torproject/metrics/collector/snowflake/SnowflakeStatsDownloader.java @@ -0,0 +1,191 @@ +/* Copyright 2019 The Tor Project + * See LICENSE for licensing information */ + +package org.torproject.metrics.collector.snowflake; + +import org.torproject.descriptor.Descriptor; +import org.torproject.descriptor.DescriptorParser; +import org.torproject.descriptor.DescriptorSourceFactory; +import org.torproject.descriptor.SnowflakeStats; +import org.torproject.metrics.collector.conf.Annotation; +import org.torproject.metrics.collector.conf.Configuration; +import org.torproject.metrics.collector.conf.ConfigurationException; +import org.torproject.metrics.collector.conf.Key; +import org.torproject.metrics.collector.cron.CollecTorMain; +import org.torproject.metrics.collector.persist.SnowflakeStatsPersistence; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.BufferedInputStream; +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStream; +import java.net.HttpURLConnection; +import java.net.URL; +import java.time.LocalDateTime; +import java.util.Arrays; +import java.util.SortedSet; +import java.util.Stack; +import java.util.TreeSet; + +public class SnowflakeStatsDownloader extends CollecTorMain { + + private static final Logger logger = LoggerFactory.getLogger( + SnowflakeStatsDownloader.class); + + private String recentPathName; + + /** Instantiate the snowflake-stats module using the given configuration. */ + public SnowflakeStatsDownloader(Configuration config) { + super(config); + this.mapPathDescriptors.put("recent/snowflakes", SnowflakeStats.class); + } + + @Override + public String module() { + return "SnowflakeStats"; + } + + @Override + protected String syncMarker() { + return "SnowflakeStats"; + } + + @Override + protected void startProcessing() throws ConfigurationException { + + this.recentPathName = config.getPath(Key.RecentPath).toString(); + logger.debug("Downloading snowflake stats..."); + URL url = config.getUrl(Key.SnowflakeStatsUrl); + ByteArrayOutputStream downloadedSnowflakeStats + = this.downloadFromHttpServer(url); + if (null == downloadedSnowflakeStats) { + return; + } + logger.debug("Finished downloading {}.", url); + + DescriptorParser descriptorParser = + DescriptorSourceFactory.createDescriptorParser(); + SortedSet<LocalDateTime> snowflakeStatsEnds = new TreeSet<>(); + String outputPathName = config.getPath(Key.OutputPath).toString(); + for (Descriptor descriptor : descriptorParser.parseDescriptors( + downloadedSnowflakeStats.toByteArray(), null, null)) { + if (descriptor instanceof SnowflakeStats) { + SnowflakeStats snowflakeStats = (SnowflakeStats) descriptor; + LocalDateTime snowflakeStatsEnd = snowflakeStats.snowflakeStatsEnd(); + snowflakeStatsEnds.add(snowflakeStatsEnd); + SnowflakeStatsPersistence persistence + = new SnowflakeStatsPersistence(snowflakeStats); + File tarballFile = new File(outputPathName + "/" + + persistence.getStoragePath()); + if (tarballFile.exists()) { + continue; + } + File rsyncFile = new File(this.recentPathName + "/" + + persistence.getRecentPath()); + File[] outputFiles = new File[] { tarballFile, rsyncFile }; + for (File outputFile : outputFiles) { + this.writeToFile(outputFile, Annotation.SnowflakeStats.bytes(), + snowflakeStats.getRawDescriptorBytes()); + } + } + } + if (snowflakeStatsEnds.isEmpty()) { + logger.warn("Could not parse downloaded snowflake stats."); + return; + } else if (snowflakeStatsEnds.last().isBefore(LocalDateTime.now() + .minusHours(48L))) { + logger.warn("The latest snowflake stats are older than 48 hours: {}.", + snowflakeStatsEnds.last()); + } + + this.cleanUpRsyncDirectory(); + } + + /** + * Download the given URL from an HTTP server and return a stream with + * downloaded bytes. + * + * <p>If anything goes wrong while downloading, log a warning and return + * {@code null}.</p> + * + * @param url URL to download. + * @return Stream with downloaded bytes, or {@code null} if an error has + * occurred. + */ + private ByteArrayOutputStream downloadFromHttpServer(URL url) { + ByteArrayOutputStream downloadedBytes = new ByteArrayOutputStream(); + try { + HttpURLConnection huc = (HttpURLConnection) url.openConnection(); + huc.setRequestMethod("GET"); + huc.setReadTimeout(5000); + huc.connect(); + int response = huc.getResponseCode(); + if (response != 200) { + logger.warn("Could not download {}. Response code {}", url, response); + return null; + } + try (BufferedInputStream in = new BufferedInputStream( + huc.getInputStream())) { + int len; + byte[] data = new byte[1024]; + while ((len = in.read(data, 0, 1024)) >= 0) { + downloadedBytes.write(data, 0, len); + } + } + } catch (IOException e) { + logger.warn("Failed downloading {}.", url, e); + return null; + } + return downloadedBytes; + } + + /** + * Write the given byte array(s) to the given file. + * + * <p>If the file already exists, it is overwritten. If the parent directory + * (or any of its parent directories) does not exist, it is created. If + * anything goes wrong, log a warning and return.</p> + * + * @param outputFile File to write to. + * @param bytes One or more byte arrays. + */ + private void writeToFile(File outputFile, byte[] ... bytes) { + try { + if (!outputFile.getParentFile().exists() + && !outputFile.getParentFile().mkdirs()) { + logger.warn("Could not create parent directories of {}.", outputFile); + return; + } + OutputStream os = new FileOutputStream(outputFile); + for (byte[] b : bytes) { + os.write(b); + } + os.close(); + } catch (IOException e) { + logger.warn("Could not write downloaded snowflake stats to {}", + outputFile.getAbsolutePath(), e); + } + } + + /** Delete all files from the rsync directory that have not been modified + * in the last three days. */ + public void cleanUpRsyncDirectory() { + long cutOffMillis = System.currentTimeMillis() + - 3L * 24L * 60L * 60L * 1000L; + Stack<File> allFiles = new Stack<>(); + allFiles.add(new File(recentPathName)); + while (!allFiles.isEmpty()) { + File file = allFiles.pop(); + if (file.isDirectory()) { + allFiles.addAll(Arrays.asList(file.listFiles())); + } else if (file.lastModified() < cutOffMillis) { + file.delete(); + } + } + } +} + diff --git a/src/main/java/org/torproject/metrics/collector/sync/SyncPersistence.java b/src/main/java/org/torproject/metrics/collector/sync/SyncPersistence.java index 0d344bf..4b3b7bc 100644 --- a/src/main/java/org/torproject/metrics/collector/sync/SyncPersistence.java +++ b/src/main/java/org/torproject/metrics/collector/sync/SyncPersistence.java @@ -13,6 +13,7 @@ import org.torproject.descriptor.RelayExtraInfoDescriptor; import org.torproject.descriptor.RelayNetworkStatusConsensus; import org.torproject.descriptor.RelayNetworkStatusVote; import org.torproject.descriptor.RelayServerDescriptor; +import org.torproject.descriptor.SnowflakeStats; import org.torproject.descriptor.TorperfResult; import org.torproject.descriptor.WebServerAccessLog; import org.torproject.metrics.collector.conf.Configuration; @@ -29,6 +30,7 @@ import org.torproject.metrics.collector.persist.MicroConsensusPersistence; import org.torproject.metrics.collector.persist.OnionPerfPersistence; import org.torproject.metrics.collector.persist.PersistenceUtils; import org.torproject.metrics.collector.persist.ServerDescriptorPersistence; +import org.torproject.metrics.collector.persist.SnowflakeStatsPersistence; import org.torproject.metrics.collector.persist.StatusPersistence; import org.torproject.metrics.collector.persist.VotePersistence; import org.torproject.metrics.collector.persist.WebServerAccessLogPersistence; @@ -143,6 +145,9 @@ public class SyncPersistence { case "BandwidthFile": descPersist = new BandwidthFilePersistence((BandwidthFile) desc); break; + case "SnowflakeStats": + descPersist = new SnowflakeStatsPersistence((SnowflakeStats) desc); + break; default: log.trace("Invalid descriptor type {} for sync-merge.", clazz.getName()); diff --git a/src/main/resources/collector.properties b/src/main/resources/collector.properties index 292e876..a4eed7a 100644 --- a/src/main/resources/collector.properties +++ b/src/main/resources/collector.properties @@ -47,6 +47,13 @@ WebstatsActivated = false WebstatsPeriodMinutes = 360 # offset in minutes since the epoch and WebstatsOffsetMinutes = 31 +# the following defines, if this module is activated +SnowflakeStatsActivated = false +# period in minutes +SnowflakeStatsPeriodMinutes = 480 +# offset in minutes since the epoch and +SnowflakeStatsOffsetMinutes = 100 + ########################################## ## All below can be changed at runtime. ##### @@ -178,3 +185,16 @@ WebstatsLocalOrigins = in/webstats # Default 'true' behaves as stated in section 4 of # https://metrics.torproject.org/web-server-logs.html WebstatsLimits = true +# +# +######## Snowflake statistics ######## +# +## Define descriptor sources +# possible values: Sync, Remote +SnowflakeStatsSources = Remote +## Retrieve files from the following instances. +## List of URLs separated by comma. +SnowflakeStatsSyncOrigins = https://collector.torproject.org +## Where to download snowflake statistics from. +SnowflakeStatsUrl = https://snowflake-broker.torproject.net/metrics +# diff --git a/src/main/resources/create-tarballs.sh b/src/main/resources/create-tarballs.sh index 7e4668a..50b7fdb 100755 --- a/src/main/resources/create-tarballs.sh +++ b/src/main/resources/create-tarballs.sh @@ -59,6 +59,8 @@ TARBALLS=( bridge-server-descriptors-$YEARTWO-$MONTHTWO bridge-extra-infos-$YEARONE-$MONTHONE bridge-extra-infos-$YEARTWO-$MONTHTWO + snowflakes-$YEARONE-$MONTHONE + snowflakes-$YEARTWO-$MONTHTWO ) TARBALLS=($(printf "%s\n" "${TARBALLS[@]}" | uniq)) @@ -86,6 +88,8 @@ DIRECTORIES=( $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/server-descriptors/ $OUTDIR/bridge-descriptors/$YEARONE/$MONTHONE/extra-infos/ $OUTDIR/bridge-descriptors/$YEARTWO/$MONTHTWO/extra-infos/ + $OUTDIR/snowflakes/$YEARONE/$MONTHONE/ + $OUTDIR/snowflakes/$YEARTWO/$MONTHTWO/ ) DIRECTORIES=($(printf "%s\n" "${DIRECTORIES[@]}" | uniq)) @@ -169,4 +173,7 @@ ln -f -s -t $ARCHIVEDIR/torperf/ $TARBALLTARGETDIR/torperf-20??-??.tar.xz mkdir -p $ARCHIVEDIR/webstats/ ln -f -s -t $ARCHIVEDIR/webstats/ $TARBALLTARGETDIR/webstats-20??-??.tar +mkdir -p $ARCHIVEDIR/snowflakes/ +ln -f -s -t $ARCHIVEDIR/snowflakes/ $TARBALLTARGETDIR/snowflakes-20??-??.tar.xz + echo `date` "Finished." diff --git a/src/main/resources/docs/PROTOCOL b/src/main/resources/docs/PROTOCOL index 58ed4dc..478f168 100644 --- a/src/main/resources/docs/PROTOCOL +++ b/src/main/resources/docs/PROTOCOL @@ -45,6 +45,7 @@ * exit-lists * relay-descriptors * torperf + * snowflakes The substructure of these folders differs depending on their content. @@ -116,6 +117,13 @@ * for bandwidths, from the file_created value if available, otherwise the timestamp. +2.5 'snowflakes' below 'archive' + + 'snowflakes' contains compressed tarballs with snowflake statistics, + named in the following way: + + 'snowflakes' DASH year DASH month DOT TAR DOT compression-type + 3.0 Index Files The index.json file and its compressed versions of various types are @@ -132,6 +140,7 @@ * exit-lists * relay-descriptors * torperf + * snowflakes 4.1 'exit-lists' and 'torperf' below 'recent' @@ -254,6 +263,16 @@ 'webstats' contains compressed log files named according to the 'Tor web server logs' specification, section 4.3 [0]. +4.5 'snowflakes' below 'recent' + + 'snowflakes' contains files named + + year DASH month DASH day DASH hour DASH minute DASH second + DASH SNOWFLAKESTATS + + Where SNOWFLAKESTATS is the string "snowflake-stats" and all time related + values are derived from the snowflake statistics interval end. + 5.0 The Tar-ball's Directory Structure and the Internal Structure The 'out' directory is the internal storage that is used for @@ -266,8 +285,9 @@ * exit-lists * relay-descriptors * torperf + * snowflakes - (There has been a fifth subdirectory 'bridge-pool-assignments' which has been + (There has been another subdirectory 'bridge-pool-assignments' which has been removed when CollecTor stopped collecting those descriptors. However, it's structure can still be found in the tarballs.) @@ -416,6 +436,17 @@ 'webstats' contains compressed log files structured and named according to the 'Tor web server logs' specification, section 4.3 [0]. +5.5 'snowflakes' below 'out' + + 'snowflakes' contains the subdirectory structure + + year SEP month SEP day + + Where the time related values are taken from the snowflake statistics + interval end. + + The files are named according to the structure in 4.5. + 6.0 The 'contrib' Folder The 'contrib' folder contains third-party contributions. diff --git a/src/test/java/org/torproject/metrics/collector/conf/ConfigurationTest.java b/src/test/java/org/torproject/metrics/collector/conf/ConfigurationTest.java index 4ac623e..3a69c0c 100644 --- a/src/test/java/org/torproject/metrics/collector/conf/ConfigurationTest.java +++ b/src/test/java/org/torproject/metrics/collector/conf/ConfigurationTest.java @@ -38,7 +38,7 @@ public class ConfigurationTest { public void testKeyCount() { assertEquals("The number of properties keys in enum Key changed." + "\n This test class should be adapted.", - 53, Key.values().length); + 59, Key.values().length); } @Test() diff --git a/src/test/java/org/torproject/metrics/collector/cron/CollecTorMainTest.java b/src/test/java/org/torproject/metrics/collector/cron/CollecTorMainTest.java index 78b6ac7..d0fe173 100644 --- a/src/test/java/org/torproject/metrics/collector/cron/CollecTorMainTest.java +++ b/src/test/java/org/torproject/metrics/collector/cron/CollecTorMainTest.java @@ -72,6 +72,7 @@ public class CollecTorMainTest { case "Exitlist": case "OnionPerf": case "Webstats": + case "SnowflakeStats": assertNotNull("Property '" + key + "' not specified in " + Main.CONF_FILE + ".", props.getProperty(key)); diff --git a/src/test/java/org/torproject/metrics/collector/cron/SchedulerTest.java b/src/test/java/org/torproject/metrics/collector/cron/SchedulerTest.java index e0496fb..3f20646 100644 --- a/src/test/java/org/torproject/metrics/collector/cron/SchedulerTest.java +++ b/src/test/java/org/torproject/metrics/collector/cron/SchedulerTest.java @@ -31,7 +31,9 @@ public class SchedulerTest { + "UpdateindexActivated=true\nUpdateindexPeriodMinutes=1\n" + "UpdateindexOffsetMinutes=0\n" + "BridgedescsActivated=true\nBridgedescsPeriodMinutes=1\n" - + "BridgedescsOffsetMinutes=0\n"; + + "BridgedescsOffsetMinutes=0\n" + + "SnowflakeStatsActivated=true\nSnowflakeStatsPeriodMinutes=1\n" + + "SnowflakeStatsOffsetMinutes=0\n"; @Test() public void testSimpleSchedule() throws Exception { @@ -42,6 +44,7 @@ public class SchedulerTest { ctms.put(Key.BridgedescsActivated, Dummy.class); ctms.put(Key.RelaydescsActivated, Dummy.class); ctms.put(Key.ExitlistsActivated, Dummy.class); + ctms.put(Key.SnowflakeStatsActivated, Dummy.class); ctms.put(Key.UpdateindexActivated, Dummy.class); Field schedulerField = Scheduler.class.getDeclaredField("scheduler"); schedulerField.setAccessible(true); @@ -74,6 +77,7 @@ public class SchedulerTest { ctms.put(Key.BridgedescsActivated, Counter.class); ctms.put(Key.RelaydescsActivated, Counter.class); ctms.put(Key.ExitlistsActivated, Counter.class); + ctms.put(Key.SnowflakeStatsActivated, Counter.class); ctms.put(Key.UpdateindexActivated, Counter.class); conf.setProperty(Key.BridgeSources.name(), "Local"); conf.setProperty(Key.RelaySources.name(), "Remote"); @@ -84,7 +88,7 @@ public class SchedulerTest { schedulerField.get(Scheduler.getInstance()); Scheduler.getInstance().scheduleModuleRuns(ctms, conf); Scheduler.getInstance().shutdownScheduler(); - assertEquals(5, Counter.count.get()); + assertEquals(6, Counter.count.get()); } @Ignore("This test takes 180 seconds, which is too long.") @@ -97,6 +101,7 @@ public class SchedulerTest { ctms.put(Key.BridgedescsActivated, Broken.class); ctms.put(Key.RelaydescsActivated, Broken.class); ctms.put(Key.ExitlistsActivated, Broken.class); + ctms.put(Key.SnowflakeStatsActivated, Broken.class); ctms.put(Key.UpdateindexActivated, Broken.class); Field schedulerField = Scheduler.class.getDeclaredField("scheduler"); schedulerField.setAccessible(true);

1 0

[collector/release] Prepare for 1.10.0 release.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit 1037677d9ec1c94a26d8dccff454e348d29bd344 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Sep 12 10:50:31 2019 +0200 Prepare for 1.10.0 release. --- CHANGELOG.md | 2 +- build.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c537d22..b38b124 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -# Changes in version 1.10.0 - 2019-??-?? +# Changes in version 1.10.0 - 2019-09-12 * Medium changes - Changed local import of bandwidth files to include the parent diff --git a/build.xml b/build.xml index d1d5b70..f4812c4 100644 --- a/build.xml +++ b/build.xml @@ -9,7 +9,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.9.1-dev" /> + <property name="release.version" value="1.10.0" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.7.0" />

1 0

[collector/release] Also update metrics-lib version for GitLab's CI.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit d132d40f8b3723e41ce7baf32d624a38d08edeba Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Sep 12 10:20:12 2019 +0200 Also update metrics-lib version for GitLab's CI. --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index b67b5a4..5bd5623 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,6 @@ variables: GIT_STRATEGY: clone - METRICS_LIB_VERSION: "2.6.2" + METRICS_LIB_VERSION: "2.7.0" JAVA_TOOL_OPTIONS: "-Dfile.encoding=UTF-8" stages:

1 0

[collector/master] Prepare for 1.10.0 release.
by karsten＠torproject.org 12 Sep '19

12 Sep '19

commit 1037677d9ec1c94a26d8dccff454e348d29bd344 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Sep 12 10:50:31 2019 +0200 Prepare for 1.10.0 release. --- CHANGELOG.md | 2 +- build.xml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c537d22..b38b124 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -# Changes in version 1.10.0 - 2019-??-?? +# Changes in version 1.10.0 - 2019-09-12 * Medium changes - Changed local import of bandwidth files to include the parent diff --git a/build.xml b/build.xml index d1d5b70..f4812c4 100644 --- a/build.xml +++ b/build.xml @@ -9,7 +9,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.9.1-dev" /> + <property name="release.version" value="1.10.0" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.7.0" />

1 0