tor-commits January 2020

tor-commits@lists.torproject.org

19 participants
1596 discussions

[tor/master] test_util: remove UTIL_LEGACY_NO_WIN.
by nickm＠torproject.org 14 Jan '20

14 Jan '20

commit 5c6fc5250e2ab4f0ffb935d018ece39bfa274f87 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Dec 20 09:57:47 2019 -0500 test_util: remove UTIL_LEGACY_NO_WIN. It was only used in one place. --- src/test/test_util.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/src/test/test_util.c b/src/test/test_util.c index ff0247fa0..cd745c442 100644 --- a/src/test/test_util.c +++ b/src/test/test_util.c @@ -6295,15 +6295,12 @@ test_util_map_anon_nofork(void *arg) #ifdef _WIN32 #define UTIL_TEST_NO_WIN(n, f) { #n, NULL, TT_SKIP, NULL, NULL } #define UTIL_TEST_WIN_ONLY(n, f) UTIL_TEST(n, (f)) -#define UTIL_LEGACY_NO_WIN(n) UTIL_TEST_NO_WIN(n, 0) #elif defined(__ANDROID__) #define UTIL_TEST_NO_WIN(n, f) { #n, NULL, TT_SKIP, NULL, NULL } #define UTIL_TEST_WIN_ONLY(n, f) { #n, NULL, TT_SKIP, NULL, NULL } -#define UTIL_LEGACY_NO_WIN(n) UTIL_TEST_NO_WIN(n, 0) #else #define UTIL_TEST_NO_WIN(n, f) UTIL_TEST(n, (f)) #define UTIL_TEST_WIN_ONLY(n, f) { #n, NULL, TT_SKIP, NULL, NULL } -#define UTIL_LEGACY_NO_WIN(n) UTIL_LEGACY(n) #endif /* defined(_WIN32) */ struct testcase_t util_tests[] = { @@ -6314,7 +6311,7 @@ struct testcase_t util_tests[] = { UTIL_LEGACY(config_line_comment_character), UTIL_LEGACY(config_line_escaped_content), UTIL_LEGACY(config_line_crlf), - UTIL_LEGACY_NO_WIN(expand_filename), + UTIL_TEST_NO_WIN(expand_filename, 0), UTIL_LEGACY(escape_string_socks), UTIL_LEGACY(string_is_key_value), UTIL_LEGACY(strmisc),

1 0

[tor/master] Android: skip tests that check UNIX user account setup
by nickm＠torproject.org 14 Jan '20

14 Jan '20

commit a96317a79dac9d82c334e206c7c524e77f13217e Author: Hans-Christoph Steiner <hans(a)eds.org> Date: Wed Oct 16 18:52:36 2019 +0200 Android: skip tests that check UNIX user account setup * there is no ~ or HOME in Android * UIDs and GIDs in Android represent apps, not users, and Android apps cannot ever change UIDs This should be replaced by whatever @nmathewson wants to name this stuff: https://github.com/torproject/tor/pull/1436 --- src/test/test_util.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/test/test_util.c b/src/test/test_util.c index 7f7e157c1..ff0247fa0 100644 --- a/src/test/test_util.c +++ b/src/test/test_util.c @@ -5653,7 +5653,7 @@ test_util_touch_file(void *arg) ; } -#ifndef _WIN32 +#if !(defined(_WIN32) || defined (__ANDROID__)) static void test_util_pwdb(void *arg) { @@ -5725,7 +5725,7 @@ test_util_pwdb(void *arg) tor_free(dir); teardown_capture_of_logs(); } -#endif /* !defined(_WIN32) */ +#endif /* !(defined(_WIN32) || defined (__ANDROID__)) */ static void test_util_calloc_check(void *arg) @@ -6296,6 +6296,10 @@ test_util_map_anon_nofork(void *arg) #define UTIL_TEST_NO_WIN(n, f) { #n, NULL, TT_SKIP, NULL, NULL } #define UTIL_TEST_WIN_ONLY(n, f) UTIL_TEST(n, (f)) #define UTIL_LEGACY_NO_WIN(n) UTIL_TEST_NO_WIN(n, 0) +#elif defined(__ANDROID__) +#define UTIL_TEST_NO_WIN(n, f) { #n, NULL, TT_SKIP, NULL, NULL } +#define UTIL_TEST_WIN_ONLY(n, f) { #n, NULL, TT_SKIP, NULL, NULL } +#define UTIL_LEGACY_NO_WIN(n) UTIL_TEST_NO_WIN(n, 0) #else #define UTIL_TEST_NO_WIN(n, f) UTIL_TEST(n, (f)) #define UTIL_TEST_WIN_ONLY(n, f) { #n, NULL, TT_SKIP, NULL, NULL }

1 0

[tor/master] changes file for 32825
by nickm＠torproject.org 14 Jan '20

14 Jan '20

commit 8ab9f93993dd53ae0fdadae88da48a660ad3f4e3 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Dec 20 10:06:08 2019 -0500 changes file for 32825 --- changes/bug32825 | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/changes/bug32825 b/changes/bug32825 new file mode 100644 index 000000000..c51f2f857 --- /dev/null +++ b/changes/bug32825 @@ -0,0 +1,4 @@ + o Minor features (portability, android): + - When building for Android, disable some tests that depend on + $HOME and/or pwdb, which Android doesn't have. Closes ticket 32825. + Patch from Hans-Christoph Steiner.

1 0

[tor/master] Merge remote-tracking branch 'tor-github/pr/1659'
by nickm＠torproject.org 14 Jan '20

14 Jan '20

commit 75741c6b3a2c180788a89a7eaca6d790e5d0606c Merge: 1ba193b7a 8d288eb3a Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Jan 14 12:40:07 2020 -0500 Merge remote-tracking branch 'tor-github/pr/1659' changes/bug32942 | 4 ++++ doc/tor.1.txt | 3 ++- src/app/config/config.c | 5 +++++ 3 files changed, 11 insertions(+), 1 deletion(-)

1 0

[tor/master] Deprecate the ClientAutoIPv6ORPort option
by nickm＠torproject.org 14 Jan '20

14 Jan '20

commit 8d288eb3a7738e50a371db63d83ea37f4cfb67fb Author: Neel Chauhan <neel(a)neelc.org> Date: Mon Jan 13 14:55:16 2020 -0800 Deprecate the ClientAutoIPv6ORPort option --- changes/bug32942 | 4 ++++ doc/tor.1.txt | 3 ++- src/app/config/config.c | 5 +++++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/changes/bug32942 b/changes/bug32942 new file mode 100644 index 000000000..0a4ef9b09 --- /dev/null +++ b/changes/bug32942 @@ -0,0 +1,4 @@ + o Deprecated features: + - Deprecate the ClientAutoIPv6ORPort option. This option was not true + Happy Eyeballs, and often failed on connections that weren't reliably + dual-stack. Closes ticket 32942. Patch by Neel Chauhan. diff --git a/doc/tor.1.txt b/doc/tor.1.txt index e193db238..d1583c3dc 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -1045,7 +1045,8 @@ The following options are useful only for clients (that is, if If this option is set to 1, Tor clients randomly prefer a node's IPv4 or IPv6 ORPort. The random preference is set every time a node is loaded from a new consensus or bridge config. When this option is set to 1, - **ClientPreferIPv6ORPort** is ignored. (Default: 0) + **ClientPreferIPv6ORPort** is ignored. (Default: 0) (DEPRECATED: This + option is unreliable if a connection isn't reliably dual-stack.) [[ClientBootstrapConsensusAuthorityDownloadInitialDelay]] **ClientBootstrapConsensusAuthorityDownloadInitialDelay** __N__:: Initial delay in seconds for when clients should download consensuses from authorities diff --git a/src/app/config/config.c b/src/app/config/config.c index d197e2b0c..7d5470d5e 100644 --- a/src/app/config/config.c +++ b/src/app/config/config.c @@ -831,6 +831,11 @@ static const config_deprecation_t option_deprecation_notes_[] = { "effect on clients since 0.2.8." }, /* End of options deprecated since 0.3.2.2-alpha. */ + /* Options deprecated since 0.4.3.1-alpha. */ + { "ClientAutoIPv6ORPort", "This option is unreliable if a connection isn't " + "reliably dual-stack."}, + /* End of options deprecated since 0.4.3.1-alpha. */ + { NULL, NULL } };

1 0

[translation/support-portal] https://gitweb.torproject.org/translation.git/commit/?h=support-portal
by translation＠torproject.org 14 Jan '20

14 Jan '20

commit fc076a23f5eb1dbf077c2254a8bcf14cf14f1611 Author: Translation commit bot <translation(a)torproject.org> Date: Tue Jan 14 16:53:51 2020 +0000 https://gitweb.torproject.org/translation.git/commit/?h=support-portal --- contents+ar.po | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/contents+ar.po b/contents+ar.po index e4f8e473b8..2da4e4de3f 100644 --- a/contents+ar.po +++ b/contents+ar.po @@ -2851,6 +2851,11 @@ msgid "" "they are Tor exit relays), and tries to warn any connections coming from " "those IP addresses that recent queries indicate an infection." msgstr "" +"هناك تفسير بديل هو أن جوجل يحاول أن يكشف أنواع معينة من برامج التجسس " +"أوالفيروسات التى ترسل استفسارات مميزة إلى بحث جوجل. يقوم جوجل بتسجيل عناوين " +"الأي بي IP التى تأتى منها هذه الاستفسارات (بدون أن يدرك أنها وصلات خروج " +"تور)، ويحاول أن يحذر أى اتصالات تأتى من عناوين الأي بي هذه أن الاستفسارات " +"الأخيرة تظهر إصابتها ببرمجيات خبيثة." #: https//support.torproject.org/tbb/tbb-44/ #: (content/tbb/tbb-44/contents+en.lrquestion.description) @@ -2859,11 +2864,13 @@ msgid "" " deter or block Tor use. The error message about an infected machine should " "clear up again after a short time." msgstr "" +"على حد علمنا، جوجل لا يفعل أي شيء عن عمد خصيصا لردع أو حجب إستخدام تور. " +"رسالة الخطأ عن إصابة الجهاز من المفترض أن تختفى ثانية بعد وقت قليل." #: https//support.torproject.org/tbb/tbb-45/ #: (content/tbb/tbb-45/contents+en.lrquestion.title) msgid "Gmail warns me that my account may have been compromised" -msgstr "" +msgstr "جي ميل Gmail يحذرني أن حسابى ربما تم اختراقه." #: https//support.torproject.org/tbb/tbb-45/ #: (content/tbb/tbb-45/contents+en.lrquestion.description) @@ -2871,6 +2878,8 @@ msgid "" "Sometimes, after you've used Gmail over Tor, Google presents a pop-up " "notification that your account may have been compromised." msgstr "" +"أحيانا بعد استخدام جى ميل عير تور، جوجل يظهر رسالة تنبيهية أن حسابك ربما " +"يكون قد تم اختراقه." #: https//support.torproject.org/tbb/tbb-45/ #: (content/tbb/tbb-45/contents+en.lrquestion.description) @@ -2878,6 +2887,8 @@ msgid "" "The notification window lists a series of IP addresses and locations " "throughout the world recently used to access your account." msgstr "" +"النافذة التنبيهية تظهر قائمة من عناوين الأي بي IP والأماكن عبر العالم التى " +"استخدمت مؤخرا للدخول إلى حسابك." #: https//support.torproject.org/tbb/tbb-45/ #: (content/tbb/tbb-45/contents+en.lrquestion.description) @@ -2887,6 +2898,9 @@ msgid "" " was a good idea to confirm the account was being accessed by its rightful " "owner." msgstr "" +"بصفة عامة، هذا انذار كاذب؛ جوجل رأى تسجيل دخول عدة مرات من أماكن مختلفة، " +"وهذا نتيجة لاستخدام الخدمة عن طريق تور، وقرر أنها فكرة ملائمة أن يتوثق أن " +"الحساب قد تم الدخول إليه من مالكه الحقيقي." #: https//support.torproject.org/tbb/tbb-45/ #: (content/tbb/tbb-45/contents+en.lrquestion.description) @@ -2894,6 +2908,8 @@ msgid "" "Even though this may be a byproduct of using the service via Tor, that " "doesn't mean you can entirely ignore the warning." msgstr "" +"بالرغم أن هذا نتيجة ثانوية لاستخدام الخدمة عن طريق تور، هذا لا يعني أنك " +"تستطيع تجاهل الإنذار بالكلية." #: https//support.torproject.org/tbb/tbb-45/ #: (content/tbb/tbb-45/contents+en.lrquestion.description)

1 0

[collector/master] Avoid reprocessing webstats files.
by karsten＠torproject.org 14 Jan '20

14 Jan '20

commit d48163379c2604626a62da775aafe68b5be62186 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Dec 11 12:22:40 2019 +0100 Avoid reprocessing webstats files. Web servers typically provide us with the last 14 days of request logs. We shouldn't process the whole 14 days over and over. Instead we should only process new logs files and any other log files containing log lines from newly written dates. In some cases web servers stop serving a given virtual host or stop acting as web server at all. However, in these cases we're left with 14 days of logs per virtual host. Ideally, these logs would get cleaned up, but until that's the case, we should at least not reprocess these files over and over. In order to avoid reprocessing webstats files, we need a new state file with log dates contained in given input files. We use that state file to determine which of the previously processed webstats files to re-process, so that we can write complete daily logs. --- CHANGELOG.md | 1 + .../metrics/collector/webstats/LogMetadata.java | 17 +++ .../collector/webstats/SanitizeWeblogs.java | 155 ++++++++++++++++++--- .../collector/webstats/WebServerAccessLogImpl.java | 19 +++ 4 files changed, 170 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 73abdea..fe7937c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,7 @@ * Medium changes - Give up on periodically checking the configuration file for updates and reloading it in case of changes. + - Avoid reprocessing webstats files. * Minor changes - Remove dependency on metrics-lib's internal package. diff --git a/src/main/java/org/torproject/metrics/collector/webstats/LogMetadata.java b/src/main/java/org/torproject/metrics/collector/webstats/LogMetadata.java index 879e8d7..b30c13a 100644 --- a/src/main/java/org/torproject/metrics/collector/webstats/LogMetadata.java +++ b/src/main/java/org/torproject/metrics/collector/webstats/LogMetadata.java @@ -81,5 +81,22 @@ public class LogMetadata { } return Optional.ofNullable(metadata); } + + @Override + public boolean equals(Object other) { + if (this == other) { + return true; + } + if (other == null || getClass() != other.getClass()) { + return false; + } + LogMetadata that = (LogMetadata) other; + return path.toString().equals(that.path.toString()); + } + + @Override + public int hashCode() { + return path.hashCode(); + } } diff --git a/src/main/java/org/torproject/metrics/collector/webstats/SanitizeWeblogs.java b/src/main/java/org/torproject/metrics/collector/webstats/SanitizeWeblogs.java index fc7c64f..b3fee06 100644 --- a/src/main/java/org/torproject/metrics/collector/webstats/SanitizeWeblogs.java +++ b/src/main/java/org/torproject/metrics/collector/webstats/SanitizeWeblogs.java @@ -25,14 +25,18 @@ import org.slf4j.LoggerFactory; import java.io.BufferedReader; import java.io.ByteArrayOutputStream; import java.io.File; +import java.io.IOException; import java.io.InputStreamReader; import java.io.OutputStream; import java.nio.file.Files; import java.nio.file.Path; +import java.nio.file.Paths; import java.time.LocalDate; import java.time.format.DateTimeFormatter; import java.util.ArrayList; import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Optional; @@ -56,8 +60,10 @@ public class SanitizeWeblogs extends CollecTorMain { private static final int LIMIT = 2; private static final String WEBSTATS = "webstats"; - private String outputPathName; - private String recentPathName; + private Path outputDirectory; + private Path recentDirectory; + private Path processedWebstatsFile; + private boolean limits; /** @@ -84,14 +90,22 @@ public class SanitizeWeblogs extends CollecTorMain { try { Files.createDirectories(this.config.getPath(Key.OutputPath)); Files.createDirectories(this.config.getPath(Key.RecentPath)); - this.outputPathName = this.config.getPath(Key.OutputPath).toString(); - this.recentPathName = this.config.getPath(Key.RecentPath).toString(); + Files.createDirectories(this.config.getPath(Key.StatsPath)); + this.outputDirectory = this.config.getPath(Key.OutputPath); + this.recentDirectory = this.config.getPath(Key.RecentPath); + this.processedWebstatsFile = this.config.getPath(Key.StatsPath) + .resolve("processed-webstats"); this.limits = this.config.getBool(Key.WebstatsLimits); Set<SourceType> sources = this.config.getSourceTypeSet( Key.WebstatsSources); if (sources.contains(SourceType.Local)) { log.info("Processing logs using batch value {}.", BATCH); - findCleanWrite(this.config.getPath(Key.WebstatsLocalOrigins)); + Map<LogMetadata, Set<LocalDate>> previouslyProcessedWebstats + = this.readProcessedWebstats(); + Map<LogMetadata, Set<LocalDate>> newlyProcessedWebstats + = this.findCleanWrite(this.config.getPath(Key.WebstatsLocalOrigins), + previouslyProcessedWebstats); + this.writeProcessedWebstats(newlyProcessedWebstats); long cutOffMillis = System.currentTimeMillis() - 3L * 24L * 60L * 60L * 1000L; PersistenceUtils.cleanDirectory(this.config.getPath(Key.RecentPath), @@ -103,7 +117,32 @@ public class SanitizeWeblogs extends CollecTorMain { } } - private void findCleanWrite(Path dir) { + private Map<LogMetadata, Set<LocalDate>> readProcessedWebstats() { + Map<LogMetadata, Set<LocalDate>> processedWebstats = new HashMap<>(); + if (Files.exists(this.processedWebstatsFile)) { + try { + for (String line : Files.readAllLines(this.processedWebstatsFile)) { + String[] lineParts = line.split(",", 2); + Optional<LogMetadata> logMetadata + = LogMetadata.create(Paths.get(lineParts[1])); + if (logMetadata.isPresent()) { + processedWebstats.putIfAbsent(logMetadata.get(), new HashSet<>()); + LocalDate containedLogDate = LocalDate.parse(lineParts[0]); + processedWebstats.get(logMetadata.get()).add(containedLogDate); + } + } + } catch (IOException e) { + log.error("Cannot read state file {}.", this.processedWebstatsFile, e); + } + log.debug("Read state file containing {} log files.", + processedWebstats.size()); + } + return processedWebstats; + } + + private Map<LogMetadata, Set<LocalDate>> findCleanWrite(Path dir, + Map<LogMetadata, Set<LocalDate>> previouslyProcessedWebstats) { + Map<LogMetadata, Set<LocalDate>> newlyProcessedWebstats = new HashMap<>(); LogFileMap fileMapIn = new LogFileMap(dir); log.info("Found log files for {} virtual hosts.", fileMapIn.size()); for (Map.Entry<String,TreeMap<String,TreeMap<LocalDate,LogMetadata>>> @@ -113,24 +152,76 @@ public class SanitizeWeblogs extends CollecTorMain { : virtualEntry.getValue().entrySet()) { String physicalHost = physicalEntry.getKey(); log.info("Processing logs for {} on {}.", virtualHost, physicalHost); - Map<LocalDate, Map<String, Long>> linesByDate - = physicalEntry.getValue().values().stream().parallel() - .flatMap(metadata -> sanitzedLineStream(metadata).entrySet() - .stream()) - .collect(groupingBy(Map.Entry::getKey, - reducing(Collections.emptyMap(), Map.Entry::getValue, - (e1, e2) -> Stream.concat(e1.entrySet().stream(), e2.entrySet() - .stream()) - .collect(groupingByConcurrent(Map.Entry::getKey, - summingLong(Map.Entry::getValue)))))); - LocalDate[] interval = determineInterval(linesByDate.keySet()); - linesByDate.entrySet().stream() - .filter((entry) -> entry.getKey().isAfter(interval[0]) - && entry.getKey().isBefore(interval[1])).parallel() + /* Go through current input log files for given virtual and physical + * host, and either look up contained log dates from the last execution, + * or parse files to memory now. */ + Map<LocalDate, Map<String, Long>> sanitizedLinesByDate + = new HashMap<>(); + Set<LogMetadata> previouslyReadFiles = new HashSet<>(); + for (LogMetadata logMetadata : physicalEntry.getValue().values()) { + Set<LocalDate> containedLogDates; + if (previouslyProcessedWebstats.containsKey(logMetadata)) { + containedLogDates = previouslyProcessedWebstats.get(logMetadata); + for (LocalDate date : containedLogDates) { + sanitizedLinesByDate.putIfAbsent(date, new TreeMap<>()); + } + previouslyReadFiles.add(logMetadata); + } else { + containedLogDates = sanitizeWebstatsLog(sanitizedLinesByDate, + logMetadata); + } + newlyProcessedWebstats.put(logMetadata, containedLogDates); + } + /* Determine log dates that are safe to be written to disk now and that + * we didn't write to disk before. */ + Set<LocalDate> storeDates = new HashSet<>(); + LocalDate[] interval = determineInterval(sanitizedLinesByDate.keySet()); + for (LocalDate newDate : sanitizedLinesByDate.keySet()) { + if (newDate.isAfter(interval[0]) && newDate.isBefore(interval[1])) { + WebServerAccessLogPersistence walp + = new WebServerAccessLogPersistence( + new WebServerAccessLogImpl(virtualHost, physicalHost, newDate)); + Path outputPath = this.outputDirectory + .resolve(walp.getStoragePath()); + if (!Files.exists(outputPath)) { + storeDates.add(newDate); + } + } + } + /* Reprocess previously read files containing log dates that we're going + * to write to disk below. */ + for (LogMetadata previouslyReadFile : previouslyReadFiles) { + if (!Collections.disjoint(storeDates, + newlyProcessedWebstats.get(previouslyReadFile))) { + sanitizeWebstatsLog(sanitizedLinesByDate, previouslyReadFile); + } + } + /* Write sanitized log files to disk. */ + sanitizedLinesByDate.entrySet().stream() + .filter((entry) -> storeDates.contains(entry.getKey())).parallel() .forEach((entry) -> storeSortedAndForget(virtualHost, physicalHost, entry.getKey(), entry.getValue())); } } + return newlyProcessedWebstats; + } + + private Set<LocalDate> sanitizeWebstatsLog( + Map<LocalDate, Map<String, Long>> sanitizedLinesByDate, + LogMetadata logFile) { + Map<LocalDate, Map<String, Long>> newlySanitizedLinesByDate + = sanitzedLineStream(logFile); + for (Map.Entry<LocalDate, Map<String, Long>> e + : newlySanitizedLinesByDate.entrySet()) { + sanitizedLinesByDate.putIfAbsent(e.getKey(), new TreeMap<>()); + Map<String, Long> newlySanitizedLines + = sanitizedLinesByDate.get(e.getKey()); + for (Map.Entry<String, Long> e1 : e.getValue().entrySet()) { + newlySanitizedLines.put(e1.getKey(), + newlySanitizedLines.getOrDefault(e1.getKey(), 0L) + e1.getValue()); + } + } + return newlySanitizedLinesByDate.keySet(); } private void storeSortedAndForget(String virtualHost, String physicalHost, @@ -149,8 +240,8 @@ public class SanitizeWeblogs extends CollecTorMain { new WebServerAccessLogImpl(toCompressedBytes(retainedLines), new File(name), name)); log.debug("Storing {}.", name); - walp.storeOut(this.outputPathName); - walp.storeRecent(this.recentPathName); + walp.storeOut(this.outputDirectory.toString()); + walp.storeRecent(this.recentDirectory.toString()); } catch (DescriptorParseException dpe) { log.error("Cannot store log desriptor {}.", name, dpe); } catch (Throwable th) { // catch all else @@ -279,5 +370,25 @@ public class SanitizeWeblogs extends CollecTorMain { return Collections.emptyMap(); } + private void writeProcessedWebstats( + Map<LogMetadata, Set<LocalDate>> newlyProcessedWebstats) { + try { + if (!Files.exists(this.processedWebstatsFile.getParent())) { + Files.createDirectories(this.processedWebstatsFile.getParent()); + } + List<String> lines = new ArrayList<>(); + for (Map.Entry<LogMetadata, Set<LocalDate>> e + : newlyProcessedWebstats.entrySet()) { + for (LocalDate logLineDate : e.getValue()) { + lines.add(String.format("%s,%s", logLineDate, e.getKey().path)); + } + } + Files.write(this.processedWebstatsFile, lines); + } catch (IOException e) { + log.error("Cannot write state file {}.", this.processedWebstatsFile, e); + } + log.debug("Wrote state file containing {} log files.", + newlyProcessedWebstats.size()); + } } diff --git a/src/main/java/org/torproject/metrics/collector/webstats/WebServerAccessLogImpl.java b/src/main/java/org/torproject/metrics/collector/webstats/WebServerAccessLogImpl.java index af77c94..f091aa1 100644 --- a/src/main/java/org/torproject/metrics/collector/webstats/WebServerAccessLogImpl.java +++ b/src/main/java/org/torproject/metrics/collector/webstats/WebServerAccessLogImpl.java @@ -104,6 +104,25 @@ public class WebServerAccessLogImpl implements WebServerAccessLog { } } + /** + * Creates an empty WebServerAccessLog from the given filename parts. + * + * This instance is not intended to be written to disk, as it doesn't have + * any content. The main intention of this instance is to compute storage + * paths. + * + * @param virtualHost Virtual host name. + * @param physicalHost Physical host name. + * @param logDate Log date. + */ + protected WebServerAccessLogImpl(String virtualHost, String physicalHost, + LocalDate logDate) { + this.descriptorFile = null; + this.virtualHost = virtualHost; + this.physicalHost = physicalHost; + this.logDate = logDate; + } + @Override public InputStream decompressedByteStream() throws DescriptorParseException { try {

1 0

[collector/master] Add some real tests for the webstats module.
by karsten＠torproject.org 14 Jan '20

14 Jan '20

commit 3002d6bc6b6bf84953cf842cbf6b3b18dc944879 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Dec 11 12:16:05 2019 +0100 Add some real tests for the webstats module. --- src/build | 2 +- .../collector/webstats/SanitizeWeblogsTest.java | 290 ++++++++++++++++++++- 2 files changed, 287 insertions(+), 5 deletions(-) diff --git a/src/build b/src/build index eb16cb3..264e498 160000 --- a/src/build +++ b/src/build @@ -1 +1 @@ -Subproject commit eb16cb359db41722e6089bafb1e26808df4338df +Subproject commit 264e498f54a20f7d299daaf2533d043f880e6a8b diff --git a/src/test/java/org/torproject/metrics/collector/webstats/SanitizeWeblogsTest.java b/src/test/java/org/torproject/metrics/collector/webstats/SanitizeWeblogsTest.java index a550c41..21617b5 100644 --- a/src/test/java/org/torproject/metrics/collector/webstats/SanitizeWeblogsTest.java +++ b/src/test/java/org/torproject/metrics/collector/webstats/SanitizeWeblogsTest.java @@ -4,17 +4,299 @@ package org.torproject.metrics.collector.webstats; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import org.torproject.descriptor.Descriptor; +import org.torproject.descriptor.DescriptorParseException; +import org.torproject.descriptor.DescriptorSourceFactory; +import org.torproject.descriptor.WebServerAccessLog; +import org.torproject.metrics.collector.Main; +import org.torproject.metrics.collector.conf.Configuration; +import org.torproject.metrics.collector.conf.Key; + +import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream; +import org.junit.Before; +import org.junit.Rule; import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.BufferedWriter; +import java.io.IOException; +import java.io.OutputStreamWriter; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.SortedMap; +import java.util.TreeMap; public class SanitizeWeblogsTest { + /** Sample original web server access logs as input for tests. */ + private static final String[][] inputLogs = new String[][] { + { "metrics.torproject.org-access.log-20191120.gz", + "0.0.0.0 - - [19/Nov/2019:00:00:00 +0000] " + + "\"GET /networksize.html HTTP/1.1\" 200 3269 \"-\" \"-\" -\n" + + "0.0.0.0 - - [19/Nov/2019:00:00:00 +0000] " + + "\"GET /networksize.png?start=2019-08-21&end=2019-11-19 HTTP/1.1\" " + + "200 39383 \"-\" \"-\" -\n" + + "0.0.0.0 - - [19/Nov/2019:00:00:00 +0000] " + + "\"GET /userstats-relay-country.html HTTP/1.1\" 200 7350 " + + "\"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"GET /collector/recent/relay-descriptors/ HTTP/1.1\" 200 10227 " + + "\"-\" \"-\" -\n" }, + { "metrics.torproject.org-access.log-20191121.gz", + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"HEAD /collector/recent/relay-descriptors/microdescs/ " + + "HTTP/1.1\" 200 - \"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"HEAD /collector/recent/exit-lists/ HTTP/1.1\" 200 " + + "- \"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"GET /collector/archive/bridge-descriptors/extra-infos/ " + + "HTTP/1.1\" 200 48013 \"-\" \"-\" -\n" + + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /images/cc/sk.png HTTP/1.1\" 200 395 \"-\" \"-\" -\n" }, + { "metrics.torproject.org-access.log-20191122.gz", + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /images/favicon.ico HTTP/1.1\" 200 1150 \"-\" \"-\" -\n" + + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /images/flags/authority.png HTTP/1.1\" 200 325 " + + "\"https://metrics.torproject.org/rs.html\" \"-\" -\n" + + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /news.atom HTTP/1.1\" 200 36362 \"-\" \"-\" -\n" + + "0.0.0.0 - - [22/Nov/2019:00:00:00 +0000] " + + "\"GET /onionperf-buildtimes.csv HTTP/1.1\" 200 270336 " + + "\"-\" \"-\" -\n" }, + { "metrics.torproject.org-access.log-20191123.gz", + "0.0.0.0 - - [22/Nov/2019:00:00:00 +0000] " + + "\"GET /userstats-relay-country.html?" + + "start=2010-01-01&end=2019-11-22&country=vn&events=off HTTP/1.1\" " + + "200 35517 \"-\" \"-\" -\n" + + "0.0.0.0 - - [22/Nov/2019:00:00:00 +0000] " + + "\"GET /userstats-relay-country.png?" + + "start=2010-01-01&end=2019-11-22&country=vn&events=off HTTP/1.1\" " + + "200 28041 \"-\" \"-\" -\n" + + "0.0.0.0 - - [22/Nov/2019:00:00:00 +0000] " + + "\"GET /userstats-relay-country.png?" + + "start=2010-01-01&end=2019-11-22&country=vn&events=off HTTP/1.1\" " + + "200 28041 \"-\" \"-\" -\n" + + "0.0.0.0 - - [23/Nov/2019:00:00:00 +0000] \"GET / HTTP/1.1\" " + + "200 3336 \"-\" \"-\" -\n" } + }; + + /** Sanitized web server access logs as output of tests. */ + private static final String[][] outputLogs = new String[][] { + { "metrics.torproject.org_meronense.torproject.org_" + + "access.log_20191120.xz", + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"GET /collector/archive/bridge-descriptors/extra-infos/ " + + "HTTP/1.1\" 200 48013\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"GET /collector/recent/relay-descriptors/ HTTP/1.1\" 200 10227\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"HEAD /collector/recent/exit-lists/ HTTP/1.1\" 200 -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"HEAD /collector/recent/relay-descriptors/microdescs/ " + + "HTTP/1.1\" 200 -\n" }, + { "metrics.torproject.org_meronense.torproject.org_" + + "access.log_20191121.xz", + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /images/cc/sk.png HTTP/1.1\" 200 395\n" + + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /images/favicon.ico HTTP/1.1\" 200 1150\n" + + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /images/flags/authority.png HTTP/1.1\" 200 325\n" + + "0.0.0.0 - - [21/Nov/2019:00:00:00 +0000] " + + "\"GET /news.atom HTTP/1.1\" 200 36362\n" } + }; + + /** Temporary folder containing all files for this test. */ + @Rule + public TemporaryFolder temporaryFolder = new TemporaryFolder(); + + /** Directory containing web server logs to sanitize. */ + private Path inputDirectory; + + /** Directory storing all intermediate state that needs to be preserved + * between processing runs. */ + private Path statsDirectory; + + /** Directory holding sanitized bridge descriptor files for tarballs. */ + private Path outDirectory; + + /** Directory holding recent sanitized web server logs. */ + private Path recentDirectory; + + /** CollecTor configuration for this test. */ + private Configuration configuration; + + /** Prepares the temporary folder and the various builders for this + * test. */ + @Before + public void createTemporaryFolderAndBuilders() + throws IOException { + this.inputDirectory = this.temporaryFolder.newFolder("in", + "webstats", "meronense.torproject.org").toPath(); + this.statsDirectory = this.temporaryFolder.newFolder("stats").toPath(); + this.outDirectory = this.temporaryFolder.newFolder("out").toPath(); + this.recentDirectory = this.temporaryFolder.newFolder("indexed", "recent") + .toPath(); + this.initializeTestConfiguration(); + } + + /** Initializes a configuration for the bridge descriptor sanitizer. */ + private void initializeTestConfiguration() throws IOException { + this.configuration = new Configuration(); + this.configuration.load(getClass().getClassLoader().getResourceAsStream( + Main.CONF_FILE)); + this.configuration.setProperty(Key.WebstatsActivated.name(), "true"); + this.configuration.setProperty(Key.WebstatsLocalOrigins.name(), + this.inputDirectory.toString()); + this.configuration.setProperty(Key.StatsPath.name(), + this.statsDirectory.toString()); + this.configuration.setProperty(Key.RecentPath.name(), + this.recentDirectory.toString()); + this.configuration.setProperty(Key.OutputPath.name(), + this.outDirectory.toString()); + } + + private void writeInputFiles(String[] ... inputLogs) throws IOException { + for (String[] inputLog : inputLogs) { + Path inputLogFile = this.inputDirectory.resolve(inputLog[0]); + if (!Files.exists(inputLogFile.getParent())) { + Files.createDirectories(inputLogFile.getParent()); + } + try (BufferedWriter bw = new BufferedWriter(new OutputStreamWriter( + new GzipCompressorOutputStream( + Files.newOutputStream(inputLogFile))))) { + bw.write(inputLog[1]); + } + } + } + + private void deleteInputFiles(String[] ... deleteLogs) throws IOException { + for (String[] deleteLog : deleteLogs) { + Path deleteLogFile = this.inputDirectory.resolve(deleteLog[0]); + Files.delete(deleteLogFile); + } + } + + private void sanitizeWeblogs() { + SanitizeWeblogs sw = new SanitizeWeblogs(this.configuration); + sw.startProcessing(); + } + + private void compareResults(String[] ... outputLogs) + throws DescriptorParseException { + SortedMap<String, WebServerAccessLog> parsedLogs = new TreeMap<>(); + for (Descriptor descriptor + : DescriptorSourceFactory.createDescriptorReader() + .readDescriptors(this.recentDirectory.toFile())) { + if (!(descriptor instanceof WebServerAccessLog)) { + fail("Parsed descriptor of unknown type."); + } else { + WebServerAccessLog wsal = (WebServerAccessLog) descriptor; + parsedLogs.put(wsal.getDescriptorFile().getName(), wsal); + } + } + assertEquals(outputLogs.length, parsedLogs.size()); + for (String[] outputLog : outputLogs) { + String expectedLogFilename = outputLog[0]; + List<String> expectedLogLines = Arrays.asList(outputLog[1].split("\n")); + assertTrue(parsedLogs.containsKey(expectedLogFilename)); + List<String> actualLogLines = new ArrayList<>(); + parsedLogs.get(expectedLogFilename).logLines() + .forEach((line) -> actualLogLines.add(line.toString())); + assertEquals(expectedLogLines, actualLogLines); + } + } + + @Test + public void testSingleRun() throws Exception { + this.writeInputFiles(inputLogs); + this.sanitizeWeblogs(); + this.compareResults(outputLogs); + } + @Test - public void bytesForTest() { - String lines = "line\nline\nline\nline\nline\n" - + "line\nline\nline\nline\nline\n"; - assertEquals(lines, new String(SanitizeWeblogs.bytesFor("line", 10))); + public void testSubsequentRuns() throws Exception { + for (String[] inputLog : inputLogs) { + this.writeInputFiles(inputLog); + this.sanitizeWeblogs(); + } + this.compareResults(outputLogs); } + @Test + public void testSubsequentRunsReverseOrder() throws Exception { + for (int i = inputLogs.length - 1; i >= 0; i--) { + this.writeInputFiles(inputLogs[i]); + this.sanitizeWeblogs(); + } + this.compareResults(outputLogs); + } + + @Test + public void testSlidingWindow() throws Exception { + this.writeInputFiles(inputLogs[0], inputLogs[1], inputLogs[2]); + this.sanitizeWeblogs(); + this.compareResults(outputLogs[0]); + this.deleteInputFiles(inputLogs[0]); + this.writeInputFiles(inputLogs[3]); + this.sanitizeWeblogs(); + this.compareResults(outputLogs); + } + + @Test + public void testSingleDayNoLimit() throws Exception { + this.configuration.setProperty(Key.WebstatsLimits.name(), "false"); + this.writeInputFiles(new String[][] { + { "metrics.torproject.org-access.log-20191120.gz", + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"GET /collector/recent/relay-descriptors/ " + + "HTTP/1.1\" 200 10227 \"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"HEAD /collector/recent/relay-descriptors/microdescs/ " + + "HTTP/1.1\" 200 - \"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"HEAD /collector/recent/exit-lists/ " + + "HTTP/1.1\" 200 - \"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] " + + "\"GET /collector/archive/bridge-descriptors/extra-infos/ " + + "HTTP/1.1\" 200 48013 \"-\" \"-\" -\n" } }); + this.sanitizeWeblogs(); + this.compareResults(outputLogs[0]); + } + + @Test + public void testErrorLog() throws Exception { + this.configuration.setProperty(Key.WebstatsLimits.name(), "false"); + this.writeInputFiles(new String[][] { + { "metrics.torproject.org-error.log-20191121.gz", + "[Thu Nov 21 15:13:15.211234 2019] [authz_core:error] " + + "[pid 12920:tid 139635582793920] [client 127.0.0.1:59912]\n" } }); + this.sanitizeWeblogs(); + this.compareResults(); + } + + @Test + public void testNonMatchingLines() throws Exception { + this.configuration.setProperty(Key.WebstatsLimits.name(), "false"); + this.writeInputFiles(new String[][] { + { "metrics.torproject.org-access.log-20191121.gz", + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] \"GET /favicon.ico " + + "HTTP/1.1\" 404 8903 \"-\" \"-\" -\n" + + "0.0.0.0 - - [20/Nov/2019:00:00:00 +0000] \"POST /con.php " + + "HTTP/1.1\" 301 320 \"http://metrics.torproject.org/con.php\" " + + "\"-\" -\n" + + "[Thu Nov 21 15:13:15.211234 2019] [authz_core:error] " + + "[pid 12920:tid 139635582793920] [client 127.0.0.1:59912]\n" } }); + this.sanitizeWeblogs(); + this.compareResults(); + } }

1 0

[metrics-base/master] Stop using a security manager for executing tests.
by karsten＠torproject.org 14 Jan '20

14 Jan '20

commit 264e498f54a20f7d299daaf2533d043f880e6a8b Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Dec 12 09:44:06 2019 +0100 Stop using a security manager for executing tests. A while ago we started using a security manager for executing tests in order to prevent them from bothering production servers. However, keeping the security policy up to date for testing new functionality is becoming difficult: - The recently extended CollecTor module for indexing files asynchronously and creating hard links broke tests in unexpected way by requiring permission java.nio.file.LinkPermission "hard"; - Turns out that parallel streams used for sanitizing web server logs internally create threads that do not have the permissions as defined in our custom security policy. All in all it seems better to throw out the security manager at all and prevent tests from bothering production servers simply by not executing them on production servers. --- java/base.xml | 2 -- java/junittest.policy | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/java/base.xml b/java/base.xml index 6cb1693..6eb3f63 100644 --- a/java/base.xml +++ b/java/base.xml @@ -185,8 +185,6 @@ haltonfailure="true" printsummary="on"> <jvmarg value="-DLOGBASE=${generated}/test-logs"/> - <jvmarg value="-Djava.security.policy=${buildresources}/junittest.policy"/> - <jvmarg value="-Djava.security.manager"/> <classpath refid="test.classpath"/> <formatter type="plain" usefile="false"/> <batchtest> diff --git a/java/junittest.policy b/java/junittest.policy deleted file mode 100644 index 156938e..0000000 --- a/java/junittest.policy +++ /dev/null @@ -1,21 +0,0 @@ -/* Prevent tests from bothering production servers. */ - -grant { - permission java.io.FilePermission "<<ALL FILES>>", "read, write, delete, execute"; - permission java.util.PropertyPermission "*", "read, write"; - permission java.lang.RuntimePermission "setIO"; - permission java.lang.RuntimePermission "accessClassInPackage.sun.nio.fs"; - permission java.lang.RuntimePermission "accessDeclaredMembers"; - permission java.lang.RuntimePermission "getFileStoreAttributes"; - permission java.lang.RuntimePermission "getStackTrace"; - permission java.lang.RuntimePermission "modifyThread"; - permission java.lang.reflect.ReflectPermission "suppressAccessChecks"; - permission java.lang.RuntimePermission "shutdownHooks"; - permission java.lang.RuntimePermission "accessClassInPackage.sun.reflect"; - permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www.protocol.http"; - permission java.lang.RuntimePermission "accessClassInPackage.sun.net.www.http"; - permission java.lang.RuntimePermission "getProtectionDomain"; - permission java.lang.RuntimePermission "reflectionFactoryAccess"; - permission java.lang.RuntimePermission "setFactory"; - permission java.nio.file.LinkPermission "hard"; -};

1 0

[metrics-web/master] Document known inaccuracy in onion service stats.
by karsten＠torproject.org 14 Jan '20

14 Jan '20

commit ac802316cb977e041d9e0b9cd3b40a4ae3f2e19a Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Tue Jan 14 16:09:44 2020 +0100 Document known inaccuracy in onion service stats. As discussed on #23367. --- src/main/resources/web/jsps/reproducible-metrics.jsp | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/main/resources/web/jsps/reproducible-metrics.jsp b/src/main/resources/web/jsps/reproducible-metrics.jsp index ed3ad4c..0058ab6 100644 --- a/src/main/resources/web/jsps/reproducible-metrics.jsp +++ b/src/main/resources/web/jsps/reproducible-metrics.jsp @@ -817,7 +817,8 @@ We need to subtract out the multiple counting of .onion addresses to come up wit the service stores two replicas per descriptor using different descriptor identifiers, both descriptor replicas get stored to three different onion-service directories each, and the service changes descriptor identifiers once every 24 hours which leads to two different descriptor identifiers per replica. To be clear, this approximation is not entirely accurate. -For example, the two replicas or the descriptors with changed descriptor identifiers could have been stored to the same directory. +For example, the descriptors of roughly 1/24 of services are seen by 3 rather than 2 sets of onion-service directories, when a service changes descriptor identifiers once at the beginning of a relay's statistics interval and once again towards the end. +In some cases, the two replicas or the descriptors with changed descriptor identifiers could have been stored to the same directory. As another example, onion-service directories might have joined or left the network and other directories might have become responsible for storing a descriptor which also include that .onion address in their statistics. However, for the subsequent analysis, we assume that neither of these cases affects results substantially.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-commits January 2020