January 2020 - tor-commits - lists.torproject.org

[collector/release] Fix non-RunOnce mode.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit a6359c607438d05e66fa1aa0126af9a44a9bf22e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 11:40:35 2020 +0100 Fix non-RunOnce mode. This was accidentally broken in #32554. The RunOnce mode worked just fine, but the non-RunOnce mode terminated immediately after scheduling tasks. --- CHANGELOG.md | 7 ++++++- .../java/org/torproject/metrics/collector/Main.java | 8 ++++++-- .../metrics/collector/cron/Scheduler.java | 8 +++++--- .../metrics/collector/cron/ShutdownHook.java | 21 +++++++++++++++++++++ 4 files changed, 38 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3088c3..7161100 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,9 @@ -# Changes in version 1.1?.? - 2020-0?-?? +# Changes in version 1.14.1 - 2020-01-16 + + * Medium changes + - Stay alive after scheduling (daemon) threads for modules. This + was broken by taking out the periodic check for configuration + file updates. # Changes in version 1.14.0 - 2020-01-15 diff --git a/src/main/java/org/torproject/metrics/collector/Main.java b/src/main/java/org/torproject/metrics/collector/Main.java index 77bb2ca..3822353 100644 --- a/src/main/java/org/torproject/metrics/collector/Main.java +++ b/src/main/java/org/torproject/metrics/collector/Main.java @@ -87,12 +87,16 @@ public class Main { } else { conf.loadAndCheckConfiguration(confPath); } - Scheduler.getInstance().scheduleModuleRuns(collecTorMains, conf); + if (!Scheduler.getInstance().scheduleModuleRuns(collecTorMains, conf)) { + return; + } } catch (ConfigurationException ce) { printUsage(ce.getMessage()); return; } - Runtime.getRuntime().addShutdownHook(new ShutdownHook()); + ShutdownHook shutdownHook = new ShutdownHook(); + Runtime.getRuntime().addShutdownHook(shutdownHook); + shutdownHook.stayAlive(); } private static void printUsage(String msg) { diff --git a/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java b/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java index 03292f4..70d27f5 100644 --- a/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java +++ b/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java @@ -50,10 +50,10 @@ public final class Scheduler implements ThreadFactory { } /** - * Schedule all classes given according to the parameters in the - * the configuration. + * Schedule all classes given according to the parameters in the configuration + * and return whether the caller should wait for completion. */ - public void scheduleModuleRuns(Map<Key, + public boolean scheduleModuleRuns(Map<Key, Class<? extends CollecTorMain>> collecTorMains, Configuration conf) { try { gracePeriodMinutes = conf.getLong(Key.ShutdownGraceWaitMinutes); @@ -89,11 +89,13 @@ public final class Scheduler implements ThreadFactory { try { if (conf.getBool(Key.RunOnce)) { scheduler.invokeAll(runOnceMains); + return false; } } catch (ConfigurationException | InterruptedException | RejectedExecutionException | NullPointerException ex) { logger.error("Cannot schedule run-once: {}", ex.getMessage(), ex); } + return true; } private void scheduleExecutions(CollecTorMain ctm, int offset, int period) { diff --git a/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java b/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java index c8f0807..ec34a19 100644 --- a/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java +++ b/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java @@ -13,15 +13,36 @@ public final class ShutdownHook extends Thread { private static final Logger log = LoggerFactory.getLogger(ShutdownHook.class); + private boolean stayAlive = true; + /** Names the shutdown thread for debugging purposes. */ public ShutdownHook() { super("CollecTor-ShutdownThread"); } + /** + * Stay alive until the shutdown thread gets run. + */ + public void stayAlive() { + synchronized (this) { + while (this.stayAlive) { + try { + this.wait(); + } catch (InterruptedException e) { + /* Nothing we can do about this. */ + } + } + } + } + @Override public void run() { log.info("Shutdown in progress ... "); Scheduler.getInstance().shutdownScheduler(); + synchronized (this) { + this.stayAlive = false; + this.notify(); + } log.info("Shutdown finished. Exiting."); } }

1 0

[collector/release] Bump version to 1.14.0-dev.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit c75f0c781ec45b6b56aedea5738692d0404bdc8a Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Wed Jan 15 23:20:01 2020 +0100 Bump version to 1.14.0-dev. --- CHANGELOG.md | 3 +++ build.xml | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 898338c..d3088c3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,6 @@ +# Changes in version 1.1?.? - 2020-0?-?? + + # Changes in version 1.14.0 - 2020-01-15 * Medium changes diff --git a/build.xml b/build.xml index 5293d41..5d90b46 100644 --- a/build.xml +++ b/build.xml @@ -9,7 +9,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.14.0" /> + <property name="release.version" value="1.14.0-dev" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.10.0" />

1 0

[collector/release] Fix smoke test for recent scheduler changes.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit 0e9aedef745e8ed3ccbe7daffb6892197873ae9d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 12:09:36 2020 +0100 Fix smoke test for recent scheduler changes. --- src/test/java/org/torproject/metrics/collector/MainTest.java | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/test/java/org/torproject/metrics/collector/MainTest.java b/src/test/java/org/torproject/metrics/collector/MainTest.java index 02c16df..bf4e398 100644 --- a/src/test/java/org/torproject/metrics/collector/MainTest.java +++ b/src/test/java/org/torproject/metrics/collector/MainTest.java @@ -99,10 +99,9 @@ public class MainTest { assertEquals(0L, conf.length()); Main.main(new String[]{conf.toString()}); assertTrue(4_000L <= conf.length()); - changeFilePathsAndSetActivation(conf, - Key.OnionPerfActivated.name()); + changeFilePathsAndSetActivationAndRunOnce(conf, + Key.ExitlistsActivated.name()); Main.main(new String[]{conf.toString()}); - waitSec(2); } /** Wait for the given number of seconds. */ @@ -119,14 +118,15 @@ public class MainTest { } while ((toWait = start + 1_000L * sec - System.currentTimeMillis()) > 0); } - private void changeFilePathsAndSetActivation(File file, String activation) - throws Exception { + private void changeFilePathsAndSetActivationAndRunOnce(File file, + String activation) throws Exception { List<String> lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); BufferedWriter bw = Files.newBufferedWriter(file.toPath(), StandardCharsets.UTF_8); File in = tmpf.newFolder(); File out = tmpf.newFolder(); + String runOnce = "RunOnce"; String inStr = "in/"; String outStr = "out/"; for (String line : lines) { @@ -134,7 +134,7 @@ public class MainTest { line = line.replace(inStr, in.toString() + inStr); } else if (line.contains(outStr)) { line = line.replace(outStr, out.toString() + outStr); - } else if (line.contains(activation)) { + } else if (line.contains(activation) || line.contains(runOnce)) { line = line.replace("false", "true"); } bw.write(line);

1 0

[collector/release] Prepare for 1.14.1 release.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit b5ac823aec79bc95303043657e06a84831d9fdab Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 12:10:12 2020 +0100 Prepare for 1.14.1 release. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index 5d90b46..2e9e597 100644 --- a/build.xml +++ b/build.xml @@ -9,7 +9,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.14.0-dev" /> + <property name="release.version" value="1.14.1" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.10.0" />

1 0

[collector/master] Prepare for 1.14.1 release.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit b5ac823aec79bc95303043657e06a84831d9fdab Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 12:10:12 2020 +0100 Prepare for 1.14.1 release. --- build.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build.xml b/build.xml index 5d90b46..2e9e597 100644 --- a/build.xml +++ b/build.xml @@ -9,7 +9,7 @@ <property name="javadoc-title" value="CollecTor API Documentation"/> <property name="implementation-title" value="CollecTor" /> - <property name="release.version" value="1.14.0-dev" /> + <property name="release.version" value="1.14.1" /> <property name="project-main-class" value="org.torproject.metrics.collector.Main" /> <property name="name" value="collector"/> <property name="metricslibversion" value="2.10.0" />

1 0

[collector/master] Fix smoke test for recent scheduler changes.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit 0e9aedef745e8ed3ccbe7daffb6892197873ae9d Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 12:09:36 2020 +0100 Fix smoke test for recent scheduler changes. --- src/test/java/org/torproject/metrics/collector/MainTest.java | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/test/java/org/torproject/metrics/collector/MainTest.java b/src/test/java/org/torproject/metrics/collector/MainTest.java index 02c16df..bf4e398 100644 --- a/src/test/java/org/torproject/metrics/collector/MainTest.java +++ b/src/test/java/org/torproject/metrics/collector/MainTest.java @@ -99,10 +99,9 @@ public class MainTest { assertEquals(0L, conf.length()); Main.main(new String[]{conf.toString()}); assertTrue(4_000L <= conf.length()); - changeFilePathsAndSetActivation(conf, - Key.OnionPerfActivated.name()); + changeFilePathsAndSetActivationAndRunOnce(conf, + Key.ExitlistsActivated.name()); Main.main(new String[]{conf.toString()}); - waitSec(2); } /** Wait for the given number of seconds. */ @@ -119,14 +118,15 @@ public class MainTest { } while ((toWait = start + 1_000L * sec - System.currentTimeMillis()) > 0); } - private void changeFilePathsAndSetActivation(File file, String activation) - throws Exception { + private void changeFilePathsAndSetActivationAndRunOnce(File file, + String activation) throws Exception { List<String> lines = Files.readAllLines(file.toPath(), StandardCharsets.UTF_8); BufferedWriter bw = Files.newBufferedWriter(file.toPath(), StandardCharsets.UTF_8); File in = tmpf.newFolder(); File out = tmpf.newFolder(); + String runOnce = "RunOnce"; String inStr = "in/"; String outStr = "out/"; for (String line : lines) { @@ -134,7 +134,7 @@ public class MainTest { line = line.replace(inStr, in.toString() + inStr); } else if (line.contains(outStr)) { line = line.replace(outStr, out.toString() + outStr); - } else if (line.contains(activation)) { + } else if (line.contains(activation) || line.contains(runOnce)) { line = line.replace("false", "true"); } bw.write(line);

1 0

[collector/master] Fix non-RunOnce mode.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit a6359c607438d05e66fa1aa0126af9a44a9bf22e Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 11:40:35 2020 +0100 Fix non-RunOnce mode. This was accidentally broken in #32554. The RunOnce mode worked just fine, but the non-RunOnce mode terminated immediately after scheduling tasks. --- CHANGELOG.md | 7 ++++++- .../java/org/torproject/metrics/collector/Main.java | 8 ++++++-- .../metrics/collector/cron/Scheduler.java | 8 +++++--- .../metrics/collector/cron/ShutdownHook.java | 21 +++++++++++++++++++++ 4 files changed, 38 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3088c3..7161100 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,9 @@ -# Changes in version 1.1?.? - 2020-0?-?? +# Changes in version 1.14.1 - 2020-01-16 + + * Medium changes + - Stay alive after scheduling (daemon) threads for modules. This + was broken by taking out the periodic check for configuration + file updates. # Changes in version 1.14.0 - 2020-01-15 diff --git a/src/main/java/org/torproject/metrics/collector/Main.java b/src/main/java/org/torproject/metrics/collector/Main.java index 77bb2ca..3822353 100644 --- a/src/main/java/org/torproject/metrics/collector/Main.java +++ b/src/main/java/org/torproject/metrics/collector/Main.java @@ -87,12 +87,16 @@ public class Main { } else { conf.loadAndCheckConfiguration(confPath); } - Scheduler.getInstance().scheduleModuleRuns(collecTorMains, conf); + if (!Scheduler.getInstance().scheduleModuleRuns(collecTorMains, conf)) { + return; + } } catch (ConfigurationException ce) { printUsage(ce.getMessage()); return; } - Runtime.getRuntime().addShutdownHook(new ShutdownHook()); + ShutdownHook shutdownHook = new ShutdownHook(); + Runtime.getRuntime().addShutdownHook(shutdownHook); + shutdownHook.stayAlive(); } private static void printUsage(String msg) { diff --git a/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java b/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java index 03292f4..70d27f5 100644 --- a/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java +++ b/src/main/java/org/torproject/metrics/collector/cron/Scheduler.java @@ -50,10 +50,10 @@ public final class Scheduler implements ThreadFactory { } /** - * Schedule all classes given according to the parameters in the - * the configuration. + * Schedule all classes given according to the parameters in the configuration + * and return whether the caller should wait for completion. */ - public void scheduleModuleRuns(Map<Key, + public boolean scheduleModuleRuns(Map<Key, Class<? extends CollecTorMain>> collecTorMains, Configuration conf) { try { gracePeriodMinutes = conf.getLong(Key.ShutdownGraceWaitMinutes); @@ -89,11 +89,13 @@ public final class Scheduler implements ThreadFactory { try { if (conf.getBool(Key.RunOnce)) { scheduler.invokeAll(runOnceMains); + return false; } } catch (ConfigurationException | InterruptedException | RejectedExecutionException | NullPointerException ex) { logger.error("Cannot schedule run-once: {}", ex.getMessage(), ex); } + return true; } private void scheduleExecutions(CollecTorMain ctm, int offset, int period) { diff --git a/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java b/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java index c8f0807..ec34a19 100644 --- a/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java +++ b/src/main/java/org/torproject/metrics/collector/cron/ShutdownHook.java @@ -13,15 +13,36 @@ public final class ShutdownHook extends Thread { private static final Logger log = LoggerFactory.getLogger(ShutdownHook.class); + private boolean stayAlive = true; + /** Names the shutdown thread for debugging purposes. */ public ShutdownHook() { super("CollecTor-ShutdownThread"); } + /** + * Stay alive until the shutdown thread gets run. + */ + public void stayAlive() { + synchronized (this) { + while (this.stayAlive) { + try { + this.wait(); + } catch (InterruptedException e) { + /* Nothing we can do about this. */ + } + } + } + } + @Override public void run() { log.info("Shutdown in progress ... "); Scheduler.getInstance().shutdownScheduler(); + synchronized (this) { + this.stayAlive = false; + this.notify(); + } log.info("Shutdown finished. Exiting."); } }

1 0

[metrics-web/master] Update metrics-lib and ExoneraTor versions.
by karsten＠torproject.org 16 Jan '20

16 Jan '20

commit af1af0a911da24b38d69b38f3ed8cc1da9dd6a11 Author: Karsten Loesing <karsten.loesing(a)gmx.net> Date: Thu Jan 16 09:35:29 2020 +0100 Update metrics-lib and ExoneraTor versions. Also configure a base URL in order to turn ExoneraTor's permanent links into https:// links. --- CHANGELOG.md | 2 ++ build.xml | 4 ++-- src/main/resources/web.xml | 4 ++++ src/submods/metrics-lib | 2 +- 4 files changed, 9 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2682cfc..d5e6188 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ * Minor changes - Make Jetty host configurable. + - Configure a base URL in order to turn ExoneraTor's permanent + links into https:// links. # Changes in version 1.3.0 - 2019-11-09 diff --git a/build.xml b/build.xml index 83e69e7..aef5f1b 100644 --- a/build.xml +++ b/build.xml @@ -10,8 +10,8 @@ <property name="javadoc-title" value="MetricsWeb API Documentation"/> <property name="implementation-title" value="metrics-web" /> <property name="release.version" value="1.3.0-dev" /> - <property name="metricslibversion" value="2.9.1" /> - <property name="exoneratorversion" value="4.2.0" /> + <property name="metricslibversion" value="2.10.0" /> + <property name="exoneratorversion" value="4.3.0" /> <property name="jetty.version" value="-9.2.21.v20170120" /> <property name="warfile" value="metrics-web-${release.version}.war"/> diff --git a/src/main/resources/web.xml b/src/main/resources/web.xml index 08a3bcd..66cc3da 100644 --- a/src/main/resources/web.xml +++ b/src/main/resources/web.xml @@ -482,6 +482,10 @@ /srv/metrics.torproject.org/metrics/website/rserve/graphs/ </param-value> </context-param> + <context-param> + <param-name>baseUrl</param-name> + <param-value>https://metrics.torproject.org</param-value> + </context-param> <listener> <listener-class> diff --git a/src/submods/metrics-lib b/src/submods/metrics-lib index 5fb36b7..6408bd9 160000 --- a/src/submods/metrics-lib +++ b/src/submods/metrics-lib @@ -1 +1 @@ -Subproject commit 5fb36b70c884073963af8e9cb3a37fa907eb35b8 +Subproject commit 6408bd9a85890f06a74f5f077e49dffe33acc36e

1 0

[stem/master] Bump test for TAKEOWNERSHIP termination SLA
by atagar＠torproject.org 16 Jan '20

16 Jan '20

commit e64a118db8d789827183b6b66a6aa6c535868727 Author: Damian Johnson <atagar(a)torproject.org> Date: Wed Jan 15 17:37:44 2020 -0800 Bump test for TAKEOWNERSHIP termination SLA Tor checks for the owning process every fifteen seconds, so this test's timeout is way too low... https://github.com/torproject/stem/issues/52 Puzzling that this passes reliably for me. With that poll rate this should be failing far more often... --- test/integ/process.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/integ/process.py b/test/integ/process.py index e40c501a..274a8ec9 100644 --- a/test/integ/process.py +++ b/test/integ/process.py @@ -631,7 +631,7 @@ class TestProcess(unittest.TestCase): # give tor a few seconds to quit start_time = time.time() - while time.time() - start_time < 5: + while time.time() - start_time < 20: if tor_process.poll() == 0: return # tor exited

1 0

[tor/release-0.4.2] Merge remote-tracking branch 'tor-github/pr/1513' into maint-0.3.5
by teor＠torproject.org 16 Jan '20

16 Jan '20

commit 19954cffd7710ccac778aa5cf15ae1bc6a5d56fa Merge: 0b3763612 4d70e725d Author: teor <teor(a)torproject.org> Date: Thu Jan 16 09:57:27 2020 +1000 Merge remote-tracking branch 'tor-github/pr/1513' into maint-0.3.5 changes/ticket32407 | 4 ++++ src/core/mainloop/mainloop.c | 4 ++++ src/lib/evloop/compat_libevent.c | 10 ++++++++++ src/lib/evloop/compat_libevent.h | 3 +++ 4 files changed, 21 insertions(+)

1 0