October 2016 - tor-commits - lists.torproject.org

[tor-launcher/maint-0.2.10] Bug 20429: Do not open progress window if TOR_SKIP_LAUNCH=1
by gk＠torproject.org 27 Oct '16

27 Oct '16

commit 8aa78d3a78bbabe01759b63d837b09acdf53be42 Author: Kathy Brade <brade(a)pearlcrescent.com> Date: Wed Oct 26 15:00:06 2016 -0400 Bug 20429: Do not open progress window if TOR_SKIP_LAUNCH=1 After saving tor settings, if TOR_SKIP_LAUNCH=1 or if extensions.torlauncher.start_tor is false, avoid opening the progress dialog to monitor bootstrap progress. This avoids displaying a progress window that will not make progress (and the situation was made worse by the fact that clicking Cancel set DisableNetwork=1). Also, leave the network settings dialog open if we fail to set DisableNetwork=0. --- src/chrome/content/network-settings.js | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/chrome/content/network-settings.js b/src/chrome/content/network-settings.js index 2736f08..dc54445 100644 --- a/src/chrome/content/network-settings.js +++ b/src/chrome/content/network-settings.js @@ -1055,11 +1055,22 @@ function useSettings() { var settings = {}; settings[kTorConfKeyDisableNetwork] = false; - setConfAndReportErrors(settings, null); + let didApply = setConfAndReportErrors(settings, null); + if (!didApply) + return; gProtocolSvc.TorSendCommand("SAVECONF"); gTorProcessService.TorClearBootstrapError(); + // If we are not responsible for starting tor we do not monitor bootstrap + // status, so just close this dialog and return rather than opening the + // progress dialog (which will make no progress). + if (!TorLauncherUtil.shouldStartAndOwnTor) + { + close(); + return; + } + gIsBootstrapComplete = gTorProcessService.TorIsBootstrapDone; if (!gIsBootstrapComplete) openProgressDialog();

1 0

[tor-launcher/maint-0.2.9] Bug 20429: Do not open progress window if TOR_SKIP_LAUNCH=1
by gk＠torproject.org 27 Oct '16

27 Oct '16

commit 2014143e0081170267a34e3a102878999dee6a29 Author: Kathy Brade <brade(a)pearlcrescent.com> Date: Wed Oct 26 15:00:06 2016 -0400 Bug 20429: Do not open progress window if TOR_SKIP_LAUNCH=1 After saving tor settings, if TOR_SKIP_LAUNCH=1 or if extensions.torlauncher.start_tor is false, avoid opening the progress dialog to monitor bootstrap progress. This avoids displaying a progress window that will not make progress (and the situation was made worse by the fact that clicking Cancel set DisableNetwork=1). Also, leave the network settings dialog open if we fail to set DisableNetwork=0. --- src/chrome/content/network-settings.js | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/chrome/content/network-settings.js b/src/chrome/content/network-settings.js index 2736f08..dc54445 100644 --- a/src/chrome/content/network-settings.js +++ b/src/chrome/content/network-settings.js @@ -1055,11 +1055,22 @@ function useSettings() { var settings = {}; settings[kTorConfKeyDisableNetwork] = false; - setConfAndReportErrors(settings, null); + let didApply = setConfAndReportErrors(settings, null); + if (!didApply) + return; gProtocolSvc.TorSendCommand("SAVECONF"); gTorProcessService.TorClearBootstrapError(); + // If we are not responsible for starting tor we do not monitor bootstrap + // status, so just close this dialog and return rather than opening the + // progress dialog (which will make no progress). + if (!TorLauncherUtil.shouldStartAndOwnTor) + { + close(); + return; + } + gIsBootstrapComplete = gTorProcessService.TorIsBootstrapDone; if (!gIsBootstrapComplete) openProgressDialog();

1 0

[tor-launcher/master] Bug 20429: Do not open progress window if TOR_SKIP_LAUNCH=1
by gk＠torproject.org 27 Oct '16

27 Oct '16

commit c12d56470b7164c33b3cb2e48a90dc65151a9a26 Author: Kathy Brade <brade(a)pearlcrescent.com> Date: Wed Oct 26 15:00:06 2016 -0400 Bug 20429: Do not open progress window if TOR_SKIP_LAUNCH=1 After saving tor settings, if TOR_SKIP_LAUNCH=1 or if extensions.torlauncher.start_tor is false, avoid opening the progress dialog to monitor bootstrap progress. This avoids displaying a progress window that will not make progress (and the situation was made worse by the fact that clicking Cancel set DisableNetwork=1). Also, leave the network settings dialog open if we fail to set DisableNetwork=0. --- src/chrome/content/network-settings.js | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/src/chrome/content/network-settings.js b/src/chrome/content/network-settings.js index e0d7d1c..5df4e1e 100644 --- a/src/chrome/content/network-settings.js +++ b/src/chrome/content/network-settings.js @@ -1209,11 +1209,22 @@ function useSettings() { var settings = {}; settings[kTorConfKeyDisableNetwork] = false; - setConfAndReportErrors(settings, null); + let didApply = setConfAndReportErrors(settings, null); + if (!didApply) + return; gProtocolSvc.TorSendCommand("SAVECONF"); gTorProcessService.TorClearBootstrapError(); + // If we are not responsible for starting tor we do not monitor bootstrap + // status, so just close this dialog and return rather than opening the + // progress dialog (which will make no progress). + if (!TorLauncherUtil.shouldStartAndOwnTor) + { + close(); + return; + } + gIsBootstrapComplete = gTorProcessService.TorIsBootstrapDone; if (!gIsBootstrapComplete) openProgressDialog();

1 0

[collector/master] Reduce file-io by first writing all to temporary files and renaming
by karsten＠torproject.org 27 Oct '16

27 Oct '16

commit 1df211186efe6b9ae21121644890c616b8601b2a Author: iwakeh <iwakeh(a)torproject.org> Date: Wed Oct 26 21:56:42 2016 +0200 Reduce file-io by first writing all to temporary files and renaming these temporary files after sync. --- .../collector/persist/PersistenceUtils.java | 68 +++++++++++++++------- .../torproject/collector/sync/SyncPersistence.java | 7 +++ .../collector/persist/PersistUtilsTest.java | 7 ++- 3 files changed, 61 insertions(+), 21 deletions(-) diff --git a/src/main/java/org/torproject/collector/persist/PersistenceUtils.java b/src/main/java/org/torproject/collector/persist/PersistenceUtils.java index 8133451..d3678f2 100644 --- a/src/main/java/org/torproject/collector/persist/PersistenceUtils.java +++ b/src/main/java/org/torproject/collector/persist/PersistenceUtils.java @@ -9,10 +9,14 @@ import org.slf4j.LoggerFactory; import java.io.File; import java.io.IOException; import java.nio.file.FileAlreadyExistsException; +import java.nio.file.FileVisitResult; import java.nio.file.Files; import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.SimpleFileVisitor; import java.nio.file.StandardCopyOption; import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.BasicFileAttributes; import java.text.SimpleDateFormat; import java.util.Date; import java.util.TimeZone; @@ -22,6 +26,8 @@ public class PersistenceUtils { private static final Logger log = LoggerFactory.getLogger( PersistenceUtils.class); + public static final String TEMPFIX = ".tmp"; + /** Stores a descriptor adding missing annotations with the given options. */ public static boolean storeToFileSystem(byte[] typeAnnotation, byte[] data, Path outputPath, StandardOpenOption option) { @@ -29,34 +35,25 @@ public class PersistenceUtils { } /** Stores a descriptor adding missing annotations with the given options. - * Uses a temporary file and the copies to the final location. */ - public static boolean storeToFileSystem(byte[] typeAnnotation, byte[] data, - Path outputPath, StandardOpenOption option, boolean useTmp) { + * Uses a temporary file and requires a run of cleanDirectory for moving + * files to the final location. */ + public static boolean storeToFileSystem(byte[] typeAnnotation, + byte[] data, Path outputPath, StandardOpenOption option, boolean useTmp) { Path tmpPath = outputPath; try { if (useTmp) { tmpPath = new File(outputPath.toFile().getParent(), - outputPath.toFile().getName() + ".tmp").toPath(); - if (Files.exists(outputPath) && StandardOpenOption.APPEND == option) { - Files.copy(outputPath, tmpPath, StandardCopyOption.REPLACE_EXISTING); - } else if (Files.exists(outputPath) + outputPath.toFile().getName() + TEMPFIX).toPath(); + if (Files.exists(outputPath) && StandardOpenOption.CREATE_NEW == option) { return false; } + if (Files.exists(outputPath) && !Files.exists(tmpPath) + && StandardOpenOption.APPEND == option) { + Files.copy(outputPath, tmpPath, StandardCopyOption.REPLACE_EXISTING); + } } - StandardOpenOption appendOption = option; - Files.createDirectories(tmpPath.getParent()); - if (data.length > 0 && data[0] != '@') { - Files.write(tmpPath, typeAnnotation, appendOption, - StandardOpenOption.CREATE); - appendOption = StandardOpenOption.APPEND; - } - Files.write(tmpPath, data, appendOption, StandardOpenOption.CREATE); - if (useTmp) { - Files.deleteIfExists(outputPath); - tmpPath.toFile().renameTo(outputPath.toFile()); - } - return true; + return createOrAppend(typeAnnotation, data, tmpPath, option); } catch (FileAlreadyExistsException faee) { log.debug("Already have descriptor(s) of type '{}': {}. Skipping.", new String(typeAnnotation), outputPath); @@ -71,6 +68,37 @@ public class PersistenceUtils { return false; } + private static boolean createOrAppend(byte[] annotation, byte[] data, + Path path, StandardOpenOption option) throws IOException { + StandardOpenOption appendOption = option; + Files.createDirectories(path.getParent()); + if (data.length > 0 && data[0] != '@') { + Files.write(path, annotation, appendOption, StandardOpenOption.CREATE); + appendOption = StandardOpenOption.APPEND; + } + Files.write(path, data, appendOption, StandardOpenOption.CREATE); + return true; + } + + /** Move temporary files to their final location. */ + public static void cleanDirectory(Path pathToClean) throws IOException { + SimpleFileVisitor<Path> sfv = new SimpleFileVisitor<Path>() { + @Override + public FileVisitResult visitFile(Path file, BasicFileAttributes attrs) + throws IOException { + String tempName = file.toString(); + if (tempName.endsWith(TEMPFIX)) { + Path outputPath = Paths + .get(tempName.substring(0, tempName.length() - TEMPFIX.length())); + Files.deleteIfExists(outputPath); + file.toFile().renameTo(outputPath.toFile()); + } + return FileVisitResult.CONTINUE; + } + }; + Files.walkFileTree(pathToClean, sfv); + } + /** Return all date-time parts as array. */ public static String[] dateTimeParts(long dateTime) { return dateTimeParts(new Date(dateTime)); diff --git a/src/main/java/org/torproject/collector/sync/SyncPersistence.java b/src/main/java/org/torproject/collector/sync/SyncPersistence.java index 535334a..5c67a75 100644 --- a/src/main/java/org/torproject/collector/sync/SyncPersistence.java +++ b/src/main/java/org/torproject/collector/sync/SyncPersistence.java @@ -13,6 +13,7 @@ import org.torproject.collector.persist.DescriptorPersistence; import org.torproject.collector.persist.ExitlistPersistence; import org.torproject.collector.persist.ExtraInfoPersistence; import org.torproject.collector.persist.MicroConsensusPersistence; +import org.torproject.collector.persist.PersistenceUtils; import org.torproject.collector.persist.ServerDescriptorPersistence; import org.torproject.collector.persist.StatusPersistence; import org.torproject.collector.persist.VotePersistence; @@ -29,6 +30,7 @@ import org.torproject.descriptor.RelayServerDescriptor; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.IOException; import java.nio.file.Path; import java.util.List; @@ -116,6 +118,11 @@ public class SyncPersistence { desc.getClass().getSimpleName(), desc.getClass().getInterfaces()); } } + try { + PersistenceUtils.cleanDirectory(recentPath); + } catch (IOException ioe) { + log.error("Cleaning of {} failed.", recentPath.toString(), ioe); + } } } diff --git a/src/test/java/org/torproject/collector/persist/PersistUtilsTest.java b/src/test/java/org/torproject/collector/persist/PersistUtilsTest.java index 50ace87..32fa76a 100644 --- a/src/test/java/org/torproject/collector/persist/PersistUtilsTest.java +++ b/src/test/java/org/torproject/collector/persist/PersistUtilsTest.java @@ -45,10 +45,15 @@ public class PersistUtilsTest { public void testCreate() throws Exception { Path out = tmpf.newFolder().toPath(); Path pathToCreate = Paths.get(out.toString(), "very-new-file"); + Path pathToCreateTmp = Paths + .get(out.toString(), "very-new-file" + PersistenceUtils.TEMPFIX); String theText = "some text"; assertTrue("Files should be created.", PersistenceUtils.storeToFileSystem(ANNO1.getBytes(), - (theText + "\n").getBytes(), pathToCreate, StandardOpenOption.CREATE)); + (theText + "\n").getBytes(), pathToCreate, StandardOpenOption.CREATE, + true)); + assertTrue("File wasn't created.", Files.exists(pathToCreateTmp)); + PersistenceUtils.cleanDirectory(out); List<String> text = Files.readAllLines(pathToCreate, StandardCharsets.UTF_8); assertEquals("File contained: " + text, 2, text.size());

1 0

[tor/master] Add consensus weight calculation tests
by nickm＠torproject.org 26 Oct '16

26 Oct '16

commit 7ba0ae942638794c4c1e80e0ff3fb9eca514839c Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Mon Oct 3 21:09:25 2016 -0400 Add consensus weight calculation tests --- changes/14881 | 3 + src/or/dirvote.c | 2 +- src/or/dirvote.h | 4 ++ src/test/test_dir.c | 158 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 166 insertions(+), 1 deletion(-) diff --git a/changes/14881 b/changes/14881 index a8ca9c9..76ea009 100644 --- a/changes/14881 +++ b/changes/14881 @@ -2,3 +2,6 @@ - Add new consensus method that initializes bw weights to 1 instead of 0. This prevents a zero weight from making it all the way to the end (happens in small testing networks) and causing an error. Fixes bug 14881. + + o Minor features (unit tests): + - Add test for networkstatus_compute_bw_weights_v10 diff --git a/src/or/dirvote.c b/src/or/dirvote.c index 4a955e3..3d23119 100644 --- a/src/or/dirvote.c +++ b/src/or/dirvote.c @@ -913,7 +913,7 @@ networkstatus_check_weights(int64_t Wgg, int64_t Wgd, int64_t Wmg, * * It returns true if weights could be computed, false otherwise. */ -static int +int networkstatus_compute_bw_weights_v10(smartlist_t *chunks, int64_t G, int64_t M, int64_t E, int64_t D, int64_t T, int64_t weight_scale) diff --git a/src/or/dirvote.h b/src/or/dirvote.h index 3347346..ac7db69 100644 --- a/src/or/dirvote.h +++ b/src/or/dirvote.h @@ -238,6 +238,10 @@ STATIC smartlist_t *dirvote_compute_params(smartlist_t *votes, int method, int total_authorities); STATIC char *compute_consensus_package_lines(smartlist_t *votes); STATIC char *make_consensus_method_list(int low, int high, const char *sep); +STATIC int +networkstatus_compute_bw_weights_v10(smartlist_t *chunks, int64_t G, + int64_t M, int64_t E, int64_t D, + int64_t T, int64_t weight_scale); #endif #endif diff --git a/src/test/test_dir.c b/src/test/test_dir.c index 419aa24..08bceae 100644 --- a/src/test/test_dir.c +++ b/src/test/test_dir.c @@ -1874,6 +1874,163 @@ test_routerstatus_for_v3ns(routerstatus_t *rs, time_t now) return; } +static void +test_dir_networkstatus_compute_bw_weights_v10(void *arg) +{ + (void) arg; + smartlist_t *chunks = smartlist_new(); + int64_t G, M, E, D, T, weight_scale; + int ret; + weight_scale = 10000; + + /* no case. one or more of the values is 0 */ + G = M = E = D = 0; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 0); + tt_int_op(smartlist_len(chunks), OP_EQ, 0); + + /* case 1 */ + /* XXX dir-spec not followed? See #20272. If it isn't closed, then this is + * testing current behavior, not spec. */ + G = E = 10; + M = D = 1; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_int_op(smartlist_len(chunks), OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=3333 " + "Wbe=3000 Wbg=3000 Wbm=10000 Wdb=10000 Web=10000 Wed=3333 Wee=7000 " + "Weg=3333 Wem=7000 Wgb=10000 Wgd=3333 Wgg=7000 Wgm=7000 Wmb=10000 Wmd=3333 " + "Wme=3000 Wmg=3000 Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 2a E scarce */ + M = 100; + G = 20; + E = D = 5; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " + "Wem=10000 Wgb=10000 Wgd=0 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=0 Wmg=0 " + "Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 2a G scarce */ + M = 100; + E = 20; + G = D = 5; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=10000 Weg=0 Wem=10000 " + "Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=0 Wmg=0 " + "Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 2b1 (Wgg=1, Wmd=Wgd) */ + M = 10; + E = 30; + G = 10; + D = 100; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=4000 Wbe=0 " + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=2000 Wee=10000 Weg=2000 Wem=10000 " + "Wgb=10000 Wgd=4000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=4000 Wme=0 Wmg=0 " + "Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 2b2 */ + M = 60; + E = 30; + G = 10; + D = 100; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=666 Wbe=0 " + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=3666 Wee=10000 Weg=3666 Wem=10000 " + "Wgb=10000 Wgd=5668 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=666 Wme=0 Wmg=0 " + "Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 2b3 */ + /* XXX I can't get a combination of values that hits this case without error, so + * this just tests that it fails. See #20285 */ + /* (E < T/3 && G < T/3) && (E+D>=G || G+D>=E) && (M > T/3) */ + M = 80; + E = 30; + G = 30; + D = 30; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 0); + smartlist_clear(chunks); + + /* case 3a G scarce */ + M = 10; + E = 30; + G = 10; + D = 5; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=3333 " + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=6667 Weg=0 Wem=6667 " + "Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=3333 Wmg=0 " + "Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 3a E scarce */ + M = 10; + E = 10; + G = 30; + D = 5; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " + "Wbg=3333 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " + "Wem=10000 Wgb=10000 Wgd=0 Wgg=6667 Wgm=6667 Wmb=10000 Wmd=0 Wme=0 " + "Wmg=3333 Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 3bg */ + M = 10; + E = 30; + G = 10; + D = 10; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=3334 " + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=6666 Weg=0 Wem=6666 " + "Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=3334 Wmg=0 " + "Wmm=10000\n"); + smartlist_clear(chunks); + + /* case 3be */ + M = 10; + E = 10; + G = 30; + D = 10; + T = G + M + E + D; + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + tt_int_op(ret, OP_EQ, 1); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " + "Wbg=3334 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " + "Wem=10000 Wgb=10000 Wgd=0 Wgg=6666 Wgm=6666 Wmb=10000 Wmd=0 Wme=0 " + "Wmg=3334 Wmm=10000\n"); + smartlist_clear(chunks); + done: + return; +} + static authority_cert_t *mock_cert; static authority_cert_t * @@ -5555,6 +5712,7 @@ struct testcase_t dir_tests[] = { DIR_ARG(find_dl_schedule, TT_FORK, "cf"), DIR_ARG(find_dl_schedule, TT_FORK, "ca"), DIR(assumed_flags, 0), + DIR(networkstatus_compute_bw_weights_v10, 0), END_OF_TESTCASES };

1 0

[tor/master] Add historic bwweight tests, comments, line len fixes
by nickm＠torproject.org 26 Oct '16

26 Oct '16

commit 6629c5c3fe16abc516660d53e50628a019047e14 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Sun Oct 9 15:41:08 2016 -0400 Add historic bwweight tests, comments, line len fixes --- changes/14881 | 2 +- src/test/test_dir.c | 136 +++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 105 insertions(+), 33 deletions(-) diff --git a/changes/14881 b/changes/14881 index 76ea009..e686c4d 100644 --- a/changes/14881 +++ b/changes/14881 @@ -4,4 +4,4 @@ small testing networks) and causing an error. Fixes bug 14881. o Minor features (unit tests): - - Add test for networkstatus_compute_bw_weights_v10 + - Add tests for networkstatus_compute_bw_weights_v10 diff --git a/src/test/test_dir.c b/src/test/test_dir.c index 08bceae..788ec61 100644 --- a/src/test/test_dir.c +++ b/src/test/test_dir.c @@ -1886,7 +1886,8 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) /* no case. one or more of the values is 0 */ G = M = E = D = 0; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 0); tt_int_op(smartlist_len(chunks), OP_EQ, 0); @@ -1896,13 +1897,14 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = E = 10; M = D = 1; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); tt_int_op(smartlist_len(chunks), OP_EQ, 1); tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=3333 " "Wbe=3000 Wbg=3000 Wbm=10000 Wdb=10000 Web=10000 Wed=3333 Wee=7000 " - "Weg=3333 Wem=7000 Wgb=10000 Wgd=3333 Wgg=7000 Wgm=7000 Wmb=10000 Wmd=3333 " - "Wme=3000 Wmg=3000 Wmm=10000\n"); + "Weg=3333 Wem=7000 Wgb=10000 Wgd=3333 Wgg=7000 Wgm=7000 Wmb=10000 " + "Wmd=3333 Wme=3000 Wmg=3000 Wmm=10000\n"); smartlist_clear(chunks); /* case 2a E scarce */ @@ -1910,12 +1912,13 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 20; E = D = 5; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " - "Wem=10000 Wgb=10000 Wgd=0 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=0 Wmg=0 " - "Wmm=10000\n"); + "Wem=10000 Wgb=10000 Wgd=0 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=0 " + "Wmg=0 Wmm=10000\n"); smartlist_clear(chunks); /* case 2a G scarce */ @@ -1923,7 +1926,8 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) E = 20; G = D = 5; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=10000 Weg=0 Wem=10000 " @@ -1937,12 +1941,13 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 10; D = 100; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); - tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=4000 Wbe=0 " - "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=2000 Wee=10000 Weg=2000 Wem=10000 " - "Wgb=10000 Wgd=4000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=4000 Wme=0 Wmg=0 " - "Wmm=10000\n"); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=4000 " + "Wbe=0 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=2000 Wee=10000 Weg=2000 " + "Wem=10000 Wgb=10000 Wgd=4000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=4000 " + "Wme=0 Wmg=0 Wmm=10000\n"); smartlist_clear(chunks); /* case 2b2 */ @@ -1951,24 +1956,27 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 10; D = 100; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=666 Wbe=0 " - "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=3666 Wee=10000 Weg=3666 Wem=10000 " - "Wgb=10000 Wgd=5668 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=666 Wme=0 Wmg=0 " - "Wmm=10000\n"); + "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=3666 Wee=10000 Weg=3666 " + "Wem=10000 Wgb=10000 Wgd=5668 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=666 " + "Wme=0 Wmg=0 Wmm=10000\n"); smartlist_clear(chunks); /* case 2b3 */ - /* XXX I can't get a combination of values that hits this case without error, so - * this just tests that it fails. See #20285 */ + /* XXX I can't get a combination of values that hits this case without error, + * so this just tests that it fails. See #20285. Also see #20284 as 2b3 does + * not follow dir-spec. */ /* (E < T/3 && G < T/3) && (E+D>=G || G+D>=E) && (M > T/3) */ M = 80; E = 30; G = 30; D = 30; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 0); smartlist_clear(chunks); @@ -1978,12 +1986,13 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 10; D = 5; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); - tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=3333 " - "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=6667 Weg=0 Wem=6667 " - "Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=3333 Wmg=0 " - "Wmm=10000\n"); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 " + "Wbe=3333 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=6667 Weg=0 " + "Wem=6667 Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 " + "Wme=3333 Wmg=0 Wmm=10000\n"); smartlist_clear(chunks); /* case 3a E scarce */ @@ -1992,7 +2001,8 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 30; D = 5; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " "Wbg=3333 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " @@ -2006,12 +2016,13 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 10; D = 10; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); - tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=3334 " - "Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=6666 Weg=0 Wem=6666 " - "Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 Wme=3334 Wmg=0 " - "Wmm=10000\n"); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 " + "Wbe=3334 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=0 Wee=6666 Weg=0 " + "Wem=6666 Wgb=10000 Wgd=10000 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=0 " + "Wme=3334 Wmg=0 Wmm=10000\n"); smartlist_clear(chunks); /* case 3be */ @@ -2020,13 +2031,74 @@ test_dir_networkstatus_compute_bw_weights_v10(void *arg) G = 30; D = 10; T = G + M + E + D; - ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, weight_scale); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); tt_int_op(ret, OP_EQ, 1); tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " "Wbg=3334 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " "Wem=10000 Wgb=10000 Wgd=0 Wgg=6666 Wgm=6666 Wmb=10000 Wmd=0 Wme=0 " "Wmg=3334 Wmm=10000\n"); smartlist_clear(chunks); + + /* case from 21 Jul 2013 (3be) */ + G = 5483409; + M = 1455379; + E = 980834; + D = 3385803; + T = 11305425; + tt_int_op(G+M+E+D, OP_EQ, T); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=883 Wbe=0 " + "Wbg=3673 Wbm=10000 Wdb=10000 Web=10000 Wed=8233 Wee=10000 Weg=8233 " + "Wem=10000 Wgb=10000 Wgd=883 Wgg=6327 Wgm=6327 Wmb=10000 Wmd=883 Wme=0 " + "Wmg=3673 Wmm=10000\n"); + smartlist_clear(chunks); + + /* case from 04 Oct 2016 (3a E scarce) */ + G=29322240; + M=4721546; + E=1522058; + D=9273571; + T=44839415; + tt_int_op(G+M+E+D, OP_EQ, T); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=0 Wbe=0 " + "Wbg=4194 Wbm=10000 Wdb=10000 Web=10000 Wed=10000 Wee=10000 Weg=10000 " + "Wem=10000 Wgb=10000 Wgd=0 Wgg=5806 Wgm=5806 Wmb=10000 Wmd=0 Wme=0 " + "Wmg=4194 Wmm=10000\n"); + smartlist_clear(chunks); + + /* case from 04 Sep 2013 (2b1) */ + G=3091352; + M=1838837; + E=2109300; + D=2469369; + T=9508858; + tt_int_op(G+M+E+D, OP_EQ, T); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=317 " + "Wbe=5938 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=9366 Wee=4061 " + "Weg=9366 Wem=4061 Wgb=10000 Wgd=317 Wgg=10000 Wgm=10000 Wmb=10000 " + "Wmd=317 Wme=5938 Wmg=0 Wmm=10000\n"); + smartlist_clear(chunks); + + /* explicitly test initializing weights to 1*/ + G=1; + M=1; + E=1; + D=1; + T=4; + tt_int_op(G+M+E+D, OP_EQ, T); + ret = networkstatus_compute_bw_weights_v10(chunks, G, M, E, D, T, + weight_scale); + tt_str_op(smartlist_get(chunks, 0), OP_EQ, "bandwidth-weights Wbd=3333 " + "Wbe=0 Wbg=0 Wbm=10000 Wdb=10000 Web=10000 Wed=3333 Wee=10000 Weg=3333 " + "Wem=10000 Wgb=10000 Wgd=3333 Wgg=10000 Wgm=10000 Wmb=10000 Wmd=3333 " + "Wme=0 Wmg=0 Wmm=10000\n"); + smartlist_clear(chunks); done: return; }

1 0

[tor/master] Small refactoring: explain G, M, E, D, T initialization
by nickm＠torproject.org 26 Oct '16

26 Oct '16

commit 77d14b5f56df69776a732612d6631b558c071fb2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 26 16:39:59 2016 -0400 Small refactoring: explain G,M,E,D,T initialization --- src/or/dirvote.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/or/dirvote.c b/src/or/dirvote.c index 3d23119..d44ed7f 100644 --- a/src/or/dirvote.c +++ b/src/or/dirvote.c @@ -1335,7 +1335,7 @@ networkstatus_compute_consensus(smartlist_t *votes, smartlist_t *flags; const char *flavor_name; uint32_t max_unmeasured_bw_kb = DEFAULT_MAX_UNMEASURED_BW_KB; - int64_t G=0, M=0, E=0, D=0, T=0; /* For bandwidth weights */ + int64_t G, M, E, D, T; /* For bandwidth weights */ const routerstatus_format_type_t rs_format = flavor == FLAV_NS ? NS_V3_CONSENSUS : NS_V3_CONSENSUS_MICRODESC; char *params = NULL; @@ -1368,8 +1368,13 @@ networkstatus_compute_consensus(smartlist_t *votes, } if (consensus_method >= MIN_METHOD_FOR_INIT_BW_WEIGHTS_ONE) { + /* It's smarter to initialize these weights to 1, so that later on, + * we can't accidentally divide by zero. */ G = M = E = D = 1; T = 4; + } else { + /* ...but originally, they were set to zero. */ + G = M = E = D = T = 0; } /* Compute medians of time-related things, and figure out how many

1 0

[tor/master] Fix default bw weights with new consensus method
by nickm＠torproject.org 26 Oct '16

26 Oct '16

commit 909ee0e55fa7f03b8b85bc220ba11d91cb495d06 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Tue Sep 6 12:09:13 2016 -0400 Fix default bw weights with new consensus method See #14881 --- changes/14881 | 4 ++++ src/or/dirvote.c | 5 +++++ src/or/dirvote.h | 6 +++++- 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/changes/14881 b/changes/14881 new file mode 100644 index 0000000..a8ca9c9 --- /dev/null +++ b/changes/14881 @@ -0,0 +1,4 @@ + o Minor bugfixes (consensus weight): + - Add new consensus method that initializes bw weights to 1 instead of 0. This + prevents a zero weight from making it all the way to the end (happens in + small testing networks) and causing an error. Fixes bug 14881. diff --git a/src/or/dirvote.c b/src/or/dirvote.c index fe66469..4a955e3 100644 --- a/src/or/dirvote.c +++ b/src/or/dirvote.c @@ -1367,6 +1367,11 @@ networkstatus_compute_consensus(smartlist_t *votes, consensus_method = MAX_SUPPORTED_CONSENSUS_METHOD; } + if (consensus_method >= MIN_METHOD_FOR_INIT_BW_WEIGHTS_ONE) { + G = M = E = D = 1; + T = 4; + } + /* Compute medians of time-related things, and figure out how many * routers we might need to talk about. */ { diff --git a/src/or/dirvote.h b/src/or/dirvote.h index efd233e..3347346 100644 --- a/src/or/dirvote.h +++ b/src/or/dirvote.h @@ -55,7 +55,7 @@ #define MIN_SUPPORTED_CONSENSUS_METHOD 13 /** The highest consensus method that we currently support. */ -#define MAX_SUPPORTED_CONSENSUS_METHOD 25 +#define MAX_SUPPORTED_CONSENSUS_METHOD 26 /** Lowest consensus method where microdesc consensuses omit any entry * with no microdesc. */ @@ -111,6 +111,10 @@ * entries. */ #define MIN_METHOD_FOR_RS_PROTOCOLS 25 +/** Lowest consensus method where authorities initialize bandwidth weights to 1 + * instead of 0. See #14881 */ +#define MIN_METHOD_FOR_INIT_BW_WEIGHTS_ONE 26 + /** Default bandwidth to clip unmeasured bandwidths to using method >= * MIN_METHOD_TO_CLIP_UNMEASURED_BW. (This is not a consensus method; do not * get confused with the above macros.) */

1 0

[tor/master] Merge branch 'ticket14881-v3'
by nickm＠torproject.org 26 Oct '16

26 Oct '16

commit 48d7b693022e911899f1798386a2f5e277512cb2 Merge: 39375db 77d14b5 Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 26 16:40:11 2016 -0400 Merge branch 'ticket14881-v3' changes/14881 | 7 ++ src/or/dirvote.c | 14 +++- src/or/dirvote.h | 10 ++- src/test/test_dir.c | 230 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 258 insertions(+), 3 deletions(-)

1 0

[tor/master] Merge remote-tracking branch 'pastly/ticket20459'
by nickm＠torproject.org 26 Oct '16

26 Oct '16

commit 39375db3dc668f11925a82e6d4cef2beed91d47b Merge: 5a1779b c09993f Author: Nick Mathewson <nickm(a)torproject.org> Date: Wed Oct 26 16:22:06 2016 -0400 Merge remote-tracking branch 'pastly/ticket20459' changes/bug20459 | 3 +++ src/or/circuitmux_ewma.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)

1 0