September 2017 - tor-commits - lists.torproject.org

[tor/master] sched: add more per-socket limit documentation; int fix
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 501c58187d7c03cdbaf428fd268163e4d859e6b9 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Fri Sep 15 10:22:11 2017 -0400 sched: add more per-socket limit documentation; int fix --- src/or/scheduler_kist.c | 55 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/src/or/scheduler_kist.c b/src/or/scheduler_kist.c index 2dc30306d..3a897f5bf 100644 --- a/src/or/scheduler_kist.c +++ b/src/or/scheduler_kist.c @@ -230,27 +230,62 @@ update_socket_info_impl, (socket_table_ent_t *ent)) ent->unacked = tcp.tcpi_unacked; ent->mss = tcp.tcpi_snd_mss; - /* TCP space is the number of bytes would could give to the kernel and it - * would be able to immediately push them to the network. */ + /* In order to reduce outbound kernel queuing delays and thus improve Tor's + * ability to prioritize circuits, KIST wants to set a socket write limit that + * is near the amount that the socket would be able to immediately send into + * the Internet. + * + * We first calculate how much the socket could send immediately (assuming + * completely full packets) according to the congestion window and the number + * of unacked packets. + * + * Then we add a little extra space in a controlled way. We do this so any + * when the kernel gets ACKs back for data currently sitting in the "TCP + * space", it will already have some more data to send immediately. It will + * not have to wait for the scheduler to run again. The amount of extra space + * is a factor of the current congestion window. With the suggested + * sock_buf_size_factor value of 1.0, we allow at most 2*cwnd bytes to sit in + * the kernel: 1 cwnd on the wire waiting for ACKs and 1 cwnd ready and + * waiting to be sent when those ACKs finally come. + * + * In the below diagram, we see some bytes in the TCP-space (denoted by '*') + * that have be sent onto the wire and are waiting for ACKs. We have a little + * more room in "TCP space" that we can fill with data that will be + * immediately sent. We also see the "extra space" KIST calculates. The sum + * of the empty "TCP space" and the "extra space" is the kist-imposed write + * limit for this socket. + * + * <----------------kernel-outbound-socket-queue----------------| + * <*********---------------------------------------------------| + * <----TCP-space-----|----extra-space-----| + * <------------------| + * ^ ((cwnd - unacked) * mss) bytes + * |--------------------| + * ^ ((cwnd * mss) * factor) bytes + */ + + /* Assuming all these values from the kernel are uint32_t still, they will + * always fit into a int64_t tcp_space variable. */ tcp_space = (ent->cwnd - ent->unacked) * ent->mss; if (tcp_space < 0) { tcp_space = 0; } - /* Imagine we have filled up tcp_space already for a socket and the - * scheduler isn't going to run again for a while. We should write a little - * extra to the kernel so it has some data to send between scheduling runs - * if it gets ACKs back so it doesn't sit idle. With the suggested - * sock_buf_size_factor of 1.0, a socket can have at most 2*cwnd data in the - * kernel: 1 cwnd on the wire waiting for ACKs and 1 cwnd ready and waiting - * to be sent when those ACKs come. */ + /* The clamp_double_to_int64 makes sure the first part fits into an int64_t. + * In fact, if sock_buf_size_factor is still forced to be >= 0 in config.c, + * then it will be positive for sure. Then we subtract a uint32_t. At worst + * we end up negative, but then we just set extra_space to 0 in the sanity + * check.*/ extra_space = clamp_double_to_int64((ent->cwnd * ent->mss) * sock_buf_size_factor) - ent->notsent; if (extra_space < 0) { extra_space = 0; } - ent->limit = tcp_space + extra_space; + + /* Finally we set the limit. Adding two positive int64_t together will always + * fit in an uint64_t. */ + ent->limit = (uint64_t)tcp_space + (uint64_t)extra_space; return; #else /* HAVE_KIST_SUPPORT */

1 0

[tor/master] wrap a wide comment
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit fea2d84ce3eeba1d5ef07ece390a7677401a48d7 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Sep 15 12:03:08 2017 -0400 wrap a wide comment --- src/or/scheduler_kist.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/or/scheduler_kist.c b/src/or/scheduler_kist.c index 913cb4dce..54755cc40 100644 --- a/src/or/scheduler_kist.c +++ b/src/or/scheduler_kist.c @@ -231,9 +231,9 @@ update_socket_info_impl, (socket_table_ent_t *ent)) ent->mss = tcp.tcpi_snd_mss; /* In order to reduce outbound kernel queuing delays and thus improve Tor's - * ability to prioritize circuits, KIST wants to set a socket write limit that - * is near the amount that the socket would be able to immediately send into - * the Internet. + * ability to prioritize circuits, KIST wants to set a socket write limit + * that is near the amount that the socket would be able to immediately send + * into the Internet. * * We first calculate how much the socket could send immediately (assuming * completely full packets) according to the congestion window and the number

1 0

[tor/master] doc: Man page entries for KIST
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 06500171434dca543e3daf74ce7033a0aef3d199 Author: David Goulet <dgoulet(a)torproject.org> Date: Fri Sep 15 11:58:04 2017 -0400 doc: Man page entries for KIST Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- doc/tor.1.txt | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/doc/tor.1.txt b/doc/tor.1.txt index aec176705..d40643380 100644 --- a/doc/tor.1.txt +++ b/doc/tor.1.txt @@ -782,6 +782,36 @@ GENERAL OPTIONS option has been set to 1, it cannot be set back to 0 without restarting Tor. (Default: 0) +[[Schedulers]] **Schedulers** **KIST**|**KISTLite**|**Vanilla**:: + Specify the scheduler type that tor should use to handle outbound data on + channels. This is an ordered list by priority which means that the first + value will be tried first and if unavailable, the second one is tried and + so on. It is possible to change thse values at runtime. + (Default: KIST,KISTLite,Vanilla) + + + The possible scheduler types are: ++ + KIST: Kernel Informed Socket Transport. Tor will use the kernel tcp + information stack per-socket to make an informed decision on if it should + send or not the data. (Only available on Linux) ++ + KISTLite: Same as KIST but without kernel support which means that tor + will use all the same mecanics as KIST but without the TCP information the + kernel can provide. ++ + Vanilla: The scheduler that tor has always used that is do as much as + possible or AMAP. + +[[KISTSchedRunInterval]] **KISTSchedRunInterval** __NUM__ **msec**:: + If KIST or KISTLite is used in Schedulers option, this control at which + interval the scheduler tick is. If the value is 0 msec, the value is taken + from the consensus if possible else it will fallback to the default 10 + msec. Maximum possible value is 100 msec. (Default: 0 msec) + +[[KISTSockBufSizeFactor]] **KISTSockBufSizeFactor** __NUM__:: + If KIST is used in Schedulers, this is a multiplier of the per-socket + limit calculation of the KIST algorithm. (Default: 1.0) + CLIENT OPTIONS --------------

1 0

[tor/master] sched: Make check-spaces happy
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit ba08f8a4ed2628bde685612f2dd4aa7c85bdac57 Author: David Goulet <dgoulet(a)torproject.org> Date: Thu Sep 14 14:29:52 2017 -0400 sched: Make check-spaces happy Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/or/scheduler_kist.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/src/or/scheduler_kist.c b/src/or/scheduler_kist.c index 2733e78f0..85d0b2a1f 100644 --- a/src/or/scheduler_kist.c +++ b/src/or/scheduler_kist.c @@ -245,13 +245,14 @@ update_socket_info_impl, (socket_table_ent_t *ent)) if (tcp_space < 0) { tcp_space = 0; } - /* Imagine we have filled up tcp_space already for a socket and the scheduler - * isn't going to run again for a while. We should write a little extra to the - * kernel so it has some data to send between scheduling runs if it gets ACKs - * back so it doesn't sit idle. With the suggested sock_buf_size_factor of - * 1.0, a socket can have at most 2*cwnd data in the kernel: 1 cwnd on the - * wire waiting for ACKs and 1 cwnd ready and waiting to be sent when those - * ACKs come. */ + + /* Imagine we have filled up tcp_space already for a socket and the + * scheduler isn't going to run again for a while. We should write a little + * extra to the kernel so it has some data to send between scheduling runs + * if it gets ACKs back so it doesn't sit idle. With the suggested + * sock_buf_size_factor of 1.0, a socket can have at most 2*cwnd data in the + * kernel: 1 cwnd on the wire waiting for ACKs and 1 cwnd ready and waiting + * to be sent when those ACKs come. */ extra_space = clamp_double_to_int64((ent->cwnd * ent->mss) * sock_buf_size_factor) - ent->notsent;

1 0

[tor/master] sched: Add Schedulers torrc option
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 7cc9621d115303ab71d8cf8e49f5dfe428636145 Author: David Goulet <dgoulet(a)torproject.org> Date: Thu Sep 14 13:23:43 2017 -0400 sched: Add Schedulers torrc option This option is a list of possible scheduler type tor can use ordered by priority. Its default value is "KIST,KISTLite,Vanilla" which means that KIST will be used first and if unavailable will fallback to KISTLite and so on. Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/or/config.c | 77 +++++++++++++++++++++++++++++++++++++++-------- src/or/or.h | 6 ++++ src/or/scheduler.c | 57 +++++++++++++++++++++++++++-------- src/or/scheduler.h | 14 +++++++-- src/or/scheduler_kist.c | 45 ++++++++++++++++++++++----- src/test/test_scheduler.c | 14 ++++----- 6 files changed, 172 insertions(+), 41 deletions(-) diff --git a/src/or/config.c b/src/or/config.c index f332ac97f..95f27b071 100644 --- a/src/or/config.c +++ b/src/or/config.c @@ -491,6 +491,7 @@ static config_var_t option_vars_[] = { OBSOLETE("SchedulerMaxFlushCells__"), V(KISTSchedRunInterval, MSEC_INTERVAL, "0 msec"), V(KISTSockBufSizeFactor, DOUBLE, "1.0"), + V(Schedulers, CSV, "KIST,KISTLite,Vanilla"), V(ShutdownWaitLength, INTERVAL, "30 seconds"), OBSOLETE("SocksListenAddress"), V(SocksPolicy, LINELIST, NULL), @@ -907,6 +908,10 @@ or_options_free(or_options_t *options) rs, routerset_free(rs)); smartlist_free(options->NodeFamilySets); } + if (options->SchedulerTypes_) { + SMARTLIST_FOREACH(options->SchedulerTypes_, int *, i, tor_free(i)); + smartlist_free(options->SchedulerTypes_); + } tor_free(options->BridgePassword_AuthDigest_); tor_free(options->command_arg); tor_free(options->master_key_fname); @@ -2888,6 +2893,61 @@ warn_about_relative_paths(or_options_t *options) } } +/* Validate options related to the scheduler. From the Schedulers list, the + * SchedulerTypes_ list is created with int values so once we select the + * scheduler, which can happen anytime at runtime, we don't have to parse + * strings and thus be quick. + * + * Return 0 on success else -1 and msg is set with an error message. */ +static int +options_validate_scheduler(or_options_t *options, char **msg) +{ + tor_assert(options); + tor_assert(msg); + + if (!options->Schedulers || smartlist_len(options->Schedulers) == 0) { + REJECT("Empty Schedulers list. Either remove the option so the defaults " + "can be used or set at least one value."); + } + /* Ok, we do have scheduler types, validate them. */ + options->SchedulerTypes_ = smartlist_new(); + SMARTLIST_FOREACH_BEGIN(options->Schedulers, const char *, type) { + int *sched_type; + if (!strcasecmp("KISTLite", type)) { + sched_type = tor_malloc_zero(sizeof(int)); + *sched_type = SCHEDULER_KIST_LITE; + smartlist_add(options->SchedulerTypes_, sched_type); + } else if (!strcasecmp("KIST", type)) { + sched_type = tor_malloc_zero(sizeof(int)); + *sched_type = SCHEDULER_KIST; + smartlist_add(options->SchedulerTypes_, sched_type); + } else if (!strcasecmp("Vanilla", type)) { + sched_type = tor_malloc_zero(sizeof(int)); + *sched_type = SCHEDULER_VANILLA; + smartlist_add(options->SchedulerTypes_, sched_type); + } else { + tor_asprintf(msg, "Unknown type %s in option Schedulers. " + "Possible values are KIST, KISTLite and Vanilla.", + escaped(type)); + return -1; + } + } SMARTLIST_FOREACH_END(type); + + if (options->KISTSockBufSizeFactor < 0) { + REJECT("KISTSockBufSizeFactor must be at least 0"); + } + + /* Don't need to validate that the Interval is less than anything because + * zero is valid and all negative values are valid. */ + if (options->KISTSchedRunInterval > KIST_SCHED_RUN_INTERVAL_MAX) { + tor_asprintf(msg, "KISTSchedRunInterval must not be more than %d (ms)", + KIST_SCHED_RUN_INTERVAL_MAX); + return -1; + } + + return 0; +} + /* Validate options related to single onion services. * Modifies some options that are incompatible with single onion services. * On failure returns -1, and sets *msg to an error string. @@ -3112,19 +3172,6 @@ options_validate(or_options_t *old_options, or_options_t *options, routerset_union(options->ExcludeExitNodesUnion_,options->ExcludeNodes); } - if (options->KISTSockBufSizeFactor < 0) { - REJECT("KISTSockBufSizeFactor must be at least 0"); - } - /* Don't need to validate that the Interval is less than anything because - * zero is valid and all negative values are valid. */ - if (options->KISTSchedRunInterval > KIST_SCHED_RUN_INTERVAL_MAX) { - char *buf = tor_calloc(80, sizeof(char)); - tor_snprintf(buf, 80, "KISTSchedRunInterval must not be more than %d (ms)", - KIST_SCHED_RUN_INTERVAL_MAX); - *msg = buf; - return -1; - } - if (options->NodeFamilies) { options->NodeFamilySets = smartlist_new(); for (cl = options->NodeFamilies; cl; cl = cl->next) { @@ -4241,6 +4288,10 @@ options_validate(or_options_t *old_options, or_options_t *options, REJECT("BridgeRelay is 1, ORPort is not set. This is an invalid " "combination."); + if (options_validate_scheduler(options, msg) < 0) { + return -1; + } + return 0; } diff --git a/src/or/or.h b/src/or/or.h index 7116fbac7..fbeaf70ab 100644 --- a/src/or/or.h +++ b/src/or/or.h @@ -4616,6 +4616,12 @@ typedef struct { /** A multiplier for the KIST per-socket limit calculation. */ double KISTSockBufSizeFactor; + + /** The list of scheduler type string ordered by priority that is first one + * has to be tried first. Default: KIST,KISTLite,Vanilla */ + smartlist_t *Schedulers; + /* An ordered list of scheduler_types mapped from Schedulers. */ + smartlist_t *SchedulerTypes_; } or_options_t; /** Persistent state for an onion router, as saved to disk. */ diff --git a/src/or/scheduler.c b/src/or/scheduler.c index 8e4cec095..aaf991407 100644 --- a/src/or/scheduler.c +++ b/src/or/scheduler.c @@ -274,6 +274,48 @@ scheduler_compare_channels, (const void *c1_v, const void *c2_v)) * Functions that can be accessed from anywhere in Tor. *****************************************************************************/ +/* Using the global options, select the scheduler we should be using. */ +static void +select_scheduler(void) +{ + const char *chosen_sched_type = NULL; + + /* This list is ordered that is first entry has the first priority. Thus, as + * soon as we find a scheduler type that we can use, we use it and stop. */ + SMARTLIST_FOREACH_BEGIN(get_options()->SchedulerTypes_, int *, type) { + switch (*type) { + case SCHEDULER_VANILLA: + the_scheduler = get_vanilla_scheduler(); + chosen_sched_type = "Vanilla"; + goto end; + case SCHEDULER_KIST: + if (!scheduler_can_use_kist()) { + log_warn(LD_SCHED, "Scheduler KIST can't be used. Consider removing " + "it from Schedulers or if you have a tor built " + "with KIST support, you should make sure " + "KISTSchedRunInterval is a non zero value"); + continue; + } + the_scheduler = get_kist_scheduler(); + chosen_sched_type = "KIST"; + scheduler_kist_set_full_mode(); + goto end; + case SCHEDULER_KIST_LITE: + chosen_sched_type = "KISTLite"; + the_scheduler = get_kist_scheduler(); + scheduler_kist_set_lite_mode(); + goto end; + default: + /* Our option validation should have caught this. */ + tor_assert_unreached(); + } + } SMARTLIST_FOREACH_END(type); + + end: + log_notice(LD_CONFIG, "Scheduler type %s has been enabled.", + chosen_sched_type); +} + /* * Little helper function called from a few different places. It changes the * scheduler implementation, if necessary. And if it did, it then tells the @@ -282,17 +324,10 @@ scheduler_compare_channels, (const void *c1_v, const void *c2_v)) static void set_scheduler(void) { - int have_kist = 0; - - /* Switch, if needed */ scheduler_t *old_scheduler = the_scheduler; - if (scheduler_should_use_kist()) { - the_scheduler = get_kist_scheduler(); - have_kist = 1; - } else { - the_scheduler = get_vanilla_scheduler(); - } - tor_assert(the_scheduler); + + /* From the options, select the scheduler type to set. */ + select_scheduler(); if (old_scheduler != the_scheduler) { /* Allow the old scheduler to clean up, if needed. */ @@ -306,8 +341,6 @@ set_scheduler(void) if (the_scheduler->init) { the_scheduler->init(); } - log_notice(LD_CONFIG, "Using the %s scheduler.", - have_kist ? "KIST" : "vanilla"); } } diff --git a/src/or/scheduler.h b/src/or/scheduler.h index 68de5cf66..b1863b119 100644 --- a/src/or/scheduler.h +++ b/src/or/scheduler.h @@ -82,6 +82,15 @@ typedef struct scheduler_s { void (*on_new_options)(void); } scheduler_t; +/** Scheduler type, we build an ordered list with those values from the + * parsed strings in Schedulers. The reason to do such a thing is so we can + * quickly and without parsing strings select the scheduler at anytime. */ +typedef enum { + SCHEDULER_VANILLA = 1, + SCHEDULER_KIST = 2, + SCHEDULER_KIST_LITE = 3, +} scheduler_types_t; + /***************************************************************************** * Globally visible scheduler variables/values * @@ -95,7 +104,6 @@ typedef struct scheduler_s { /* Maximum interval that KIST runs (in ms). */ #define KIST_SCHED_RUN_INTERVAL_MAX 100 - /***************************************************************************** * Globally visible scheduler functions * @@ -169,7 +177,9 @@ MOCK_DECL(int, channel_should_write_to_kernel, MOCK_DECL(void, channel_write_to_kernel, (channel_t *chan)); MOCK_DECL(void, update_socket_info_impl, (socket_table_ent_t *ent)); -int scheduler_should_use_kist(void); +int scheduler_can_use_kist(void); +void scheduler_kist_set_full_mode(void); +void scheduler_kist_set_lite_mode(void); scheduler_t *get_kist_scheduler(void); int32_t kist_scheduler_run_interval(const networkstatus_t *ns); diff --git a/src/or/scheduler_kist.c b/src/or/scheduler_kist.c index aec8192ed..8646169ee 100644 --- a/src/or/scheduler_kist.c +++ b/src/or/scheduler_kist.c @@ -26,12 +26,18 @@ * Data structures and supporting functions *****************************************************************************/ +#ifdef HAVE_KIST_SUPPORT +/* Indicate if KIST lite mode is on or off. We can disable it at runtime. + * Important to have because of the KISTLite -> KIST possible transition. */ +static unsigned int kist_lite_mode = 0; /* Indicate if we don't have the kernel support. This can happen if the kernel * changed and it doesn't recognized the values passed to the syscalls needed * by KIST. In that case, fallback to the naive approach. */ -#ifdef HAVE_KIST_SUPPORT static unsigned int kist_no_kernel_support = 0; -#endif /* HAVE_KIST_SUPPORT */ +#else +static unsigned int kist_no_kernel_support = 1; +static unsigned int kist_lite_mode = 1; +#endif /* Socket_table hash table stuff. The socket_table keeps track of per-socket * limit information imposed by kist and used by kist. */ @@ -193,7 +199,7 @@ update_socket_info_impl, (socket_table_ent_t *ent)) struct tcp_info tcp; socklen_t tcp_info_len = sizeof(tcp); - if (kist_no_kernel_support) { + if (kist_no_kernel_support || kist_lite_mode) { goto fallback; } @@ -686,15 +692,40 @@ kist_scheduler_run_interval(const networkstatus_t *ns) return run_interval; } +/* Set KISTLite mode that is KIST without kernel support. */ +void +scheduler_kist_set_lite_mode(void) +{ + kist_lite_mode = 1; + log_info(LD_SCHED, + "Setting KIST scheduler without kernel support (KISTLite mode)"); +} + +/* Set KIST mode that is KIST with kernel support. */ +void +scheduler_kist_set_full_mode(void) +{ + kist_lite_mode = 0; + log_info(LD_SCHED, + "Setting KIST scheduler with kernel support (KIST mode)"); +} + #ifdef HAVE_KIST_SUPPORT /* Return true iff the scheduler subsystem should use KIST. */ int -scheduler_should_use_kist(void) +scheduler_can_use_kist(void) { + if (kist_no_kernel_support) { + /* We have no kernel support so we can't use KIST. */ + return 0; + } + + /* We do have the support, time to check if we can get the interval that the + * consensus can be disabling. */ int64_t run_interval = kist_scheduler_run_interval(NULL); - log_info(LD_SCHED, "Determined sched_run_interval should be %" PRId64 ". " - "Will%s use KIST.", + log_debug(LD_SCHED, "Determined KIST sched_run_interval should be " + "%" PRId64 ". Can%s use KIST.", run_interval, (run_interval > 0 ? "" : " not")); return run_interval > 0; } @@ -702,7 +733,7 @@ scheduler_should_use_kist(void) #else /* HAVE_KIST_SUPPORT */ int -scheduler_should_use_kist(void) +scheduler_can_use_kist(void) { return 0; } diff --git a/src/test/test_scheduler.c b/src/test/test_scheduler.c index 7c24bf146..0cc6d88db 100644 --- a/src/test/test_scheduler.c +++ b/src/test/test_scheduler.c @@ -887,7 +887,7 @@ test_scheduler_initfree(void *arg) } static void -test_scheduler_should_use_kist(void *arg) +test_scheduler_can_use_kist(void *arg) { (void)arg; @@ -897,7 +897,7 @@ test_scheduler_should_use_kist(void *arg) /* Test force disabling of KIST */ clear_options(); mocked_options.KISTSchedRunInterval = -1; - res_should = scheduler_should_use_kist(); + res_should = scheduler_can_use_kist(); res_freq = kist_scheduler_run_interval(NULL); tt_int_op(res_should, ==, 0); tt_int_op(res_freq, ==, -1); @@ -905,7 +905,7 @@ test_scheduler_should_use_kist(void *arg) /* Test force enabling of KIST */ clear_options(); mocked_options.KISTSchedRunInterval = 1234; - res_should = scheduler_should_use_kist(); + res_should = scheduler_can_use_kist(); res_freq = kist_scheduler_run_interval(NULL); #ifdef HAVE_KIST_SUPPORT tt_int_op(res_should, ==, 1); @@ -917,7 +917,7 @@ test_scheduler_should_use_kist(void *arg) /* Test defer to consensus, but no consensus available */ clear_options(); mocked_options.KISTSchedRunInterval = 0; - res_should = scheduler_should_use_kist(); + res_should = scheduler_can_use_kist(); res_freq = kist_scheduler_run_interval(NULL); #ifdef HAVE_KIST_SUPPORT tt_int_op(res_should, ==, 1); @@ -930,7 +930,7 @@ test_scheduler_should_use_kist(void *arg) MOCK(networkstatus_get_param, mock_kist_networkstatus_get_param); clear_options(); mocked_options.KISTSchedRunInterval = 0; - res_should = scheduler_should_use_kist(); + res_should = scheduler_can_use_kist(); res_freq = kist_scheduler_run_interval(NULL); #ifdef HAVE_KIST_SUPPORT tt_int_op(res_should, ==, 1); @@ -944,7 +944,7 @@ test_scheduler_should_use_kist(void *arg) MOCK(networkstatus_get_param, mock_vanilla_networkstatus_get_param); clear_options(); mocked_options.KISTSchedRunInterval = 0; - res_should = scheduler_should_use_kist(); + res_should = scheduler_can_use_kist(); res_freq = kist_scheduler_run_interval(NULL); tt_int_op(res_should, ==, 0); tt_int_op(res_freq, ==, -1); @@ -1023,7 +1023,7 @@ struct testcase_t scheduler_tests[] = { { "loop_vanilla", test_scheduler_loop_vanilla, TT_FORK, NULL, NULL }, { "loop_kist", test_scheduler_loop_kist, TT_FORK, NULL, NULL }, { "ns_changed", test_scheduler_ns_changed, TT_FORK, NULL, NULL}, - { "should_use_kist", test_scheduler_should_use_kist, TT_FORK, NULL, NULL }, + { "should_use_kist", test_scheduler_can_use_kist, TT_FORK, NULL, NULL }, END_OF_TESTCASES };

1 0

[tor/master] sched: add changes file
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 63204be48faccc2e8a7422944700cf899faa1f89 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Fri Sep 15 10:22:31 2017 -0400 sched: add changes file --- changes/ticket12541 | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/changes/ticket12541 b/changes/ticket12541 new file mode 100644 index 000000000..9543eceec --- /dev/null +++ b/changes/ticket12541 @@ -0,0 +1,23 @@ + o Major feature (scheduler, channel): + - Introducing the KIST scheduler which stands for Kernel Informed Socket + Transport. It is only available on Linux systems. This comes from a + researched and published paper you can find here: + + http://www.robgjansen.com/publications/kist-sec2014.pdf + https://arxiv.org/abs/1709.01044 + + This is also a major refactoring of the entire scheduler subsystem in + order for it to be more modular and thus much more easier to add more + scheduler type later. The current scheduler has been named "Vanilla" but + we favor KIST if available in this version. + + A new torrc option has been added and named "Schedulers type1,type2,..." + which allows a user to select which scheduler type it wants tor to use. + It is also possible to change it at runtime. It is an ordered list by + priority. KIST might not be available on all platforms so there is a + fallback to "KISTLite" that uses the same mechanisms but without the + kernel support. + + The current default values are: Schedulers KIST,KISTLite,Vanilla. + + Closes ticket 12541.

1 0

[tor/master] fixup! sched: add changes file (spaces)
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 47390d1c9e16cd5973e10896e46c2ef13fc617fb Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Fri Sep 15 10:28:01 2017 -0400 fixup! sched: add changes file (spaces) --- changes/ticket12541 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/changes/ticket12541 b/changes/ticket12541 index 9543eceec..db6d2ad50 100644 --- a/changes/ticket12541 +++ b/changes/ticket12541 @@ -5,7 +5,7 @@ http://www.robgjansen.com/publications/kist-sec2014.pdf https://arxiv.org/abs/1709.01044 - + This is also a major refactoring of the entire scheduler subsystem in order for it to be more modular and thus much more easier to add more scheduler type later. The current scheduler has been named "Vanilla" but @@ -16,7 +16,7 @@ It is also possible to change it at runtime. It is an ordered list by priority. KIST might not be available on all platforms so there is a fallback to "KISTLite" that uses the same mechanisms but without the - kernel support. + kernel support. The current default values are: Schedulers KIST,KISTLite,Vanilla.

1 0

[tor/master] sched: add comment about how we determine if a socket should write
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit ae9e6b547df8efe3eab40237cb4646e158d4d8b0 Author: Matt Traudt <sirmatt(a)ksu.edu> Date: Fri Sep 15 10:28:52 2017 -0400 sched: add comment about how we determine if a socket should write --- src/or/scheduler_kist.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/or/scheduler_kist.c b/src/or/scheduler_kist.c index 3a897f5bf..913cb4dce 100644 --- a/src/or/scheduler_kist.c +++ b/src/or/scheduler_kist.c @@ -375,6 +375,10 @@ socket_can_write(socket_table_t *table, const channel_t *chan) return 1; // Just return true, saying that kist wouldn't limit the socket } + /* We previously caclulated a write limit for this socket. In the below + * calculation, first determine how much room is left in bytes. Then divide + * that by the amount of space a cell takes. If there's room for at least 1 + * cell, then KIST will allow the socket to write. */ int64_t kist_limit_space = (int64_t) (ent->limit - ent->written) / (CELL_MAX_NETWORK_SIZE + TLS_PER_CELL_OVERHEAD);

1 0

[tor/master] Add a timestamp field to the CIRC_BW and STREAM_BW events
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 2be4f793e63e3e998e95bc678a231f225ccad441 Author: Donncha O'Cearbhaill <donncha(a)donncha.is> Date: Fri Jun 3 10:45:38 2016 +0100 Add a timestamp field to the CIRC_BW and STREAM_BW events Closes ticket 19254. --- changes/feature19254 | 3 +++ src/or/control.c | 28 ++++++++++++++++++++++------ 2 files changed, 25 insertions(+), 6 deletions(-) diff --git a/changes/feature19254 b/changes/feature19254 new file mode 100644 index 000000000..598ecc88d --- /dev/null +++ b/changes/feature19254 @@ -0,0 +1,3 @@ + o Minor features (controller): + - Adds a timestamp field to the CIRC_BW and STREAM_BW bandwidth + events. Closes ticket 19254. Patch by "DonnchaC". diff --git a/src/or/control.c b/src/or/control.c index dc431d6d5..2d1c60295 100644 --- a/src/or/control.c +++ b/src/or/control.c @@ -5567,15 +5567,20 @@ control_event_stream_bandwidth(edge_connection_t *edge_conn) { circuit_t *circ; origin_circuit_t *ocirc; + struct timeval now; + char tbuf[ISO_TIME_USEC_LEN+1]; if (EVENT_IS_INTERESTING(EVENT_STREAM_BANDWIDTH_USED)) { if (!edge_conn->n_read && !edge_conn->n_written) return 0; + tor_gettimeofday(&now); + format_iso_time_nospace_usec(tbuf, &now); send_control_event(EVENT_STREAM_BANDWIDTH_USED, - "650 STREAM_BW "U64_FORMAT" %lu %lu\r\n", + "650 STREAM_BW "U64_FORMAT" %lu %lu %s\r\n", U64_PRINTF_ARG(edge_conn->base_.global_identifier), (unsigned long)edge_conn->n_read, - (unsigned long)edge_conn->n_written); + (unsigned long)edge_conn->n_written, + tbuf); circ = circuit_get_by_edge_conn(edge_conn); if (circ && CIRCUIT_IS_ORIGIN(circ)) { @@ -5597,6 +5602,8 @@ control_event_stream_bandwidth_used(void) if (EVENT_IS_INTERESTING(EVENT_STREAM_BANDWIDTH_USED)) { smartlist_t *conns = get_connection_array(); edge_connection_t *edge_conn; + struct timeval now; + char tbuf[ISO_TIME_USEC_LEN+1]; SMARTLIST_FOREACH_BEGIN(conns, connection_t *, conn) { @@ -5606,11 +5613,14 @@ control_event_stream_bandwidth_used(void) if (!edge_conn->n_read && !edge_conn->n_written) continue; + tor_gettimeofday(&now); + format_iso_time_nospace_usec(tbuf, &now); send_control_event(EVENT_STREAM_BANDWIDTH_USED, - "650 STREAM_BW "U64_FORMAT" %lu %lu\r\n", + "650 STREAM_BW "U64_FORMAT" %lu %lu %s\r\n", U64_PRINTF_ARG(edge_conn->base_.global_identifier), (unsigned long)edge_conn->n_read, - (unsigned long)edge_conn->n_written); + (unsigned long)edge_conn->n_written, + tbuf); edge_conn->n_written = edge_conn->n_read = 0; } @@ -5626,6 +5636,8 @@ int control_event_circ_bandwidth_used(void) { origin_circuit_t *ocirc; + struct timeval now; + char tbuf[ISO_TIME_USEC_LEN+1]; if (!EVENT_IS_INTERESTING(EVENT_CIRC_BANDWIDTH_USED)) return 0; @@ -5635,11 +5647,15 @@ control_event_circ_bandwidth_used(void) ocirc = TO_ORIGIN_CIRCUIT(circ); if (!ocirc->n_read_circ_bw && !ocirc->n_written_circ_bw) continue; + tor_gettimeofday(&now); + format_iso_time_nospace_usec(tbuf, &now); send_control_event(EVENT_CIRC_BANDWIDTH_USED, - "650 CIRC_BW ID=%d READ=%lu WRITTEN=%lu\r\n", + "650 CIRC_BW ID=%d READ=%lu WRITTEN=%lu " + "TIME=%s\r\n", ocirc->global_identifier, (unsigned long)ocirc->n_read_circ_bw, - (unsigned long)ocirc->n_written_circ_bw); + (unsigned long)ocirc->n_written_circ_bw, + tbuf); ocirc->n_written_circ_bw = ocirc->n_read_circ_bw = 0; } SMARTLIST_FOREACH_END(circ);

1 0

[tor/master] Merge remote-tracking branch 'dgoulet/ticket19254_032_01'
by nickm＠torproject.org 15 Sep '17

15 Sep '17

commit 962b0b849bf0c2fcca387e334985b349a255de0a Merge: e88fb4f4f 2be4f793e Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Sep 15 11:52:10 2017 -0400 Merge remote-tracking branch 'dgoulet/ticket19254_032_01' changes/feature19254 | 3 +++ src/or/control.c | 28 ++++++++++++++++++++++------ 2 files changed, 25 insertions(+), 6 deletions(-)

1 0