March 2020 - tor-commits - lists.torproject.org

[tor/maint-0.4.2] dirauth: Resume sending 503 directory error code
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit 6d9113d2f65b6e3142efdaa91a5b4761cd197be8 Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Jan 28 09:39:09 2020 -0500 dirauth: Resume sending 503 directory error code Authorities were never sending back 503 error code because by design they should be able to always answer directory requests regardless of bandwidth capacity. However, that recently backfired because of a large number of requests from unknown source using the DirPort that are _not_ getting their 503 code which overloaded the DirPort leading to the authority to be unable to answer to its fellow authorities. This is not a complete solution to the problem but it will help ease off the load on the authority side by sending back 503 codes *unless* the connection is from a known relay or an authority. Fixes #33029 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- changes/ticket33029 | 5 +++++ src/core/mainloop/connection.c | 17 +++++++++++++++-- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/changes/ticket33029 b/changes/ticket33029 new file mode 100644 index 000000000..c32ee4ad8 --- /dev/null +++ b/changes/ticket33029 @@ -0,0 +1,5 @@ + o Major bugfixes (directory authority): + - Directory authorities will now send a 503 (not enough bandwidth) code to + clients when under bandwidth pressure. Known relays and other authorities + will always be answered regardless of the bandwidth situation. Fixes bug + 33029; bugfix on 0.1.2.5-alpha. diff --git a/src/core/mainloop/connection.c b/src/core/mainloop/connection.c index a157c0f3f..50cd3810a 100644 --- a/src/core/mainloop/connection.c +++ b/src/core/mainloop/connection.c @@ -3211,8 +3211,21 @@ connection_dir_is_global_write_low(const connection_t *conn, size_t attempt) size_t smaller_bucket = MIN(token_bucket_rw_get_write(&global_bucket), token_bucket_rw_get_write(&global_relayed_bucket)); - if (authdir_mode(get_options())) - return false; /* there's always room to answer v2 if we're an auth dir */ + + /* Special case for authorities (directory only). */ + if (authdir_mode_v3(get_options())) { + /* Are we configured to possibly reject requests under load? */ + if (!get_options()->AuthDirRejectRequestsUnderLoad) { + /* Answer request no matter what. */ + return false; + } + /* Always answer requests from a known relay which includes the other + * authorities. The following looks up the addresses for relays that we + * have their descriptor _and_ any configured trusted directories. */ + if (nodelist_probably_contains_address(&conn->addr)) { + return false; + } + } if (!connection_is_rate_limited(conn)) return false; /* local conns don't get limited */

1 0

[tor/maint-0.4.2] Merge remote-tracking branch 'tor-github/pr/1718/head' into maint-0.4.2
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit b504942331dabe1beb1ddb24a3aea75a565573e2 Merge: f021389e9 7b4d9fabe Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Mar 13 16:42:47 2020 -0400 Merge remote-tracking branch 'tor-github/pr/1718/head' into maint-0.4.2 changes/ticket33029 | 5 + doc/tor.1.txt | 7 ++ src/app/config/config.c | 1 + src/app/config/or_options_st.h | 7 ++ src/core/mainloop/connection.c | 55 +++++------ src/core/mainloop/connection.h | 5 +- src/feature/dircache/dircache.c | 13 ++- src/feature/nodelist/dirlist.c | 31 ++++++ src/feature/nodelist/dirlist.h | 2 + src/feature/nodelist/nodelist.c | 43 +++++++-- src/feature/nodelist/nodelist.h | 2 + src/test/test_address_set.c | 15 ++- src/test/test_bwmgt.c | 209 +++++++++++++++++++++++++++++++++++++++- 13 files changed, 348 insertions(+), 47 deletions(-)

1 0

[tor/maint-0.4.3] Merge remote-tracking branch 'tor-github/pr/1718/head' into maint-0.4.2
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit b504942331dabe1beb1ddb24a3aea75a565573e2 Merge: f021389e9 7b4d9fabe Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Mar 13 16:42:47 2020 -0400 Merge remote-tracking branch 'tor-github/pr/1718/head' into maint-0.4.2 changes/ticket33029 | 5 + doc/tor.1.txt | 7 ++ src/app/config/config.c | 1 + src/app/config/or_options_st.h | 7 ++ src/core/mainloop/connection.c | 55 +++++------ src/core/mainloop/connection.h | 5 +- src/feature/dircache/dircache.c | 13 ++- src/feature/nodelist/dirlist.c | 31 ++++++ src/feature/nodelist/dirlist.h | 2 + src/feature/nodelist/nodelist.c | 43 +++++++-- src/feature/nodelist/nodelist.h | 2 + src/test/test_address_set.c | 15 ++- src/test/test_bwmgt.c | 209 +++++++++++++++++++++++++++++++++++++++- 13 files changed, 348 insertions(+), 47 deletions(-)

1 0

[tor/maint-0.4.2] dirauth: Rename function for better clarity
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit 7b4d9fabefa97e1121482ace6bc712cd2b00ff0f Author: David Goulet <dgoulet(a)torproject.org> Date: Thu Feb 6 09:49:00 2020 -0500 dirauth: Rename function for better clarity Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/feature/nodelist/dirlist.c | 2 +- src/feature/nodelist/dirlist.h | 2 +- src/feature/nodelist/nodelist.c | 2 +- src/test/test_address_set.c | 13 +++++++------ 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/src/feature/nodelist/dirlist.c b/src/feature/nodelist/dirlist.c index 749ff06a5..ccbb37851 100644 --- a/src/feature/nodelist/dirlist.c +++ b/src/feature/nodelist/dirlist.c @@ -67,7 +67,7 @@ add_trusted_dir_to_nodelist_addr_set(const dir_server_t *dir) /** Go over the trusted directory server list and add their address(es) to the * nodelist address set. This is called everytime a new consensus is set. */ MOCK_IMPL(void, -dirlist_add_trusted_addresses, (void)) +dirlist_add_trusted_dir_addresses, (void)) { if (!trusted_dir_servers) { return; diff --git a/src/feature/nodelist/dirlist.h b/src/feature/nodelist/dirlist.h index d4bdb4ced..c49162f1e 100644 --- a/src/feature/nodelist/dirlist.h +++ b/src/feature/nodelist/dirlist.h @@ -44,6 +44,6 @@ void dir_server_add(dir_server_t *ent); void clear_dir_servers(void); void dirlist_free_all(void); -MOCK_DECL(void, dirlist_add_trusted_addresses, (void)); +MOCK_DECL(void, dirlist_add_trusted_dir_addresses, (void)); #endif /* !defined(TOR_DIRLIST_H) */ diff --git a/src/feature/nodelist/nodelist.c b/src/feature/nodelist/nodelist.c index 34214880d..9191173c3 100644 --- a/src/feature/nodelist/nodelist.c +++ b/src/feature/nodelist/nodelist.c @@ -691,7 +691,7 @@ nodelist_set_consensus(networkstatus_t *ns) } SMARTLIST_FOREACH_END(node); /* Then, add all trusted configured directories. Some might not be in the * consensus so make sure we know them. */ - dirlist_add_trusted_addresses(); + dirlist_add_trusted_dir_addresses(); if (! authdir) { SMARTLIST_FOREACH_BEGIN(the_nodelist->nodes, node_t *, node) { diff --git a/src/test/test_address_set.c b/src/test/test_address_set.c index 81dc6dff5..6e299d779 100644 --- a/src/test/test_address_set.c +++ b/src/test/test_address_set.c @@ -33,7 +33,7 @@ mock_networkstatus_get_latest_consensus_by_flavor(consensus_flavor_t f) } static void -mock_dirlist_add_trusted_addresses(void) +mock_dirlist_add_trusted_dir_addresses(void) { return; } @@ -105,8 +105,8 @@ test_nodelist(void *arg) mock_networkstatus_get_latest_consensus_by_flavor); MOCK(get_estimated_address_per_node, mock_get_estimated_address_per_node); - MOCK(dirlist_add_trusted_addresses, - mock_dirlist_add_trusted_addresses); + MOCK(dirlist_add_trusted_dir_addresses, + mock_dirlist_add_trusted_dir_addresses); dummy_ns = tor_malloc_zero(sizeof(*dummy_ns)); dummy_ns->flavor = FLAV_MICRODESC; @@ -123,8 +123,9 @@ test_nodelist(void *arg) addr_per_node = 1024; /* No node no nothing. The lookups should be empty. We've mocked the - * dirlist_add_trusted_addresses in order for _no_ authorities to be added - * to the filter else it makes this test to trigger many false positive. */ + * dirlist_add_trusted_dir_addresses in order for _no_ authorities to be + * added to the filter else it makes this test to trigger many false + * positive. */ nodelist_set_consensus(dummy_ns); /* The address set should be empty. */ @@ -178,7 +179,7 @@ test_nodelist(void *arg) UNMOCK(networkstatus_get_latest_consensus); UNMOCK(networkstatus_get_latest_consensus_by_flavor); UNMOCK(get_estimated_address_per_node); - UNMOCK(dirlist_add_trusted_addresses); + UNMOCK(dirlist_add_trusted_dir_addresses); } struct testcase_t address_set_tests[] = {

1 0

[tor/master] Merge branch 'maint-0.4.2' into maint-0.4.3
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit f686631918a302344c7f1bf6effb7f0d7903470a Merge: dbadfe75e b50494233 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Mar 13 16:45:17 2020 -0400 Merge branch 'maint-0.4.2' into maint-0.4.3

1 0

[tor/master] Merge branch 'maint-0.4.3'
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit 3168b44857458e829ddbaf9a92d5b9eb6d6ea65e Merge: 9cf7839df f68663191 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Mar 13 16:45:26 2020 -0400 Merge branch 'maint-0.4.3'

1 0

[tor/master] Merge remote-tracking branch 'tor-github/pr/1718/head' into maint-0.4.2
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit b504942331dabe1beb1ddb24a3aea75a565573e2 Merge: f021389e9 7b4d9fabe Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Mar 13 16:42:47 2020 -0400 Merge remote-tracking branch 'tor-github/pr/1718/head' into maint-0.4.2 changes/ticket33029 | 5 + doc/tor.1.txt | 7 ++ src/app/config/config.c | 1 + src/app/config/or_options_st.h | 7 ++ src/core/mainloop/connection.c | 55 +++++------ src/core/mainloop/connection.h | 5 +- src/feature/dircache/dircache.c | 13 ++- src/feature/nodelist/dirlist.c | 31 ++++++ src/feature/nodelist/dirlist.h | 2 + src/feature/nodelist/nodelist.c | 43 +++++++-- src/feature/nodelist/nodelist.h | 2 + src/test/test_address_set.c | 15 ++- src/test/test_bwmgt.c | 209 +++++++++++++++++++++++++++++++++++++++- 13 files changed, 348 insertions(+), 47 deletions(-)

1 0

[tor/maint-0.4.3] Merge branch 'maint-0.4.2' into maint-0.4.3
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit f686631918a302344c7f1bf6effb7f0d7903470a Merge: dbadfe75e b50494233 Author: Nick Mathewson <nickm(a)torproject.org> Date: Fri Mar 13 16:45:17 2020 -0400 Merge branch 'maint-0.4.2' into maint-0.4.3

1 0

[tor/release-0.4.2] mainloop: Rename global_write_bucket_low()
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit ee55823a11b711fc4af6c3c906c341c057a414f3 Author: David Goulet <dgoulet(a)torproject.org> Date: Wed Jan 22 11:06:11 2020 -0500 mainloop: Rename global_write_bucket_low() That function is only used to test the global bucket write limit for a directory connection. It should _not_ be used for anything else since that function looks to see if we are a directory authority. Rename it to something more meaningful. No change in behavior at this commit, only renaming. Part of #33029 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/core/mainloop/connection.c | 7 ++++--- src/core/mainloop/connection.h | 5 +++-- src/feature/dircache/dircache.c | 10 +++++----- 3 files changed, 12 insertions(+), 10 deletions(-) diff --git a/src/core/mainloop/connection.c b/src/core/mainloop/connection.c index 6094f33e4..2c075ba6b 100644 --- a/src/core/mainloop/connection.c +++ b/src/core/mainloop/connection.c @@ -3186,9 +3186,9 @@ connection_bucket_write_limit(connection_t *conn, time_t now) global_bucket_val, conn_bucket); } -/** Return 1 if the global write buckets are low enough that we +/** Return true iff the global write buckets are low enough that we * shouldn't send attempt bytes of low-priority directory stuff - * out to conn. Else return 0. + * out to conn. * Priority was 1 for v1 requests (directories and running-routers), * and 2 for v2 requests and later (statuses and descriptors). @@ -3206,7 +3206,8 @@ connection_bucket_write_limit(connection_t *conn, time_t now) * that's harder to quantify and harder to keep track of. */ int -global_write_bucket_low(connection_t *conn, size_t attempt, int priority) +connection_dir_is_global_write_low(connection_t *conn, size_t attempt, + int priority) { size_t smaller_bucket = MIN(token_bucket_rw_get_write(&global_bucket), diff --git a/src/core/mainloop/connection.h b/src/core/mainloop/connection.h index c93f1ef8e..4f15c1dd6 100644 --- a/src/core/mainloop/connection.h +++ b/src/core/mainloop/connection.h @@ -195,8 +195,9 @@ int retry_all_listeners(smartlist_t *new_conns, void connection_mark_all_noncontrol_listeners(void); void connection_mark_all_noncontrol_connections(void); -ssize_t connection_bucket_write_limit(connection_t *conn, time_t now); -int global_write_bucket_low(connection_t *conn, size_t attempt, int priority); +ssize_t connection_bucket_write_limit(struct connection_t *conn, time_t now); +int connection_dir_is_global_write_low(struct connection_t *conn, + size_t attempt, int priority); void connection_bucket_init(void); void connection_bucket_adjust(const or_options_t *options); void connection_bucket_refill_all(time_t now, diff --git a/src/feature/dircache/dircache.c b/src/feature/dircache/dircache.c index d4d0ad993..266729cdd 100644 --- a/src/feature/dircache/dircache.c +++ b/src/feature/dircache/dircache.c @@ -951,7 +951,7 @@ handle_get_current_consensus(dir_connection_t *conn, goto done; } - if (global_write_bucket_low(TO_CONN(conn), size_guess, 2)) { + if (connection_dir_is_global_write_low(TO_CONN(conn), size_guess, 2)) { log_debug(LD_DIRSERV, "Client asked for network status lists, but we've been " "writing too many bytes lately. Sending 503 Dir busy."); @@ -1060,7 +1060,7 @@ handle_get_status_vote(dir_connection_t *conn, const get_handler_args_t *args) } }); - if (global_write_bucket_low(TO_CONN(conn), estimated_len, 2)) { + if (connection_dir_is_global_write_low(TO_CONN(conn), estimated_len, 2)) { write_short_http_response(conn, 503, "Directory busy, try again later"); goto vote_done; } @@ -1119,7 +1119,7 @@ handle_get_microdesc(dir_connection_t *conn, const get_handler_args_t *args) write_short_http_response(conn, 404, "Not found"); goto done; } - if (global_write_bucket_low(TO_CONN(conn), size_guess, 2)) { + if (connection_dir_is_global_write_low(TO_CONN(conn), size_guess, 2)) { log_info(LD_DIRSERV, "Client asked for server descriptors, but we've been " "writing too many bytes lately. Sending 503 Dir busy."); @@ -1217,7 +1217,7 @@ handle_get_descriptor(dir_connection_t *conn, const get_handler_args_t *args) msg = "Not found"; write_short_http_response(conn, 404, msg); } else { - if (global_write_bucket_low(TO_CONN(conn), size_guess, 2)) { + if (connection_dir_is_global_write_low(TO_CONN(conn), size_guess, 2)) { log_info(LD_DIRSERV, "Client asked for server descriptors, but we've been " "writing too many bytes lately. Sending 503 Dir busy."); @@ -1313,7 +1313,7 @@ handle_get_keys(dir_connection_t *conn, const get_handler_args_t *args) SMARTLIST_FOREACH(certs, authority_cert_t *, c, len += c->cache_info.signed_descriptor_len); - if (global_write_bucket_low(TO_CONN(conn), + if (connection_dir_is_global_write_low(TO_CONN(conn), compress_method != NO_METHOD ? len/2 : len, 2)) { write_short_http_response(conn, 503, "Directory busy, try again later");

1 0

[tor/release-0.4.2] nodelist: Helper to add an address to the nodelist address set
by nickm＠torproject.org 13 Mar '20

13 Mar '20

commit 4152c349b4b9f7b2336a9d97356a28471e8d8f27 Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Jan 28 09:17:34 2020 -0500 nodelist: Helper to add an address to the nodelist address set We separate v4 and v6 because we often use an IPv4 address represented with a uint32_t instead of a tor_addr_t. This will be used to also add the trusted directory addresses taken from the configuration. The trusted directories from the consensus are already added to the address set from their descriptor. Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- src/feature/nodelist/nodelist.c | 31 ++++++++++++++++++++++++++----- src/feature/nodelist/nodelist.h | 2 ++ 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/src/feature/nodelist/nodelist.c b/src/feature/nodelist/nodelist.c index bd80fc1b4..90c655d12 100644 --- a/src/feature/nodelist/nodelist.c +++ b/src/feature/nodelist/nodelist.c @@ -455,22 +455,43 @@ node_add_to_address_set(const node_t *node) if (node->rs) { if (node->rs->addr) - address_set_add_ipv4h(the_nodelist->node_addrs, node->rs->addr); + nodelist_add_addr4_to_address_set(node->rs->addr); if (!tor_addr_is_null(&node->rs->ipv6_addr)) - address_set_add(the_nodelist->node_addrs, &node->rs->ipv6_addr); + nodelist_add_addr6_to_address_set(&node->rs->ipv6_addr); } if (node->ri) { if (node->ri->addr) - address_set_add_ipv4h(the_nodelist->node_addrs, node->ri->addr); + nodelist_add_addr4_to_address_set(node->ri->addr); if (!tor_addr_is_null(&node->ri->ipv6_addr)) - address_set_add(the_nodelist->node_addrs, &node->ri->ipv6_addr); + nodelist_add_addr6_to_address_set(&node->ri->ipv6_addr); } if (node->md) { if (!tor_addr_is_null(&node->md->ipv6_addr)) - address_set_add(the_nodelist->node_addrs, &node->md->ipv6_addr); + nodelist_add_addr6_to_address_set(&node->md->ipv6_addr); } } +/** Add the given v4 address into the nodelist address set. */ +void +nodelist_add_addr4_to_address_set(const uint32_t addr) +{ + if (!the_nodelist || !the_nodelist->node_addrs || addr == 0) { + return; + } + address_set_add_ipv4h(the_nodelist->node_addrs, addr); +} + +/** Add the given v6 address into the nodelist address set. */ +void +nodelist_add_addr6_to_address_set(const tor_addr_t *addr) +{ + if (BUG(!addr) || tor_addr_is_null(addr) || tor_addr_is_v4(addr) || + !the_nodelist || !the_nodelist->node_addrs) { + return; + } + address_set_add(the_nodelist->node_addrs, addr); +} + /** Return true if addr is the address of some node in the nodelist. * If not, probably return false. */ int diff --git a/src/feature/nodelist/nodelist.h b/src/feature/nodelist/nodelist.h index af144c197..87cfa48e2 100644 --- a/src/feature/nodelist/nodelist.h +++ b/src/feature/nodelist/nodelist.h @@ -35,6 +35,8 @@ node_t *nodelist_add_microdesc(microdesc_t *md); void nodelist_set_consensus(networkstatus_t *ns); void nodelist_ensure_freshness(networkstatus_t *ns); int nodelist_probably_contains_address(const tor_addr_t *addr); +void nodelist_add_addr4_to_address_set(const uint32_t addr); +void nodelist_add_addr6_to_address_set(const tor_addr_t *addr); void nodelist_remove_microdesc(const char *identity_digest, microdesc_t *md); void nodelist_remove_routerinfo(routerinfo_t *ri);

1 0