November 2020 - tor-commits - lists.torproject.org

[tor/master] Merge branch 'maint-0.4.5' into master
by nickm＠torproject.org 17 Nov '20

17 Nov '20

commit 3900b1937973141b5c75f7221162375f7a2e1f9c Merge: 7c06707750 d04a27bed2 Author: Nick Mathewson <nickm(a)torproject.org> Date: Tue Nov 17 10:53:39 2020 -0500 Merge branch 'maint-0.4.5' into master changes/ticket40195 | 5 +++++ src/feature/relay/relay_config.c | 4 ++++ 2 files changed, 9 insertions(+)

1 0

[tor/master] config: Really ignore non ORPorts when removing duplicates
by nickm＠torproject.org 17 Nov '20

17 Nov '20

commit d04a27bed20cf44766fc30583c7a00108f42a09a Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Nov 17 09:40:16 2020 -0500 config: Really ignore non ORPorts when removing duplicates The function in charge of removing duplicate ORPorts from our configured ports was skipping all non ORPorts port but only for the outer loop thus resulting in comparing an ORPort with a non-ORPort which lead to problems. For example, tor configured with the following would fail: ORPort auto DirPort auto Both end up being the same configuration except that one is a OR listener and one is a Dir listener. Thus because of the missing check in the inner loop, they looked exactly the same and thus one is removed. Fixes #40195 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- changes/ticket40195 | 5 +++++ src/feature/relay/relay_config.c | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/changes/ticket40195 b/changes/ticket40195 new file mode 100644 index 0000000000..caa0bace94 --- /dev/null +++ b/changes/ticket40195 @@ -0,0 +1,5 @@ + o Minor bugfixes (configuration, ports): + - Fix an issue where an ORPort was validated against other type of ports + when it should have been only checked against other ORPorts. This lead to + "DirPort auto" to be ignored and failing to be used. Fixes bug 40195; + bugfix on 0.4.5.1-alpha. diff --git a/src/feature/relay/relay_config.c b/src/feature/relay/relay_config.c index ea03f43e13..e8c29fa7ed 100644 --- a/src/feature/relay/relay_config.c +++ b/src/feature/relay/relay_config.c @@ -227,6 +227,10 @@ remove_duplicate_orports(smartlist_t *ports) if (removing[j]) { continue; } + /* Skip non ORPorts. */ + if (next->type != CONN_TYPE_OR_LISTENER) { + continue; + } /* Same address family and same port number, we have a match. */ if (tor_addr_family(&current->addr) == tor_addr_family(&next->addr) && current->port == next->port) {

1 0

[tor/maint-0.4.5] config: Really ignore non ORPorts when removing duplicates
by nickm＠torproject.org 17 Nov '20

17 Nov '20

commit d04a27bed20cf44766fc30583c7a00108f42a09a Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Nov 17 09:40:16 2020 -0500 config: Really ignore non ORPorts when removing duplicates The function in charge of removing duplicate ORPorts from our configured ports was skipping all non ORPorts port but only for the outer loop thus resulting in comparing an ORPort with a non-ORPort which lead to problems. For example, tor configured with the following would fail: ORPort auto DirPort auto Both end up being the same configuration except that one is a OR listener and one is a Dir listener. Thus because of the missing check in the inner loop, they looked exactly the same and thus one is removed. Fixes #40195 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- changes/ticket40195 | 5 +++++ src/feature/relay/relay_config.c | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/changes/ticket40195 b/changes/ticket40195 new file mode 100644 index 0000000000..caa0bace94 --- /dev/null +++ b/changes/ticket40195 @@ -0,0 +1,5 @@ + o Minor bugfixes (configuration, ports): + - Fix an issue where an ORPort was validated against other type of ports + when it should have been only checked against other ORPorts. This lead to + "DirPort auto" to be ignored and failing to be used. Fixes bug 40195; + bugfix on 0.4.5.1-alpha. diff --git a/src/feature/relay/relay_config.c b/src/feature/relay/relay_config.c index ea03f43e13..e8c29fa7ed 100644 --- a/src/feature/relay/relay_config.c +++ b/src/feature/relay/relay_config.c @@ -227,6 +227,10 @@ remove_duplicate_orports(smartlist_t *ports) if (removing[j]) { continue; } + /* Skip non ORPorts. */ + if (next->type != CONN_TYPE_OR_LISTENER) { + continue; + } /* Same address family and same port number, we have a match. */ if (tor_addr_family(&current->addr) == tor_addr_family(&next->addr) && current->port == next->port) {

1 0

[tor/release-0.4.5] config: Really ignore non ORPorts when removing duplicates
by nickm＠torproject.org 17 Nov '20

17 Nov '20

commit d04a27bed20cf44766fc30583c7a00108f42a09a Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Nov 17 09:40:16 2020 -0500 config: Really ignore non ORPorts when removing duplicates The function in charge of removing duplicate ORPorts from our configured ports was skipping all non ORPorts port but only for the outer loop thus resulting in comparing an ORPort with a non-ORPort which lead to problems. For example, tor configured with the following would fail: ORPort auto DirPort auto Both end up being the same configuration except that one is a OR listener and one is a Dir listener. Thus because of the missing check in the inner loop, they looked exactly the same and thus one is removed. Fixes #40195 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- changes/ticket40195 | 5 +++++ src/feature/relay/relay_config.c | 4 ++++ 2 files changed, 9 insertions(+) diff --git a/changes/ticket40195 b/changes/ticket40195 new file mode 100644 index 0000000000..caa0bace94 --- /dev/null +++ b/changes/ticket40195 @@ -0,0 +1,5 @@ + o Minor bugfixes (configuration, ports): + - Fix an issue where an ORPort was validated against other type of ports + when it should have been only checked against other ORPorts. This lead to + "DirPort auto" to be ignored and failing to be used. Fixes bug 40195; + bugfix on 0.4.5.1-alpha. diff --git a/src/feature/relay/relay_config.c b/src/feature/relay/relay_config.c index ea03f43e13..e8c29fa7ed 100644 --- a/src/feature/relay/relay_config.c +++ b/src/feature/relay/relay_config.c @@ -227,6 +227,10 @@ remove_duplicate_orports(smartlist_t *ports) if (removing[j]) { continue; } + /* Skip non ORPorts. */ + if (next->type != CONN_TYPE_OR_LISTENER) { + continue; + } /* Same address family and same port number, we have a match. */ if (tor_addr_family(&current->addr) == tor_addr_family(&next->addr) && current->port == next->port) {

1 0

[torspec/master] dir-spec.txt: Introduce extrainfo fields for v3 stats.
by dgoulet＠torproject.org 17 Nov '20

17 Nov '20

commit 1337eb21c8d4368979fb207306e6fcb7ad006463 Author: George Kadianakis <desnacked(a)riseup.net> Date: Tue Oct 27 11:04:00 2020 +0200 dir-spec.txt: Introduce extrainfo fields for v3 stats. --- dir-spec.txt | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/dir-spec.txt b/dir-spec.txt index 1aca15b..858114d 100644 --- a/dir-spec.txt +++ b/dir-spec.txt @@ -1213,6 +1213,8 @@ "hidserv-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL [At most once.] + "hidserv-v3-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL + [At most once.] YYYY-MM-DD HH:MM:SS defines the end of the included measurement interval of length NSEC seconds (86400 seconds by default). @@ -1223,6 +1225,8 @@ "hidserv-rend-relayed-cells" SP NUM SP key=val SP key=val ... NL [At most once.] + "hidserv-rend-v3-relayed-cells" SP NUM SP key=val SP key=val ... NL + [At most once.] Approximate number of RELAY cells seen in either direction on a circuit after receiving and successfully processing a RENDEZVOUS1 @@ -1241,6 +1245,8 @@ "hidserv-dir-onions-seen" SP NUM SP key=val SP key=val ... NL [At most once.] + "hidserv-dir-v3-onions-seen" SP NUM SP key=val SP key=val ... NL + [At most once.] Approximate number of unique hidden-service identities seen in descriptors published to and accepted by this hidden-service

1 0

[torspec/master] Merge branch 'tor-gitlab/mr/15' into master
by dgoulet＠torproject.org 17 Nov '20

17 Nov '20

commit ae025d53979dc62136843c1917fad499ce1abe9b Merge: bfaf4c2 1337eb2 Author: David Goulet <dgoulet(a)torproject.org> Date: Tue Nov 17 10:50:27 2020 -0500 Merge branch 'tor-gitlab/mr/15' into master dir-spec.txt | 6 ++++++ 1 file changed, 6 insertions(+)

1 0

[tor/master] Make room for v3: s/hs_stats/hs_v2_stats/
by dgoulet＠torproject.org 17 Nov '20

17 Nov '20

commit 1de7843658e00baa2271048fb3e3e436639e70c4 Author: George Kadianakis <desnacked(a)riseup.net> Date: Mon Oct 26 18:19:25 2020 +0200 Make room for v3: s/hs_stats/hs_v2_stats/ --- src/core/mainloop/mainloop.c | 2 +- src/feature/stats/rephist.c | 98 ++++++++++++++++++++++---------------------- src/feature/stats/rephist.h | 8 ++-- 3 files changed, 54 insertions(+), 54 deletions(-) diff --git a/src/core/mainloop/mainloop.c b/src/core/mainloop/mainloop.c index c75039b378..4b4ac5c4a7 100644 --- a/src/core/mainloop/mainloop.c +++ b/src/core/mainloop/mainloop.c @@ -1937,7 +1937,7 @@ write_stats_file_callback(time_t now, const or_options_t *options) next_time_to_write_stats_files = next_write; } if (options->HiddenServiceStatistics) { - time_t next_write = rep_hist_hs_stats_write(now); + time_t next_write = rep_hist_hs_v2_stats_write(now); if (next_write && next_write < next_time_to_write_stats_files) next_time_to_write_stats_files = next_write; } diff --git a/src/feature/stats/rephist.c b/src/feature/stats/rephist.c index b66eed80ef..739ead195f 100644 --- a/src/feature/stats/rephist.c +++ b/src/feature/stats/rephist.c @@ -1710,92 +1710,92 @@ rep_hist_log_circuit_handshake_stats(time_t now) /** Start of the current hidden service stats interval or 0 if we're * not collecting hidden service statistics. */ -static time_t start_of_hs_stats_interval; +static time_t start_of_hs_v2_stats_interval; /** Carries the various hidden service statistics, and any other * information needed. */ -typedef struct hs_stats_t { +typedef struct hs_v2_stats_t { /** How many relay cells have we seen as rendezvous points? */ uint64_t rp_relay_cells_seen; /** Set of unique public key digests we've seen this stat period * (could also be implemented as sorted smartlist). */ digestmap_t *v2_onions_seen_this_period; -} hs_stats_t; +} hs_v2_stats_t; /** Our statistics structure singleton. */ -static hs_stats_t *hs_stats = NULL; +static hs_v2_stats_t *hs_v2_stats = NULL; -/** Allocate, initialize and return an hs_stats_t structure. */ -static hs_stats_t * -hs_stats_new(void) +/** Allocate, initialize and return an hs_v2_stats_t structure. */ +static hs_v2_stats_t * +hs_v2_stats_new(void) { - hs_stats_t *new_hs_stats = tor_malloc_zero(sizeof(hs_stats_t)); - new_hs_stats->v2_onions_seen_this_period = digestmap_new(); + hs_v2_stats_t *new_hs_v2_stats = tor_malloc_zero(sizeof(hs_v2_stats_t)); + new_hs_v2_stats->v2_onions_seen_this_period = digestmap_new(); - return new_hs_stats; + return new_hs_v2_stats; } -#define hs_stats_free(val) \ - FREE_AND_NULL(hs_stats_t, hs_stats_free_, (val)) +#define hs_v2_stats_free(val) \ + FREE_AND_NULL(hs_v2_stats_t, hs_v2_stats_free_, (val)) -/** Free an hs_stats_t structure. */ +/** Free an hs_v2_stats_t structure. */ static void -hs_stats_free_(hs_stats_t *victim_hs_stats) +hs_v2_stats_free_(hs_v2_stats_t *victim_hs_v2_stats) { - if (!victim_hs_stats) { + if (!victim_hs_v2_stats) { return; } - digestmap_free(victim_hs_stats->v2_onions_seen_this_period, NULL); - tor_free(victim_hs_stats); + digestmap_free(victim_hs_v2_stats->v2_onions_seen_this_period, NULL); + tor_free(victim_hs_v2_stats); } /** Initialize hidden service statistics. */ void -rep_hist_hs_stats_init(time_t now) +rep_hist_hs_v2_stats_init(time_t now) { - if (!hs_stats) { - hs_stats = hs_stats_new(); + if (!hs_v2_stats) { + hs_v2_stats = hs_v2_stats_new(); } - start_of_hs_stats_interval = now; + start_of_hs_v2_stats_interval = now; } /** Clear history of hidden service statistics and set the measurement * interval start to now. */ static void -rep_hist_reset_hs_stats(time_t now) +rep_hist_reset_hs_v2_stats(time_t now) { - if (!hs_stats) { - hs_stats = hs_stats_new(); + if (!hs_v2_stats) { + hs_v2_stats = hs_v2_stats_new(); } - hs_stats->rp_relay_cells_seen = 0; + hs_v2_stats->rp_relay_cells_seen = 0; - digestmap_free(hs_stats->v2_onions_seen_this_period, NULL); - hs_stats->v2_onions_seen_this_period = digestmap_new(); + digestmap_free(hs_v2_stats->v2_onions_seen_this_period, NULL); + hs_v2_stats->v2_onions_seen_this_period = digestmap_new(); - start_of_hs_stats_interval = now; + start_of_hs_v2_stats_interval = now; } /** Stop collecting hidden service stats in a way that we can re-start * doing so in rep_hist_buffer_stats_init(). */ void -rep_hist_hs_stats_term(void) +rep_hist_hs_v2_stats_term(void) { - rep_hist_reset_hs_stats(0); + rep_hist_reset_hs_v2_stats(0); } /** We saw a new HS relay cell, Count it! */ void rep_hist_seen_new_rp_cell(void) { - if (!hs_stats) { + if (!hs_v2_stats) { return; // We're not collecting stats } - hs_stats->rp_relay_cells_seen++; + hs_v2_stats->rp_relay_cells_seen++; } /** As HSDirs, we saw another hidden service with public key @@ -1806,7 +1806,7 @@ rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey) { char pubkey_hash[DIGEST_LEN]; - if (!hs_stats) { + if (!hs_v2_stats) { return; // We're not collecting stats } @@ -1820,9 +1820,9 @@ rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey) /* Check if this is the first time we've seen this hidden service. If it is, count it as new. */ - if (!digestmap_get(hs_stats->v2_onions_seen_this_period, + if (!digestmap_get(hs_v2_stats->v2_onions_seen_this_period, pubkey_hash)) { - digestmap_set(hs_stats->v2_onions_seen_this_period, + digestmap_set(hs_v2_stats->v2_onions_seen_this_period, pubkey_hash, (void*)(uintptr_t)1); } } @@ -1853,15 +1853,15 @@ rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey) /** Allocate and return a string containing hidden service stats that * are meant to be placed in the extra-info descriptor. */ static char * -rep_hist_format_hs_stats(time_t now) +rep_hist_format_hs_v2_stats(time_t now) { char t[ISO_TIME_LEN+1]; - char *hs_stats_string; + char *hs_v2_stats_string; int64_t obfuscated_cells_seen; int64_t obfuscated_onions_seen; uint64_t rounded_cells_seen - = round_uint64_to_next_multiple_of(hs_stats->rp_relay_cells_seen, + = round_uint64_to_next_multiple_of(hs_v2_stats->rp_relay_cells_seen, REND_CELLS_BIN_SIZE); rounded_cells_seen = MIN(rounded_cells_seen, INT64_MAX); obfuscated_cells_seen = add_laplace_noise((int64_t)rounded_cells_seen, @@ -1870,7 +1870,7 @@ rep_hist_format_hs_stats(time_t now) uint64_t rounded_onions_seen = round_uint64_to_next_multiple_of((size_t)digestmap_size( - hs_stats->v2_onions_seen_this_period), + hs_v2_stats->v2_onions_seen_this_period), ONIONS_SEEN_BIN_SIZE); rounded_onions_seen = MIN(rounded_onions_seen, INT64_MAX); obfuscated_onions_seen = add_laplace_noise((int64_t)rounded_onions_seen, @@ -1878,19 +1878,19 @@ rep_hist_format_hs_stats(time_t now) ONIONS_SEEN_EPSILON); format_iso_time(t, now); - tor_asprintf(&hs_stats_string, "hidserv-stats-end %s (%d s)\n" + tor_asprintf(&hs_v2_stats_string, "hidserv-stats-end %s (%d s)\n" "hidserv-rend-relayed-cells %"PRId64" delta_f=%d " "epsilon=%.2f bin_size=%d\n" "hidserv-dir-onions-seen %"PRId64" delta_f=%d " "epsilon=%.2f bin_size=%d\n", - t, (unsigned) (now - start_of_hs_stats_interval), + t, (unsigned) (now - start_of_hs_v2_stats_interval), (obfuscated_cells_seen), REND_CELLS_DELTA_F, REND_CELLS_EPSILON, REND_CELLS_BIN_SIZE, (obfuscated_onions_seen), ONIONS_SEEN_DELTA_F, ONIONS_SEEN_EPSILON, ONIONS_SEEN_BIN_SIZE); - return hs_stats_string; + return hs_v2_stats_string; } /** If 24 hours have passed since the beginning of the current HS @@ -1899,23 +1899,23 @@ rep_hist_format_hs_stats(time_t now) * when we would next want to write buffer stats or 0 if we never want to * write. */ time_t -rep_hist_hs_stats_write(time_t now) +rep_hist_hs_v2_stats_write(time_t now) { char *str = NULL; - if (!start_of_hs_stats_interval) { + if (!start_of_hs_v2_stats_interval) { return 0; /* Not initialized. */ } - if (start_of_hs_stats_interval + WRITE_STATS_INTERVAL > now) { + if (start_of_hs_v2_stats_interval + WRITE_STATS_INTERVAL > now) { goto done; /* Not ready to write */ } /* Generate history string. */ - str = rep_hist_format_hs_stats(now); + str = rep_hist_format_hs_v2_stats(now); /* Reset HS history. */ - rep_hist_reset_hs_stats(now); + rep_hist_reset_hs_v2_stats(now); /* Try to write to disk. */ if (!check_or_create_data_subdir("stats")) { @@ -1925,7 +1925,7 @@ rep_hist_hs_stats_write(time_t now) done: tor_free(str); - return start_of_hs_stats_interval + WRITE_STATS_INTERVAL; + return start_of_hs_v2_stats_interval + WRITE_STATS_INTERVAL; } static uint64_t link_proto_count[MAX_LINK_PROTO+1][2]; @@ -2134,7 +2134,7 @@ rep_hist_log_link_protocol_counts(void) void rep_hist_free_all(void) { - hs_stats_free(hs_stats); + hs_v2_stats_free(hs_v2_stats); digestmap_free(history_map, free_or_history); tor_free(exit_bytes_read); diff --git a/src/feature/stats/rephist.h b/src/feature/stats/rephist.h index c9ebc5c328..abcb70249f 100644 --- a/src/feature/stats/rephist.h +++ b/src/feature/stats/rephist.h @@ -63,10 +63,10 @@ void rep_hist_log_circuit_handshake_stats(time_t now); MOCK_DECL(int, rep_hist_get_circuit_handshake_requested, (uint16_t type)); MOCK_DECL(int, rep_hist_get_circuit_handshake_assigned, (uint16_t type)); -void rep_hist_hs_stats_init(time_t now); -void rep_hist_hs_stats_term(void); -time_t rep_hist_hs_stats_write(time_t now); -char *rep_hist_get_hs_stats_string(void); +void rep_hist_hs_v2_stats_init(time_t now); +void rep_hist_hs_v2_stats_term(void); +time_t rep_hist_hs_v2_stats_write(time_t now); +char *rep_hist_get_hs_v2_stats_string(void); void rep_hist_seen_new_rp_cell(void); void rep_hist_stored_maybe_new_hs(const crypto_pk_t *pubkey);

1 0

[tor/master] Make room for v3: Rename rephist objects to signify they are v2-only.
by dgoulet＠torproject.org 17 Nov '20

17 Nov '20

commit 62d682e62422eecd40f2e2ba4d222bb2795dad53 Author: George Kadianakis <desnacked(a)riseup.net> Date: Wed Jul 1 13:29:33 2020 +0300 Make room for v3: Rename rephist objects to signify they are v2-only. --- src/feature/rend/rendcache.c | 2 +- src/feature/stats/rephist.c | 18 +++++++++--------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/src/feature/rend/rendcache.c b/src/feature/rend/rendcache.c index 53fec7532f..715e70eba7 100644 --- a/src/feature/rend/rendcache.c +++ b/src/feature/rend/rendcache.c @@ -718,7 +718,7 @@ rend_cache_store_v2_desc_as_dir(const char *desc) safe_str(desc_id_base32), (int)encoded_size); /* Statistics: Note down this potentially new HS. */ if (options->HiddenServiceStatistics) { - rep_hist_stored_maybe_new_hs(e->parsed->pk); + rep_hist_hsdir_stored_maybe_new_v2_onion(e->parsed->pk); } number_stored++; diff --git a/src/feature/stats/rephist.c b/src/feature/stats/rephist.c index b6730e1226..b66eed80ef 100644 --- a/src/feature/stats/rephist.c +++ b/src/feature/stats/rephist.c @@ -1720,7 +1720,7 @@ typedef struct hs_stats_t { /** Set of unique public key digests we've seen this stat period * (could also be implemented as sorted smartlist). */ - digestmap_t *onions_seen_this_period; + digestmap_t *v2_onions_seen_this_period; } hs_stats_t; /** Our statistics structure singleton. */ @@ -1731,7 +1731,7 @@ static hs_stats_t * hs_stats_new(void) { hs_stats_t *new_hs_stats = tor_malloc_zero(sizeof(hs_stats_t)); - new_hs_stats->onions_seen_this_period = digestmap_new(); + new_hs_stats->v2_onions_seen_this_period = digestmap_new(); return new_hs_stats; } @@ -1747,7 +1747,7 @@ hs_stats_free_(hs_stats_t *victim_hs_stats) return; } - digestmap_free(victim_hs_stats->onions_seen_this_period, NULL); + digestmap_free(victim_hs_stats->v2_onions_seen_this_period, NULL); tor_free(victim_hs_stats); } @@ -1773,8 +1773,8 @@ rep_hist_reset_hs_stats(time_t now) hs_stats->rp_relay_cells_seen = 0; - digestmap_free(hs_stats->onions_seen_this_period, NULL); - hs_stats->onions_seen_this_period = digestmap_new(); + digestmap_free(hs_stats->v2_onions_seen_this_period, NULL); + hs_stats->v2_onions_seen_this_period = digestmap_new(); start_of_hs_stats_interval = now; } @@ -1802,7 +1802,7 @@ rep_hist_seen_new_rp_cell(void) * pubkey. Check whether we have counted it before, if not * count it now! */ void -rep_hist_stored_maybe_new_hs(const crypto_pk_t *pubkey) +rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey) { char pubkey_hash[DIGEST_LEN]; @@ -1820,9 +1820,9 @@ rep_hist_stored_maybe_new_hs(const crypto_pk_t *pubkey) /* Check if this is the first time we've seen this hidden service. If it is, count it as new. */ - if (!digestmap_get(hs_stats->onions_seen_this_period, + if (!digestmap_get(hs_stats->v2_onions_seen_this_period, pubkey_hash)) { - digestmap_set(hs_stats->onions_seen_this_period, + digestmap_set(hs_stats->v2_onions_seen_this_period, pubkey_hash, (void*)(uintptr_t)1); } } @@ -1870,7 +1870,7 @@ rep_hist_format_hs_stats(time_t now) uint64_t rounded_onions_seen = round_uint64_to_next_multiple_of((size_t)digestmap_size( - hs_stats->onions_seen_this_period), + hs_stats->v2_onions_seen_this_period), ONIONS_SEEN_BIN_SIZE); rounded_onions_seen = MIN(rounded_onions_seen, INT64_MAX); obfuscated_onions_seen = add_laplace_noise((int64_t)rounded_onions_seen,

1 0

[tor/master] Make room for v3: Complete move from hs_stats_t to hs_v2_stats_t.
by dgoulet＠torproject.org 17 Nov '20

17 Nov '20

commit 5ed7fcec41db16820c3777451d083d0d74f124ce Author: George Kadianakis <desnacked(a)riseup.net> Date: Mon Oct 26 18:27:16 2020 +0200 Make room for v3: Complete move from hs_stats_t to hs_v2_stats_t. --- src/feature/stats/rephist.c | 31 +++++++++++++++---------------- src/feature/stats/rephist.h | 40 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 53 insertions(+), 18 deletions(-) diff --git a/src/feature/stats/rephist.c b/src/feature/stats/rephist.c index 739ead195f..bde65ea9d9 100644 --- a/src/feature/stats/rephist.c +++ b/src/feature/stats/rephist.c @@ -1712,20 +1712,11 @@ rep_hist_log_circuit_handshake_stats(time_t now) * not collecting hidden service statistics. */ static time_t start_of_hs_v2_stats_interval; -/** Carries the various hidden service statistics, and any other - * information needed. */ -typedef struct hs_v2_stats_t { - /** How many relay cells have we seen as rendezvous points? */ - uint64_t rp_relay_cells_seen; - - /** Set of unique public key digests we've seen this stat period - * (could also be implemented as sorted smartlist). */ - digestmap_t *v2_onions_seen_this_period; -} hs_v2_stats_t; - -/** Our statistics structure singleton. */ +/** Our v2 statistics structure singleton. */ static hs_v2_stats_t *hs_v2_stats = NULL; +/** HSv2 stats */ + /** Allocate, initialize and return an hs_v2_stats_t structure. */ static hs_v2_stats_t * hs_v2_stats_new(void) @@ -1771,7 +1762,7 @@ rep_hist_reset_hs_v2_stats(time_t now) hs_v2_stats = hs_v2_stats_new(); } - hs_v2_stats->rp_relay_cells_seen = 0; + hs_v2_stats->rp_v2_relay_cells_seen = 0; digestmap_free(hs_v2_stats->v2_onions_seen_this_period, NULL); hs_v2_stats->v2_onions_seen_this_period = digestmap_new(); @@ -1861,7 +1852,7 @@ rep_hist_format_hs_v2_stats(time_t now) int64_t obfuscated_onions_seen; uint64_t rounded_cells_seen - = round_uint64_to_next_multiple_of(hs_v2_stats->rp_relay_cells_seen, + = round_uint64_to_next_multiple_of(hs_v2_stats->rp_v2_relay_cells_seen, REND_CELLS_BIN_SIZE); rounded_cells_seen = MIN(rounded_cells_seen, INT64_MAX); obfuscated_cells_seen = add_laplace_noise((int64_t)rounded_cells_seen, @@ -1886,8 +1877,7 @@ rep_hist_format_hs_v2_stats(time_t now) t, (unsigned) (now - start_of_hs_v2_stats_interval), (obfuscated_cells_seen), REND_CELLS_DELTA_F, REND_CELLS_EPSILON, REND_CELLS_BIN_SIZE, - (obfuscated_onions_seen), - ONIONS_SEEN_DELTA_F, + (obfuscated_onions_seen), ONIONS_SEEN_DELTA_F, ONIONS_SEEN_EPSILON, ONIONS_SEEN_BIN_SIZE); return hs_v2_stats_string; @@ -2155,3 +2145,12 @@ rep_hist_free_all(void) tor_assert_nonfatal(rephist_total_alloc == 0); tor_assert_nonfatal_once(rephist_total_num == 0); } + +#ifdef TOR_UNIT_TESTS +/* only exists for unit tests: get HSv2 stats object */ +const hs_v2_stats_t * +rep_hist_get_hs_v2_stats(void) +{ + return hs_v2_stats; +} + diff --git a/src/feature/stats/rephist.h b/src/feature/stats/rephist.h index abcb70249f..d5ad21e228 100644 --- a/src/feature/stats/rephist.h +++ b/src/feature/stats/rephist.h @@ -64,11 +64,10 @@ MOCK_DECL(int, rep_hist_get_circuit_handshake_requested, (uint16_t type)); MOCK_DECL(int, rep_hist_get_circuit_handshake_assigned, (uint16_t type)); void rep_hist_hs_v2_stats_init(time_t now); -void rep_hist_hs_v2_stats_term(void); time_t rep_hist_hs_v2_stats_write(time_t now); char *rep_hist_get_hs_v2_stats_string(void); void rep_hist_seen_new_rp_cell(void); -void rep_hist_stored_maybe_new_hs(const crypto_pk_t *pubkey); +void rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey); void rep_hist_free_all(void); @@ -83,6 +82,38 @@ extern int onion_handshakes_requested[MAX_ONION_HANDSHAKE_TYPE+1]; extern int onion_handshakes_assigned[MAX_ONION_HANDSHAKE_TYPE+1]; #endif +#ifdef REPHIST_PRIVATE +/** Carries the various hidden service statistics, and any other + * information needed. */ +typedef struct hs_v2_stats_t { + /** How many v2 relay cells have we seen as rendezvous points? */ + uint64_t rp_v2_relay_cells_seen; + + /** Set of unique public key digests we've seen this stat period + * (could also be implemented as sorted smartlist). */ + digestmap_t *v2_onions_seen_this_period; +} hs_v2_stats_t; + +/** Structure that contains the various statistics we keep about v3 + * services. + * + * Because of the time period logic of v3 services, v3 statistics are more + * sensitive to time than v2 stats. For this reason, we collect v3 + * statistics strictly from 12:00UTC to 12:00UTC as dictated by + * 'start_of_hs_v3_stats_interval'. + **/ +typedef struct hs_v3_stats_t { + /** How many v3 relay cells have we seen as a rendezvous point? */ + uint64_t rp_v3_relay_cells_seen; + + /* The number of unique v3 onion descriptors (actually, unique v3 blind keys) + * we've seen during the measurement period */ + digestmap_t *v3_onions_seen_this_period; +} hs_v3_stats_t; + +STATIC char *rep_hist_format_hs_v2_stats(time_t now, bool is_v3); +#endif /* defined(REPHIST_PRIVATE) */ + /** * Represents the type of a cell for padding accounting */ @@ -108,4 +139,9 @@ void rep_hist_reset_padding_counts(void); void rep_hist_prep_published_padding_counts(time_t now); void rep_hist_padding_count_timers(uint64_t num_timers); +#ifdef TOR_UNIT_TESTS +typedef struct hs_v2_stats_t hs_v2_stats_t; +const hs_v2_stats_t *rep_hist_get_hs_v2_stats(void); +#endif + #endif /* !defined(TOR_REPHIST_H) */

1 0

[tor/master] Introduce v3_stats_t structure and some of its methods.
by dgoulet＠torproject.org 17 Nov '20

17 Nov '20

commit bd28551763bd008be5a6f676410d9b6b1d011bf6 Author: George Kadianakis <desnacked(a)riseup.net> Date: Wed Oct 21 14:32:30 2020 +0300 Introduce v3_stats_t structure and some of its methods. --- src/app/main/main.c | 2 + src/core/mainloop/mainloop.c | 4 ++ src/feature/hs/hs_cache.c | 1 + src/feature/relay/router.c | 5 +++ src/feature/stats/rephist.c | 98 ++++++++++++++++++++++++++++++++++++++++++++ src/feature/stats/rephist.h | 8 +++- 6 files changed, 117 insertions(+), 1 deletion(-) diff --git a/src/app/main/main.c b/src/app/main/main.c index ff530c0ad0..c2ced30851 100644 --- a/src/app/main/main.c +++ b/src/app/main/main.c @@ -1070,6 +1070,7 @@ sandbox_init_filter(void) OPEN_DATADIR2_SUFFIX("stats", "buffer-stats", ".tmp"); OPEN_DATADIR2_SUFFIX("stats", "conn-stats", ".tmp"); OPEN_DATADIR2_SUFFIX("stats", "hidserv-stats", ".tmp"); + OPEN_DATADIR2_SUFFIX("stats", "hidserv-v3-stats", ".tmp"); OPEN_DATADIR("approved-routers"); OPEN_DATADIR_SUFFIX("fingerprint", ".tmp"); @@ -1095,6 +1096,7 @@ sandbox_init_filter(void) RENAME_SUFFIX2("stats", "buffer-stats", ".tmp"); RENAME_SUFFIX2("stats", "conn-stats", ".tmp"); RENAME_SUFFIX2("stats", "hidserv-stats", ".tmp"); + RENAME_SUFFIX2("stats", "hidserv-v3-stats", ".tmp"); RENAME_SUFFIX("hashed-fingerprint", ".tmp"); RENAME_SUFFIX("router-stability", ".tmp"); diff --git a/src/core/mainloop/mainloop.c b/src/core/mainloop/mainloop.c index 4b4ac5c4a7..25555a3f22 100644 --- a/src/core/mainloop/mainloop.c +++ b/src/core/mainloop/mainloop.c @@ -1940,6 +1940,10 @@ write_stats_file_callback(time_t now, const or_options_t *options) time_t next_write = rep_hist_hs_v2_stats_write(now); if (next_write && next_write < next_time_to_write_stats_files) next_time_to_write_stats_files = next_write; + + next_write = rep_hist_hs_v3_stats_write(now); + if (next_write && next_write < next_time_to_write_stats_files) + next_time_to_write_stats_files = next_write; } if (options->ExitPortStatistics) { time_t next_write = rep_hist_exit_stats_write(now); diff --git a/src/feature/hs/hs_cache.c b/src/feature/hs/hs_cache.c index 44cd2505fd..0688d7765d 100644 --- a/src/feature/hs/hs_cache.c +++ b/src/feature/hs/hs_cache.c @@ -19,6 +19,7 @@ #include "feature/hs/hs_descriptor.h" #include "feature/nodelist/networkstatus.h" #include "feature/rend/rendcache.h" +#include "feature/stats/rephist.h" #include "feature/hs/hs_cache.h" diff --git a/src/feature/relay/router.c b/src/feature/relay/router.c index 5ca21964b6..ea631e18fe 100644 --- a/src/feature/relay/router.c +++ b/src/feature/relay/router.c @@ -3288,6 +3288,11 @@ extrainfo_dump_to_string_stats_helper(smartlist_t *chunks, "hidserv-stats-end", now, &contents) > 0) { smartlist_add(chunks, contents); } + if (options->HiddenServiceStatistics && + load_stats_file("stats"PATH_SEPARATOR"hidserv-v3-stats", + "hidserv-v3-stats-end", now, &contents) > 0) { + smartlist_add(chunks, contents); + } if (options->EntryStatistics && load_stats_file("stats"PATH_SEPARATOR"entry-stats", "entry-stats-end", now, &contents) > 0) { diff --git a/src/feature/stats/rephist.c b/src/feature/stats/rephist.c index bde65ea9d9..daf9db074c 100644 --- a/src/feature/stats/rephist.c +++ b/src/feature/stats/rephist.c @@ -1818,6 +1818,96 @@ rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey) } } +/*** HSv3 stats ******/ + +/** Start of the current hidden service stats interval or 0 if we're not + * collecting hidden service statistics. + * + * This is particularly important for v3 statistics since this variable + * controls the start time of initial v3 stats collection. It's initialized by + * rep_hist_hs_stats_init() to the next time period start (i.e. 12:00UTC), and + * should_collect_v3_stats() ensures that functions that collect v3 stats do + * not do so sooner than that. + * + * Collecting stats from 12:00UTC to 12:00UTC is extremely important for v3 + * stats because rep_hist_hsdir_stored_maybe_new_v3_onion() uses the blinded + * key of each onion service as its double-counting index. Onion services + * rotate their descriptor at around 00:00UTC which means that their blinded + * key also changes around that time. However the precise time that onion + * services rotate their descriptors is actually when they fetch a new + * 00:00UTC consensus and that happens at a random time (e.g. it can even + * happen at 02:00UTC). This means that if we started keeping v3 stats at + * around 00:00UTC we wouldn't be able to tell when onion services change + * their blinded key and hence we would double count an unpredictable amount + * of them (for example, if an onion service fetches the 00:00UTC consensus at + * 01:00UTC it would upload to its old HSDir at 00:45UTC, and then to a + * different HSDir at 01:50UTC). + * + * For this reason, we start collecting statistics at 12:00UTC. This way we + * know that by the time we stop collecting statistics for that time period 24 + * hours later, all the onion services have switched to their new blinded + * key. This way we can predict much better how much double counting has been + * performed. + */ +static time_t start_of_hs_v3_stats_interval; + +/** Our v3 statistics structure singleton. */ +static hs_v3_stats_t *hs_v3_stats = NULL; + +/** Allocate, initialize and return an hs_v3_stats_t structure. */ +static hs_v3_stats_t * +hs_v3_stats_new(void) +{ + hs_v3_stats_t *new_hs_v3_stats = tor_malloc_zero(sizeof(hs_v3_stats_t)); + new_hs_v3_stats->v3_onions_seen_this_period = digestmap_new(); + + return new_hs_v3_stats; +} + +#define hs_v3_stats_free(val) \ + FREE_AND_NULL(hs_v3_stats_t, hs_v3_stats_free_, (val)) + +/** Free an hs_v3_stats_t structure. */ +static void +hs_v3_stats_free_(hs_v3_stats_t *victim_hs_v3_stats) +{ + if (!victim_hs_v3_stats) { + return; + } + + digestmap_free(victim_hs_v3_stats->v3_onions_seen_this_period, NULL); + tor_free(victim_hs_v3_stats); +} + +/** Clear history of hidden service statistics and set the measurement + * interval start to now. */ +static void +rep_hist_reset_hs_v3_stats(time_t now) +{ + if (!hs_v3_stats) { + hs_v3_stats = hs_v3_stats_new(); + } + + digestmap_free(hs_v3_stats->v3_onions_seen_this_period, NULL); + hs_v3_stats->v3_onions_seen_this_period = digestmap_new(); + + hs_v3_stats->rp_v3_relay_cells_seen = 0; + + start_of_hs_v3_stats_interval = now; +} + +/** Return true if it's a good time to collect v3 stats. + * + * v3 stats have a strict stats collection period (from 12:00UTC to 12:00UTC + * on the real network) and hence we don't want to collect statistics if it's + * not yet the time to do so. + */ +static bool +should_collect_v3_stats(void) +{ + return start_of_hs_v3_stats_interval <= approx_time(); +} + /* The number of cells that are supposed to be hidden from the adversary * by adding noise from the Laplace distribution. This value, divided by * EPSILON, is Laplace parameter b. It must be greather than 0. */ @@ -2125,6 +2215,7 @@ void rep_hist_free_all(void) { hs_v2_stats_free(hs_v2_stats); + hs_v3_stats_free(hs_v3_stats); digestmap_free(history_map, free_or_history); tor_free(exit_bytes_read); @@ -2154,3 +2245,10 @@ rep_hist_get_hs_v2_stats(void) return hs_v2_stats; } +/* only exists for unit tests: get HSv2 stats object */ +const hs_v3_stats_t * +rep_hist_get_hs_v3_stats(void) +{ + return hs_v3_stats; +} +#endif /* defined(TOR_UNIT_TESTS) */ diff --git a/src/feature/stats/rephist.h b/src/feature/stats/rephist.h index d5ad21e228..e6c1509498 100644 --- a/src/feature/stats/rephist.h +++ b/src/feature/stats/rephist.h @@ -69,6 +69,9 @@ char *rep_hist_get_hs_v2_stats_string(void); void rep_hist_seen_new_rp_cell(void); void rep_hist_hsdir_stored_maybe_new_v2_onion(const crypto_pk_t *pubkey); +time_t rep_hist_hs_v3_stats_write(time_t now); +char *rep_hist_get_hs_v3_stats_string(void); + void rep_hist_free_all(void); void rep_hist_note_negotiated_link_proto(unsigned link_proto, @@ -111,7 +114,8 @@ typedef struct hs_v3_stats_t { digestmap_t *v3_onions_seen_this_period; } hs_v3_stats_t; -STATIC char *rep_hist_format_hs_v2_stats(time_t now, bool is_v3); +STATIC char *rep_hist_format_hs_v2_stats(time_t now); +STATIC char *rep_hist_format_hs_v3_stats(time_t now); #endif /* defined(REPHIST_PRIVATE) */ /** @@ -142,6 +146,8 @@ void rep_hist_padding_count_timers(uint64_t num_timers); #ifdef TOR_UNIT_TESTS typedef struct hs_v2_stats_t hs_v2_stats_t; const hs_v2_stats_t *rep_hist_get_hs_v2_stats(void); +typedef struct hs_v3_stats_t hs_v3_stats_t; +const hs_v3_stats_t *rep_hist_get_hs_v3_stats(void); #endif #endif /* !defined(TOR_REPHIST_H) */

1 0