February 2021 - tor-commits - lists.torproject.org

[tor/master] Merge remote-tracking branch 'tor-gitlab/mr/308'
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit cf1f03fe1d2a3b6e500ef1bf716976d6c2d9f192 Merge: e8cce1058c 9f61d9238c Author: George Kadianakis <desnacked(a)riseup.net> Date: Fri Feb 19 13:25:47 2021 +0200 Merge remote-tracking branch 'tor-gitlab/mr/308' changes/ticket40295 | 3 +++ src/core/mainloop/connection.c | 2 ++ src/feature/metrics/metrics.c | 9 +++++++++ src/feature/metrics/metrics.h | 1 + 4 files changed, 15 insertions(+)

1 0

[tor/master] Merge branch 'maint-0.4.5'
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit 67938e019844d8143d683321faf45d7623a3a9ae Merge: cf1f03fe1d 03c686563b Author: George Kadianakis <desnacked(a)riseup.net> Date: Fri Feb 19 13:26:52 2021 +0200 Merge branch 'maint-0.4.5'

1 0

[tor/release-0.4.5] metrics: Handle the connection finished flushing event
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit 9f61d9238ce81444b988137508436255f275051d Author: David Goulet <dgoulet(a)torproject.org> Date: Wed Feb 17 09:54:05 2021 -0500 metrics: Handle the connection finished flushing event Turns out, we forgot to add the METRICS connection type fo the finished flushing handler. Fixes #40295 Signed-off-by: David Goulet <dgoulet(a)torproject.org> --- changes/ticket40295 | 3 +++ src/core/mainloop/connection.c | 2 ++ src/feature/metrics/metrics.c | 9 +++++++++ src/feature/metrics/metrics.h | 1 + 4 files changed, 15 insertions(+) diff --git a/changes/ticket40295 b/changes/ticket40295 new file mode 100644 index 0000000000..e6dba80120 --- /dev/null +++ b/changes/ticket40295 @@ -0,0 +1,3 @@ + o Minor bugfixes (metrics port): + - Fix a BUG() on the MetricsPort for an internal missing handler. Fixes bug + 40295; bugfix on 0.4.5.1-alpha. diff --git a/src/core/mainloop/connection.c b/src/core/mainloop/connection.c index 85cdec6e1e..ada542629a 100644 --- a/src/core/mainloop/connection.c +++ b/src/core/mainloop/connection.c @@ -5187,6 +5187,8 @@ connection_finished_flushing(connection_t *conn) return connection_dir_finished_flushing(TO_DIR_CONN(conn)); case CONN_TYPE_CONTROL: return connection_control_finished_flushing(TO_CONTROL_CONN(conn)); + case CONN_TYPE_METRICS: + return metrics_connection_finished_flushing(conn); default: log_err(LD_BUG,"got unexpected conn type %d.", conn->type); tor_fragile_assert(); diff --git a/src/feature/metrics/metrics.c b/src/feature/metrics/metrics.c index 7a77ab1104..9a72fe7145 100644 --- a/src/feature/metrics/metrics.c +++ b/src/feature/metrics/metrics.c @@ -258,6 +258,15 @@ metrics_connection_reached_eof(connection_t *conn) return 0; } +/** Called when conn has no more bytes left on its outbuf. Return 0 indicating + * success. */ +int +metrics_connection_finished_flushing(connection_t *conn) +{ + tor_assert(conn); + return 0; +} + /** Initialize the subsystem. */ void metrics_init(void) diff --git a/src/feature/metrics/metrics.h b/src/feature/metrics/metrics.h index 858722de59..e072519d10 100644 --- a/src/feature/metrics/metrics.h +++ b/src/feature/metrics/metrics.h @@ -28,6 +28,7 @@ buf_t *metrics_get_output(const metrics_format_t fmt); /* Connection. */ int metrics_connection_process_inbuf(struct connection_t *conn); int metrics_connection_reached_eof(struct connection_t *conn); +int metrics_connection_finished_flushing(struct connection_t *conn); /* Configuration. */ int metrics_parse_ports(or_options_t *options, smartlist_t *ports,

1 0

[tor/master] Elaborate a few of the items from Gaba's initial templates.
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit 003e237dfc604edb7833701f61e869b0ad3b19b1 Author: Alexander Færøy <ahf(a)torproject.org> Date: Fri Feb 12 20:38:21 2021 +0000 Elaborate a few of the items from Gaba's initial templates. --- .gitlab/issue_templates/bug.md | 26 ++++++++------------------ .gitlab/issue_templates/feature.md | 6 ------ 2 files changed, 8 insertions(+), 24 deletions(-) diff --git a/.gitlab/issue_templates/bug.md b/.gitlab/issue_templates/bug.md index d27dea552d..3702c1f0b6 100644 --- a/.gitlab/issue_templates/bug.md +++ b/.gitlab/issue_templates/bug.md @@ -1,16 +1,11 @@ - - ### Summary ### Steps to reproduce: -1. step 1 -2. step 2 -3. ... +1. Step 1 +2. ... ### What is the current bug behavior? @@ -22,21 +17,16 @@ ### Environment - - -- Version: e.g. 0.10, output of --version (without backslashes) -- Operating system: Debian GNU/Linux 9.1 (stretch), Windows 7, Ubuntu Xenial, etc -- Install method: distribution package, from source tarball, from git, etc -- etc... +- Which version of Tor are you using? Run `tor --version` to get the version if you are unsure. +- Which operating system are you using? For example: Debian GNU/Linux 10.1, Windows 10, Ubuntu Xenial, FreeBSD 12.2, etc. +- Which installation method did you use? Distribution package (apt, pkg, homebrew), from source tarball, from Git, etc. -### Relevant logs and/or screenshots: +### Relevant logs and/or screenshots -### Possible fixes: +### Possible fixes -/label ~bug +/label ~Bug diff --git a/.gitlab/issue_templates/feature.md b/.gitlab/issue_templates/feature.md index f0c3f1bdfe..62b2f8e944 100644 --- a/.gitlab/issue_templates/feature.md +++ b/.gitlab/issue_templates/feature.md @@ -1,7 +1,3 @@ - - ### Summary @@ -10,6 +6,4 @@ - - /label ~Feature

1 0

[tor/master] Adds issue creation templates for bugs, features, proposals and documentation.
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit 3b0e0a87e77a120f3b18285ca223c33f83b8a5c7 Author: Gaba <gaba(a)riseup.net> Date: Tue Dec 22 09:05:55 2020 -0300 Adds issue creation templates for bugs, features, proposals and documentation. --- .gitlab/issue_templates/bug.md | 42 ++++++++++++++++++++++++++++++++ .gitlab/issue_templates/documentation.md | 35 ++++++++++++++++++++++++++ .gitlab/issue_templates/feature.md | 15 ++++++++++++ .gitlab/issue_templates/proposal.md | 5 ++++ 4 files changed, 97 insertions(+) diff --git a/.gitlab/issue_templates/bug.md b/.gitlab/issue_templates/bug.md new file mode 100644 index 0000000000..d27dea552d --- /dev/null +++ b/.gitlab/issue_templates/bug.md @@ -0,0 +1,42 @@ + + +### Summary + + + +### Steps to reproduce: + +1. step 1 +2. step 2 +3. ... + +### What is the current bug behavior? + + + +### What is the expected behavior? + + + +### Environment + + + +- Version: e.g. 0.10, output of --version (without backslashes) +- Operating system: Debian GNU/Linux 9.1 (stretch), Windows 7, Ubuntu Xenial, etc +- Install method: distribution package, from source tarball, from git, etc +- etc... + +### Relevant logs and/or screenshots: + + + +### Possible fixes: + + + +/label ~bug diff --git a/.gitlab/issue_templates/documentation.md b/.gitlab/issue_templates/documentation.md new file mode 100644 index 0000000000..6f0d4b49b8 --- /dev/null +++ b/.gitlab/issue_templates/documentation.md @@ -0,0 +1,35 @@ + + +### Problem to solve + + + +* What feature(s) affected? +* What docs or doc section affected? Include links or paths. +* Is there a problem with a specific document, or a feature/process that's not addressed sufficiently in docs? +* Any other ideas or requests? + + +### Further details + + + +### Proposal + + + +### Who can address the issue + + + +### Other links/references + + + +/label ~documentation diff --git a/.gitlab/issue_templates/feature.md b/.gitlab/issue_templates/feature.md new file mode 100644 index 0000000000..f0c3f1bdfe --- /dev/null +++ b/.gitlab/issue_templates/feature.md @@ -0,0 +1,15 @@ + + +### Summary + + + +### What is the expected behavior? + + + + + +/label ~Feature diff --git a/.gitlab/issue_templates/proposal.md b/.gitlab/issue_templates/proposal.md new file mode 100644 index 0000000000..990ca70a0a --- /dev/null +++ b/.gitlab/issue_templates/proposal.md @@ -0,0 +1,5 @@ + + +### Proposal + +

1 0

[tor/master] Remove documentation issue template.
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit cdc66380021dcec7f2276164c9df4aa57e62f87f Author: Alexander Færøy <ahf(a)torproject.org> Date: Fri Feb 12 20:36:17 2021 +0000 Remove documentation issue template. I think the issues we get in here will be covered by either the bug or feature document. --- .gitlab/issue_templates/documentation.md | 35 -------------------------------- 1 file changed, 35 deletions(-) diff --git a/.gitlab/issue_templates/documentation.md b/.gitlab/issue_templates/documentation.md deleted file mode 100644 index 6f0d4b49b8..0000000000 --- a/.gitlab/issue_templates/documentation.md +++ /dev/null @@ -1,35 +0,0 @@ - - -### Problem to solve - - - -* What feature(s) affected? -* What docs or doc section affected? Include links or paths. -* Is there a problem with a specific document, or a feature/process that's not addressed sufficiently in docs? -* Any other ideas or requests? - - -### Further details - - - -### Proposal - - - -### Who can address the issue - - - -### Other links/references - - - -/label ~documentation

1 0

[tor/master] Use titlecase for template filenames.
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit 22df6a3127fe4f55217186be068048905a24d14d Author: Alexander Færøy <ahf(a)torproject.org> Date: Fri Feb 12 20:39:28 2021 +0000 Use titlecase for template filenames. --- .gitlab/issue_templates/{bug.md => Bug.md} | 0 .gitlab/issue_templates/{feature.md => Feature.md} | 0 .gitlab/issue_templates/{proposal.md => Proposal.md} | 0 3 files changed, 0 insertions(+), 0 deletions(-) diff --git a/.gitlab/issue_templates/bug.md b/.gitlab/issue_templates/Bug.md similarity index 100% rename from .gitlab/issue_templates/bug.md rename to .gitlab/issue_templates/Bug.md diff --git a/.gitlab/issue_templates/feature.md b/.gitlab/issue_templates/Feature.md similarity index 100% rename from .gitlab/issue_templates/feature.md rename to .gitlab/issue_templates/Feature.md diff --git a/.gitlab/issue_templates/proposal.md b/.gitlab/issue_templates/Proposal.md similarity index 100% rename from .gitlab/issue_templates/proposal.md rename to .gitlab/issue_templates/Proposal.md

1 0

[tor/master] Remove Proposal.md.
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit 7f32ae0eb55382b9ef59a36713887b7a8927b794 Author: Alexander Færøy <ahf(a)torproject.org> Date: Fri Feb 12 20:39:48 2021 +0000 Remove Proposal.md. I don't think the Proposal.md fits well with the kind of tickets we see. Let's start with Features and Bug and expand upon it as we go. --- .gitlab/issue_templates/Proposal.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/.gitlab/issue_templates/Proposal.md b/.gitlab/issue_templates/Proposal.md deleted file mode 100644 index 990ca70a0a..0000000000 --- a/.gitlab/issue_templates/Proposal.md +++ /dev/null @@ -1,5 +0,0 @@ - - -### Proposal - -

1 0

[tor/master] Merge remote-tracking branch 'tor-gitlab/mr/305'
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit e8cce1058c06e8f1bec486c9e49bc2e81243b672 Merge: b2f025cb56 7f32ae0eb5 Author: George Kadianakis <desnacked(a)riseup.net> Date: Fri Feb 19 13:24:34 2021 +0200 Merge remote-tracking branch 'tor-gitlab/mr/305' .gitlab/issue_templates/Bug.md | 32 ++++++++++++++++++++++++++++++++ .gitlab/issue_templates/Feature.md | 9 +++++++++ 2 files changed, 41 insertions(+)

1 0

[torspec/master] Update path-spec.txt for CBT estimator fixes for #40168.
by asn＠torproject.org 19 Feb '21

19 Feb '21

commit df031d7bf33f68948b2c915d681b08ca59c1da37 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Tue Nov 3 16:28:52 2020 -0600 Update path-spec.txt for CBT estimator fixes for #40168. Also clarify and improve wording of the timeout calculation section. --- path-spec.txt | 202 ++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 117 insertions(+), 85 deletions(-) diff --git a/path-spec.txt b/path-spec.txt index 483e37d..c16ef50 100644 --- a/path-spec.txt +++ b/path-spec.txt @@ -361,112 +361,147 @@ of their choices. 2.4. Learning when to give up ("timeout") on circuit construction - Since version 0.2.2.8-alpha, Tor attempts to learn when to give up on - circuits based on network conditions. + Since version 0.2.2.8-alpha, Tor clients attempt to learn when to give + up on circuits based on network conditions. -2.4.1. Distribution choice and parameter estimation +2.4.1. Distribution choice Based on studies of build times, we found that the distribution of - circuit build times appears to be a Frechet distribution. However, - estimators and quantile functions of the Frechet distribution are - difficult to work with and slow to converge. So instead, since we - are only interested in the accuracy of the tail, we approximate - the tail of the distribution with a Pareto curve. + circuit build times appears to be a Frechet distribution (and a multi-modal + Frechet distribution, if more than one guard or bridge is used). However, + estimators and quantile functions of the Frechet distribution are difficult + to work with and slow to converge. So instead, since we are only interested + in the accuracy of the tail, clients approximate the tail of the multi-modal + distribution with a single Pareto curve. - We calculate the parameters for a Pareto distribution fitting the data - using the estimators in equation 4 from: - http://portal.acm.org/citation.cfm?id=1647962.1648139 +2.4.2. How much data to record + + From our observations, the minimum number of circuit build times for a + reasonable fit appears to be on the order of 100. However, to keep a + good fit over the long term, clients store 1000 most recent circuit build + times in a circular array. - This is: + The Tor client should build test circuits at a rate of one every 'cbttestfreq' + (10 seconds) until 'cbtmincircs' (100 circuits) are built, with a maximum of + 'cbtmaxopencircs' (default: 10) circuits open at once. This allows a fresh + Tor to have a CircuitBuildTimeout estimated within 30 minutes after install + or network change (see section 2.4.5 below). - alpha_m = s/(ln(U(X)/Xm^n)) + Timeouts are stored on disk in a histogram of 10ms bin width, the same + width used to calculate the Xm value above. This histogram must be shuffled + after being read from disk, to preserve a proper expiration of old values + after restart. - where s is the total number of completed circuits we have seen, and + Thus, some build time resolution is lost during restart. Implementations may + choose a different persistence mechanism than this histogram, but be aware + that build time binning is still needed for parameter estimation. - U(X) = x_max^u * Prod_s{x_i} +2.4.3. Parameter estimation - with x_i as our i-th completed circuit time, x_max as the longest - completed circuit build time we have yet observed, u as the - number of unobserved timeouts that have no exact value recorded, - and n as u+s, the total number of circuits that either timeout or - complete. + Once 'cbtmincircs' build times are recorded, Tor clients update the + distribution parameters and recompute the timeout every circuit completion + (though see section 2.4.5 for when to pause and reset timeout due to + network change or connectivity loss). - Using log laws, we compute this as the sum of logs to avoid - overflow and ln(1.0+epsilon) precision issues: + Tor clients calculate the parameters for a Pareto distribution fitting the + data using the maximum likelihood estimator. For derivation, see: + https://en.wikipedia.org/wiki/Pareto_distribution#Estimation_of_parameters - alpha_m = s/(u*ln(x_max) + Sum_s{ln(x_i)} - n*ln(Xm)) + Because build times are not a true Pareto distribution, we alter how Xm is + computed. In a max likelihood estimator, the mode of the distribution is + used directly as Xm. - This estimator is closely related to the parameters present in: - http://en.wikipedia.org/wiki/Pareto_distribution#Parameter_estimation - except they are adjusted to handle the fact that our samples are - right-censored at the timeout cutoff. + Instead of using the mode of discrete build times directly, Tor clients + compute the Xm parameter using the weighted average of the the midpoints + of the 'cbtnummodes' (10) most frequently occurring 10ms histogram bins. + (The use of 10 modes was found to minimize error from the selected + cbtquantile, with 10ms bins for quantiles 60-80, compared to many other + heuristics). - Additionally, because this is not a true Pareto distribution, we alter - how Xm is computed. The Xm parameter is computed as the midpoint of the most - frequently occurring 50ms histogram bin, until the point where 1000 - circuits are recorded. After this point, the weighted average of the top - 'cbtnummodes' (default: 3) midpoint modes is used as Xm. All times below - this value are counted as having the midpoint value of this weighted average - bin. + To avoid ln(1.0+epsilon) precision issues, use log laws to rewrite the + estimator for 'alpha' as the sum of logs followed by subtraction, rather + than multiplication and division: + + alpha = n/(Sum_n{ln(MAX(Xm, x_i))} - n*ln(Xm)) + + In this, n is the total number of build times that have completed, x_i is + the ith recorded build time, and Xm is the modes of x_i as above. + + All times below Xm are counted as having the Xm value via the MAX(), + because in Pareto estimators, Xm is supposed to be the lowest value. + However, since clients use mode averaging to estimatre Xm, there can be + values below our Xm. Effectively, the Pareto estimator then treats that + everything smaller than Xm happened at Xm. One can also see that if + clients did not do this, alpha could underflow to become negative, which + results in an exponential curve, not a Pareto probability distribution. The timeout itself is calculated by using the Pareto Quantile function (the inverted CDF) to give us the value on the CDF such that 80% of the mass - of the distribution is below the timeout value. + of the distribution is below the timeout value (parameter 'cbtquantile'). - Thus, we expect that the Tor client will accept the fastest 80% of - the total number of paths on the network. + The Pareto Quantile Function (inverse CDF) is: -2.4.2. How much data to record + F(q) = Xm/((1.0-q)^(1.0/alpha)) - From our observations, the minimum number of circuit build times for a - reasonable fit appears to be on the order of 100. However, to keep a - good fit over the long term, we store 1000 most recent circuit build times - in a circular array. + Thus, clients obtain their circuit build timeout by computing: - The Tor client should build test circuits at a rate of one per - minute up until 100 circuits are built. This allows a fresh Tor to have - a CircuitBuildTimeout estimated within 1.5 hours after install, - upgrade, or network change (see below). + timeout_ms = F(0.8) # 'cbtquantile' == 0.8 - Timeouts are stored on disk in a histogram of 50ms bin width, the same - width used to calculate the Xm value above. This histogram must be shuffled - after being read from disk, to preserve a proper expiration of old values - after restart. + With this, we expect that the Tor client will accept the fastest 80% of the + total number of paths on the network. + + Clients obtain the circuit close time to completely abandon circuits as: + + close_ms = F(0.99) # 'cbtclosequantile' == 0.99 + + To avoid waiting an unreasonably long period of time for circuits that + simply have relays that are down, Tor clients cap timeout_ms at the max + build time actually observed so far, and cap close_ms at twice this max, + but at least 60 seconds: -2.4.3. How to record timeouts + timeout_ms = MIN(timeout_ms, max_observed_timeout) + close_ms = MAX(MIN(close_ms, 2*max_observed_timeout), 'cbtinitialtimeout') - Circuits that pass the timeout threshold should be allowed to continue - building until a time corresponding to the point 'cbtclosequantile' - (default 95) on the Pareto curve, or 60 seconds, whichever is greater. +2.4.4. How to record timeouts - The actual completion times for these circuits should be recorded. - Implementations should completely abandon a circuit and record a value - as an 'unknown' timeout if the total build time exceeds this threshold. + Pareto estimators begin to lose their accuracy if the tail is omitted. + Hence, Tor clients actually calculate two timeouts: a usage timeout, and a + close timeout. - The reason for this is that right-censored pareto estimators begin to lose - their accuracy if more than approximately 5% of the values are censored. - Since we wish to set the cutoff at 20%, we must allow circuits to continue - building past this cutoff point up to the 95th percentile. + Circuits that pass the usage timeout are marked as measurement circuits, + and are allowed to continue to build until the close timeout corresponding + to the point 'cbtclosequantile' (default 99) on the Pareto curve, or 60 + seconds, whichever is greater. -2.4.4. Detecting Changing Network Conditions + The actual completion times for these measurements circuits should be + recorded. - We attempt to detect both network connectivity loss and drastic + Implementations should completely abandon a circuit and ignore the circuit + if the total build time exceeds the close threshold. Such closed circuits + should be ignored, as this typically means one of the relays in the path is + offline. + +2.4.5. Detecting Changing Network Conditions + + Tor clients attempt to detect both network connectivity loss and drastic changes in the timeout characteristics. - We assume that we've had network connectivity loss if a circuit - times out and we've received no cells or TLS handshakes since that - circuit began. We then temporarily stop counting timeouts until - network activity resumes. + Clients assume that they have had network connectivity loss if a circuit + times out and have received no cells or TLS handshakes since that + circuit began. Clients then temporarily stop counting timeouts until + network activity resumes (ie: until a TLS handshake completes or a cell + arrives at the client). - To detect changing network conditions, we keep a history of - the timeout or non-timeout status of the past 20 circuits that - successfully completed at least one hop. If more than 90% of - these circuits timeout, we discard all buildtimes history, reset - the timeout to 60, and then begin recomputing the timeout. + To detect changing network conditions, clients keep a history of + the timeout or non-timeout status of the past 'cbtrecentcount' circuits + (20 circuits) that successfully completed at least one hop. If more than + 90% of these circuits timeout, the client discards all buildtimes history, + resets the timeout to 'cbtinitialtimeout' (60 seconds), and then begins + recomputing the timeout. - If the timeout was already 60 or higher, we double the timeout. + If the timeout was already 60 or higher, the client doubles the timeout. -2.4.5. Consensus parameters governing behavior +2.4.6. Consensus parameters governing behavior Clients that implement circuit build timeout learning should obey the following consensus parameters that govern behavior, in order to allow @@ -483,14 +518,13 @@ of their choices. emergency situations only. cbtnummodes - Default: 3 + Default: 10 Min: 1 Max: 20 Effect: This value governs how many modes to use in the weighted - average calculation of Pareto parameter Xm. A value of 3 introduces - some bias (2-5% of CDF) under ideal conditions, but allows for better - performance in the event that a client chooses guard nodes of radically - different performance characteristics. + average calculation of Pareto parameter Xm. Selecting Xm as the + average of multiple modes improves accuracy of the Pareto tail + for quantile cutoffs from 60-80% (see cbtquantile). cbtrecentcount Default: 20 @@ -522,7 +556,7 @@ of their choices. timeout value. It is a percent (10-99). cbtclosequantile - Default: 95 + Default: 99 Min: Value of cbtquantile parameter Max: 99 Effect: This is the position on the quantile curve to use to set the @@ -530,7 +564,7 @@ of their choices. percent (0-99). cbttestfreq - Default: 60 + Default: 10 Min: 1 Max: 2147483647 (INT32_MAX) Effect: Describes how often in seconds to build a test circuit to @@ -538,12 +572,10 @@ of their choices. have been recorded. cbtmintimeout - Default: 2000 - Min: 500 + Default: 10 + Min: 10 Max: 2147483647 (INT32_MAX) Effect: This is the minimum allowed timeout value in milliseconds. - The minimum is to prevent rounding to 0 (we only check once - per second). cbtinitialtimeout Default: 60000

1 0