November 2018 - tor-commits - lists.torproject.org

[torspec/master] Correct bandwidth variable in step 3
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 35479626bfa8d13244521e2312f032e04700fd29 Author: teor <teor(a)torproject.org> Date: Wed Nov 7 17:42:59 2018 +0000 Correct bandwidth variable in step 3 Co-Authored-By: juga0 <juga(a)riseup.net> --- bandwidth-file-spec.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 7073539..120789a 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -567,7 +567,7 @@ would be included in the next bandwidth authority's upcoming vote. In pseudocode: r_filt_i = bw_filt_i / bw_avg_filt - r_strm_i = bw_avg_strm / bw_avg_strm + r_strm_i = bw_i / bw_avg_strm 4. Calculate the final ratio for each relay: The final ratio is the larger between the filtered bandwidth and the

1 0

[torspec/master] Merge remote-tracking branch 'tor-github/pr/37'
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 78fdde420a9a8bf2d6584cca6a4c557973743485 Merge: d41a32a a9138a0 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon Nov 26 16:52:40 2018 -0500 Merge remote-tracking branch 'tor-github/pr/37' bandwidth-file-spec.txt | 66 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)

1 0

[torspec/master] Format to maximum 72 characters per line
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit a9138a07b9bde7287ffddebbdf75f5a98029497c Author: juga0 <juga(a)riseup.net> Date: Wed Nov 7 18:08:53 2018 +0000 Format to maximum 72 characters per line --- bandwidth-file-spec.txt | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 87bdb7e..014dbe0 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -527,16 +527,16 @@ B.4. Torflow aggreation Torflow implements two methods to compute the bandwidth values from the (stream) bandwidth measurements: with and without PID control feedback. -The method described here is without PID control (see Torflow specification, -section 2.2). +The method described here is without PID control (see Torflow +specification, section 2.2). In the following sections, the relays' measured bandwidth refer to the ones that this bandwidth authority has measured for the relays that would be included in the next bandwidth authority's upcoming vote. 1. Calculate the filtered bandwidth for each relay: - - choose the relay's measurements (`bw_j`) that are equal or greater than the - mean of the measurements for this relay + - choose the relay's measurements (`bw_j`) that are equal or greater + than the mean of the measurements for this relay - calculate the mean of those measurements In pseudocode: @@ -546,12 +546,12 @@ would be included in the next bandwidth authority's upcoming vote. 2. Calculate network averages: - calculate the filtered average by dividing the sum of all the relays' filtered bandwidth by the number of relays that have been - measured (`n`), ie, calculate the mean average of the relays' filtered - bandwidth. + measured (`n`), ie, calculate the mean average of the relays' + filtered bandwidth. - calculate the stream average by dividing the sum of all the relays' filtered bandwidth by the number of relays that have been - measured (`n`), ie, calculate the mean average or the relays' measured - bandwidth. + measured (`n`), ie, calculate the mean average or the relays' + measured bandwidth. In pseudocode: @@ -559,10 +559,10 @@ would be included in the next bandwidth authority's upcoming vote. bw_avg_strm = bw_i / n 3. Calculate ratios for each relay: - - calculate the filtered ratio by dividing each relay filtered bandwidth - by the filtered average - - calculate the stream ratio by dividing each relay measured bandwidth - by the stream average + - calculate the filtered ratio by dividing each relay filtered + bandwidth by the filtered average + - calculate the stream ratio by dividing each relay measured + bandwidth by the stream average In pseudocode: @@ -578,12 +578,13 @@ would be included in the next bandwidth authority's upcoming vote. r_i = max(r_filt_i, r_strm_i) 5. Calculate the scaled bandwidth for each relay: - The most recent descriptor observed bandwidth (`bw_obs_i`) is multiplied by the ratio + The most recent descriptor observed bandwidth (`bw_obs_i`) is + multiplied by the ratio In pseudocode: bw_new_i = r_i * bw_obs_i - <<In this way, the resulting network status consensus bandwidth values - are effectively re-weighted proportional to how much faster the node - was as compared to the rest of the network.>> + <<In this way, the resulting network status consensus bandwidth + values are effectively re-weighted proportional to how much faster + the node was as compared to the rest of the network.>>

1 0

[torspec/master] Include bw_obs_i variable in step 5
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 15d2c00685bc83deaa2422ab79771d2204e989c8 Author: teor <teor(a)torproject.org> Date: Wed Nov 7 17:44:18 2018 +0000 Include bw_obs_i variable in step 5 Co-Authored-By: juga0 <juga(a)riseup.net> --- bandwidth-file-spec.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 120789a..50e7294 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -578,7 +578,7 @@ would be included in the next bandwidth authority's upcoming vote. r_i = max(r_filt_i, r_strm_i) 5. Calculate the scaled bandwidth for each relay: - The most recent descriptor observed bandwidth is multiplied by the ratio + The most recent descriptor observed bandwidth (`bw_obs_i`) is multiplied by the ratio In pseudocode:

1 0

[torspec/master] Correct n_filt by n in step 2
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 45da870809210ba30330e5cffd876b83f443c76d Author: juga0 <juga(a)riseup.net> Date: Wed Nov 7 18:00:32 2018 +0000 Correct n_filt by n in step 2 The number of relays does not change for a relay filtered bandwdith. --- bandwidth-file-spec.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 50e7294..87bdb7e 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -546,7 +546,7 @@ would be included in the next bandwidth authority's upcoming vote. 2. Calculate network averages: - calculate the filtered average by dividing the sum of all the relays' filtered bandwidth by the number of relays that have been - measured (`n_filt`), ie, calculate the mean average of the relays' filtered + measured (`n`), ie, calculate the mean average of the relays' filtered bandwidth. - calculate the stream average by dividing the sum of all the relays' filtered bandwidth by the number of relays that have been @@ -555,7 +555,7 @@ would be included in the next bandwidth authority's upcoming vote. In pseudocode: - bw_avg_filt_ = bw_filt_i / n_filt + bw_avg_filt_ = bw_filt_i / n bw_avg_strm = bw_i / n 3. Calculate ratios for each relay:

1 0

[torspec/master] Add KeyValues in sbws section
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 41bd88c03d3e71789771fde0029e62f11f696b20 Author: juga0 <juga(a)riseup.net> Date: Fri Oct 26 12:52:46 2018 +0000 Add KeyValues in sbws section --- bandwidth-file-spec.txt | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index cfa2ab9..107c58a 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -495,6 +495,36 @@ Every RelayLine in sbws version 0.1.0 consists of: The number of times that the bandwidth measurements for this relay failed because of other reasons. + "bw_mean=" Int NL + + [Zero or one time.] + + The measured bandwidth mean for this relay. + + "bw_median=" Int NL + + [Zero or one time.] + + The measured bandwidth median for this relay. + + "desc_bw_average=" Int NL + + [Zero or one time.] + + The descriptor average bandwidth for this relay. + + "desc_bw_obs_last=" Int NL + + [Zero or one time.] + + The last descriptor observed bandwidth for this relay. + + "desc_bw_obs_mean=" Int NL + + [Zero or one time.] + + The descriptor observed bandwidth mean for this relay. + 2.4.2.2. Torflow Torflow RelayLines include node_id and bw, and other KeyValue pairs [2].

1 0

[torspec/master] Include definition of minimum_percent_eligible_relays
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 3c69441436f176c55a919928b825664c65f5b128 Author: teor <teor(a)torproject.org> Date: Wed Nov 7 13:50:42 2018 +0000 Include definition of minimum_percent_eligible_relays Change made by teor. Co-Authored-By: juga0 <juga(a)riseup.net> --- bandwidth-file-spec.txt | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 910b904..c57b480 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -266,6 +266,14 @@ It consists of: [Zero or one time.] + The percentage of relays in the consensus that SHOULD be + included in every generated bandwidth file. If there are not + enough eligible relays, the bandwidth file SHOULD contain a + header, but no relays. + + The minimum percentage is 60% in Torflow, so sbws uses + 60% as the default. + Implementations SHOULD include this KeyValue when "minimum_percent_eligible_relays" is minor than "percent_eligible_relays" and MAY not include any RelayLine.

1 0

[torspec/master] Rephrase percent_eligible_relays and add calculation
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 964937e145fbc6de901fa949b383a3b53b5d62f2 Author: teor <teor(a)torproject.org> Date: Wed Nov 7 13:47:31 2018 +0000 Rephrase percent_eligible_relays and add calculation As suggested by teor. Co-Authored-By: juga0 <juga(a)riseup.net> --- bandwidth-file-spec.txt | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index bcccd7a..910b904 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -212,7 +212,11 @@ It consists of: [Zero or one time.] - The minimum percentage of eligible measured relays with respect + The number of eligible relays, as a percentage of the number + of relays in the consensus. + + This line SHOULD be equal to: + (number_eligible_relays * 100.0) / number_consensus_relays to the number of relays in the consensus to include in this file. Implementations SHOULD include this KeyValue when

1 0

[torspec/master] Add new header KeyValues
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit a955a35f05cb37bd152e9ab59ff2d93af43b2eba Author: juga0 <juga(a)riseup.net> Date: Fri Oct 26 12:52:23 2018 +0000 Add new header KeyValues --- bandwidth-file-spec.txt | 60 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 9a3f63f..cfa2ab9 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -208,6 +208,66 @@ It consists of: This Line has been added in version 1.1.0 of this specification. + "percent_eligible_relays=" Int NL + + [Zero or one time.] + + The minimum percentage of eligible measured relays with respect + to the number of relays in the consensus to include in this file. + + Implementations SHOULD include this KeyValue when + "minimum_percent_eligible_relays" is minor than + "percent_eligible_relays" and MAY not include any RelayLine. + + This Line has been added in version 1.2.0 of this specification. + + "minimum_number_eligible_relays=" Int NL + + [Zero or one time.] + + The minimum number of relays to include in the bandwidth file. + + Implementations SHOULD include this KeyValue when + "minimum_percent_eligible_relays" is minor than + "percent_eligible_relays" and MAY not include any RelayLine. + + This Line has been added in version 1.2.0 of this specification. + + "number_consensus_relays=" Int NL + + [Zero or one time.] + + The number of relays in the consensus. + + Implementations SHOULD include this KeyValue when + "minimum_percent_eligible_relays" is minor than + "percent_eligible_relays" and MAY not include any RelayLine. + + This Line has been added in version 1.2.0 of this specification. + + "number_eligible_relays=" Int NL + + [Zero or one time.] + + The number of relays that has been measured and are eligible to + include in the bandwidth file. + + Implementations SHOULD include this KeyValue when + "minimum_percent_eligible_relays" is minor than + "percent_eligible_relays" and MAY not include any RelayLine. + + This Line has been added in version 1.2.0 of this specification. + + "minimum_percent_eligible_relays=" Int NL + + [Zero or one time.] + + Implementations SHOULD include this KeyValue when + "minimum_percent_eligible_relays" is minor than + "percent_eligible_relays" and MAY not include any RelayLine. + + This Line has been added in version 1.2.0 of this specification. + KeyValue NL [Zero or more times.]

1 0

[torspec/master] Add sbws example generating 1.2.X version
by nickm＠torproject.org 26 Nov '18

26 Nov '18

commit 4d7114ddb266c0fa3bf9807f2a4050367b7151d3 Author: juga0 <juga(a)riseup.net> Date: Fri Oct 26 13:07:10 2018 +0000 Add sbws example generating 1.2.X version --- bandwidth-file-spec.txt | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt index 107c58a..bcccd7a 100644 --- a/bandwidth-file-spec.txt +++ b/bandwidth-file-spec.txt @@ -550,7 +550,6 @@ node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test measured_at=1 node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 measured_at=1523911623 updated_at=1523911623 pid_error=3.96703337994 pid_error_sum=3.96703337994 pid_bw=47422125 pid_delta=2.65469736988 circ_fail=0.0 scanner=/filepath A.2. Generated by sbws version 0.1.X -[TODO: this needs to be implemented when this spec is finished] 1523911758 version=1.1.0 @@ -564,6 +563,36 @@ earliest_bandwidth=2018-04-16T15:13:26 bw=380 error_circ=0 error_misc=0 error_stream=1 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ nick=Test node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 rtt=380 success=1 time=2018-05-08T16:13:26 bw=189 error_circ=0 error_misc=0 error_stream=0 master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I nick=Test2 node_id=$96C15995F30895689291F455587BD94CA427B6FC rtt=378 success=1 time=2018-05-08T16:13:36 +A.3. Generated by sbws version 0.2.X + +1523911758 +version=1.1.0 +software=sbws +software_version=0.1.0 +latest_bandwidth=2018-04-16T20:49:18 +file_created=2018-04-16T21:49:18 +generator_started=2018-04-16T15:13:25 +earliest_bandwidth=2018-04-16T15:13:26 +==== +bw=38000 bw_mean=1127824 bw_median=1180062 desc_avg_bw=1073741824 desc_obs_bw_last=17230879 desc_obs_bw_mean=14732306 error_circ=0 error_misc=0 error_stream=1 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ nick=Test node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 rtt=380 success=1 time=2018-05-08T16:13:26 +bw=1 bw_mean=199162 bw_median=185675 desc_avg_bw=409600 desc_obs_bw_last=836165 desc_obs_bw_mean=858030 error_circ=0 error_misc=0 error_stream=0 master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I nick=Test2 node_id=$96C15995F30895689291F455587BD94CA427B6FC rtt=378 success=1 time=2018-05-08T16:13:36 + +When there are not enough eligible measured relays: + +1540496079 +version=1.1.0 +earliest_bandwidth=2018-10-20T19:35:52 +file_created=2018-10-25T19:35:03 +generator_started=2018-10-25T11:42:56 +latest_bandwidth=2018-10-25T19:34:39 +num_measured_relays=2960 +num_net_relays=6436 +num_target_relays=3862 +perc_measured_relays=46 +perc_measured_targed=60 +software=sbws +software_version=0.8.1-dev0 + B. Scaling bandwidths B.1. Scaling requirements

1 0