May 2018 - tor-commits - lists.torproject.org

[torspec/master] Bandwidth-measurement file specification, as sent to tor-dev
by nickm＠torproject.org 14 May '18

14 May '18

commit 9465f9c0713b2185643a976ef14e0959d3885e80 Author: juga0 <juga(a)riseup.net> Date: Mon May 14 14:31:05 2018 -0400 Bandwidth-measurement file specification, as sent to tor-dev --- bandwidth-file-spec.txt | 412 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 412 insertions(+) diff --git a/bandwidth-file-spec.txt b/bandwidth-file-spec.txt new file mode 100644 index 0000000..0755329 --- /dev/null +++ b/bandwidth-file-spec.txt @@ -0,0 +1,412 @@ + Tor Bandwidth List Format + juga + teor + +1. Scope and preliminaries + + This document describes the format of Tor's Bandwidth List, + version 1.0.0, 1.1.0 and later. + It is new specification for the existing format 1.0.0. + Describes a new format 1.1.0, which is backwards compatible with + 1.0.0 parsers. + + Since Tor version 0.2.4.12-alpha the directory authorities use + the Bandwidth List file called "V3BandwidthsFile" generated by + Torflow [1]. The format is described in Torflow's README.spec.txt and + is considered to be version 1.0.0. + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL + NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + RFC 2119. + +1.2. Acknowledgements + + The original bandwidth generator (Torflow) and format was + created by mike. Teor suggested to write this specification while + contributing on pastly's new bandwidth generator implementation. + + This specification was revised after feedback from: + + Nick Mathewson (nickm) + Iain Learmonth (irl) + +1.3 Outline + + The Tor directory protocol (dir-spec.txt [3]) sections 3.4.1 + and 3.4.2, use the term bandwidth measurements, to refer to what + here is called Bandwidth List. + A Bandwidth List file contains information on relays' bandwidth + capacities and is produced by bandwidth generators, previously known + as bandwidth scanners. + +1.4. Format Versions + + 1.0.0 - The legacy fallback Bandwidth List format + + 1.1.0 - Adds KeyValue Lines to the Header List section, add KeyValues + to RelayLines and format versions. + + All Tor versions can consume format version 1.0.0. + All Tor versions can consume format version 1.1.0, + but they warn on additional header Lines. + [TODO: this might be fixed, and if it is fixed should be said which + version of Tor] + +2. Format details + + The Bandwidth List MUST contain the following sections: + - Header List (exactly once) + - Relays' Bandwidth List (zero or more times) + If it does not contain these sections, parsers SHOULD ignore the file. + +2.1. Definitions + + The following nonterminals are defined in Tor directory protocol + sections 1.2., 2.1.1., 2.1.3.: + + Int + SP (space) + NL (newline) + Keyword + ArgumentChar + nickname + hexdigest (a '$', followed by 40 hexadecimal characters + ([A-Fa-f0-9])) + + Nonterminal defined section 2 of version-spec.txt [4]: + + version_number + + We define the following nonterminals: + + Line ::= ArgumentChar* NL + RelayLine ::= KeyValue (SP KeyValue)* NL + KeyValue ::= Keyword "=" Value + Value ::= ArgumentCharValue+ + ArgumentCharValue ::= any printing ASCII character except NL and SP. + Terminator ::= "=====" + Timestamp ::= Int + Bandwidth ::= Int + MasterKey ::= a base64-encoded Ed25519 public key, with + padding characters omitted. + DateTime ::= "YYYY-MM-DDTHH:MM:SS", as in ISO 8601 + + Note that key_value and value are defined in Tor directory protocol + with different formats to KeyValue and Value here. + + All Lines in the file MUST be 510 characters or less, to allow for the + trailing newline and NULL characters. + The previous limit was 254 characters in Tor 0.2.6.2-alpha and + earlier. + The parser MAY ignore longer Lines. + [TODO: Change this restriction in 1.1.0 or later] + +2.2. Header List format + +Some header Lines MUST appear in specific positions, as documented +below. +All other Lines can appear in any order. +If a parser does not recognize any extra material in a header Line, +the Line MUST be ignored. +If a header Line does not conform to this format, the Line SHOULD be +ignored by parsers. + +It consists of: + + Timestamp NL + + [At start, exactly once.] + + The Unix Epoch time in seconds when the file was created. + It does not follow the KeyValue format for backwards + compatibility with version 1.0.0. + + "version=" version_number NL + + [In second position, zero or one time.] + + The specification document format version. + It uses semantic versioning [5]. + + This Line has been added in version 1.1.0 of this specification. + + Version 1.0.0 documents do not contain this Line, and the + version_number is considered to be "1.0.0". + + "software=" Value NL + + [Zero or one time.] + + The name of the software that created the document. + + This Line has been added in version 1.1.0 of this specification. + + Version 1.0.0 documents do not contain this Line, and the software + is considered to be "torflow". + + "software_version=" Value NL + + [Zero or one time.] + + The version of the software that created the document. + The version may be a version_number, a git commit, or some other + version scheme. + + This Line has been added in version 1.1.0 of this specification. + + "generator_started=" DateTime NL + + [Zero or one time.] + + The date and time timestamp in ISO 8601 format and UTC time zone + when the generator started. + + This Line has been added in version 1.1.0 of this specification. + + "earliest_bandwidth=" DateTime NL + + [Zero or one time.] + + The date and time timestamp in ISO 8601 format and UTC time zone + when the first relay bandwidth was obtained. + + This Line has been added in version 1.1.0 of this specification. + + KeyValue NL + + [Zero or more times.] + + There MUST NOT be multiple KeyValue header Lines with the same key. + If there are, the parser SHOULD choose an arbitrary Line. + + If a parser does not recognize a Keyword in a KeyValue Line, it + MUST be ignored. + + Future format versions may include additional KeyValue header Lines. + Additional header Lines will be accompanied by a minor version + increment. + + Implementations MAY add additional header Lines as needed. This + specification SHOULD be updated to avoid conflicting meanings for + the same header keys. + + Parsers MUST NOT rely on the order of these additional Lines. + + Additional header Lines MUST NOT use any keywords specified in the + relay measurements format. + If there are, the parser MAY ignore conflicting keywords. + + Terminator NL + + [Zero or one time.] + + The Header List section ends with this Terminator. + + In version 1.0.0, Header List ends when the first relay bandwidth + is found conforming to the next section. + Implementations of version 1.1.0 SHOULD include this Line. + +2.3. Relays' Bandwidth List format + +It consists of zero or more RelayLines with the relays' bandwidth +in arbitrary order. + +There MUST NOT be multiple KeyValue pairs with the same key in the same +RelayLine. +If there are, the parser SHOULD choose an arbitrary Value. + +There MUST NOT be multiple RelayLine per relay identity (node_id or +master_key_ed25519). +If there are, parsers SHOULD issue a warning and MAY choose an arbitrary +value or ignore both values. + +If a parser does not recognize any extra material in a RelayLine, +the extra material MUST be ignored. + +Each RelayLine MUST include the following KeyValue pairs: +In version 1.0.0, node_id MUST NOT be at the end of the Line. +In version 1.1.0, the KeyValue can be in any arbitrary order. +[TODO: list of Tor version that support it, when it's done] + + "node_id=" hexdigest + + [Exactly once.] + + The fingerprint for the relay's RSA identity key. + + "master_key_ed25519=" MasterKey + + [Zero or one time.] + + The relays's master Ed25519 key, base64 encoded, + without trailing "="s, to avoid ambiguity with KeyValue "=" + character. + + Implementations of version 1.1.0 SHOULD include both node_id and + master_key_ed25519. + Parsers SHOULD accept Lines that contain at least one of them. + + "bw=" Bandwidth + + [Exactly once.] + + The measured bandwidth of this relay. + + Tor accepts zero bandwidths, but they trigger bugs in older Tor + implementations. Therefore, implementations SHOULD NOT produce zero + bandwidths. Instead, they SHOULD use one as their minimum bandwidth. + If there are zero bandwidths, the parser MAY ignore them. + + Multiple measurements can be aggregated using an averaging scheme, + such as a mean, median, or decaying average. + + Torflow scales bandwidths to kilobytes per second. Other + implementations SHOULD use kilobytes per second for their initial + bandwidth scaling. + + If different implementations or configurations are used in votes for + the same network, their measurements MAY need further scaling. See + Appendix B for information about scaling, and one possible scaling + method. + + KeyValue + + [Zero or more times.] + + Future format versions may include additional KeyValue pairs on a + RelayLine. + Additional KeyValue pairs will be accompanied by a minor version + increment. + + Implementations MAY add additional relay KeyValue pairs as needed. + This specification SHOULD be updated to avoid conflicting meanings + for the same Keywords. + + Parsers MUST NOT rely on the order of these additional KeyValue + pairs. + + Additional KeyValue pairs MUST NOT use any keywords specified in the + header format. + If there are, the parser MAY ignore conflicting keywords. + +2.4. Implementation notes + +KeyValue pairs in RelayLines that current implementations generate. + +2.4.1. Simple Bandwidth Scanner + +Every RelayLine in sbws version 0.1.0 consists of: + + "node_id=" hexdigest SP + + As above. + + "bw=" Bandwidth SP + + As above. + + "nick=" nickname SP + + [Exactly once.] + + The relay nickname. + + "rtt=" Int SP + + [Exactly once.] + + The Round Trip Time in milliseconds to obtain 1 byte of data. + + "time=" DateTime NL + + [Exactly once.] + + The date and time timestamp in ISO 8601 format and UTC time zone + when the last bandwidth was obtained. + +2.4.2. Torflow + +Torflow RelayLines include node_id and bw, and other KeyValue pairs [2]. + +References: + +1. https://gitweb.torproject.org/torflow.git +2. https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/… +3. https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt +4. https://gitweb.torproject.org/torspec.git/tree/version-spec.txt +5. https://semver.org/ + +A. Sample data + +The following has not been obtained from any real measurement. + +A.1. Generated by Torflow + +This an example version 1.0.0 document: + +1523911758 +node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test measured_at=1523911725 updated_at=1523911725 pid_error=4.11374090719 pid_error_sum=4.11374090719 pid_bw=57136645 pid_delta=2.12168374577 circ_fail=0.2 scanner=/filepath +node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 measured_at=1523911623 updated_at=1523911623 pid_error=3.96703337994 pid_error_sum=3.96703337994 pid_bw=47422125 pid_delta=2.65469736988 circ_fail=0.0 scanner=/filepath + +A.2. Generated by sbws version 0.1.X +[TODO: this needs to be implemented when this spec is finished] + +1523911758 +version=1.1.0 +software=sbws +software_version=0.1.0 +generator_started=2018-05-08T16:13:25 +earliest_bandwidth=2018-05-08T16:13:26 +==== +node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ bw=760 nick=Test rtt=380 time=2018-05-08T16:13:26 +node_id=$96C15995F30895689291F455587BD94CA427B6FC master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I bw=189 nick=Test2 rtt=378 time=2018-05-08T16:13:36 + +B. Scaling bandwidths + +B.1. Scaling requirements + +Tor accepts zero bandwidths, but they trigger bugs in older Tor +implementations. Therefore, scaling methods SHOULD perform the +following checks: + * If the total bandwidth is zero, all relays should be given equal + bandwidths. + * If the scaled bandwidth is zero, it should be rounded up to one. + +Initial experiments indicate that scaling may not be needed for +torflow and sbws, because their measured bandwidths are similar +enough already. + +B.2. A linear scaling method + +If scaling is required, here is a simple linear bandwith scaling +method, which ensures that all bandwidth votes contain approximately +the same total bandwidth: + +1. Calculate the relay quota by dividing the total measured bandwidth + in all votes, by the number of relays with measured bandwidth + votes. In the public tor network, this is approximately 7500 as of + April 2018. The quota should be a consensus parameter, so it can be + adjusted for all generators on the network. + +2. Calculate a vote quota by multiplying the relay quota by the number + of relays this bandwidth authority has measured + bandwidths for. + +3. Calculate a scaling factor by dividing the vote quota by the + total unscaled measured bandwidth in this bandwidth + authority's upcoming vote. + +4. Multiply each unscaled measured bandwidth by the scaling + factor. + +Now, the total scaled bandwidth in the upcoming vote is +approximately equal to the quota. + +B.3. Quota changes + +If all generators are using scaling, the quota can be gradually +reduced or increased as needed. Smaller quotas decrease the size +of uncompressed consensuses, and may decrease the size of +consensus diffs and compressed consensuses. But if the relay +quota is too small, some relays may be over- or under-weighted.

1 0

[tor/master] Merge branch 'bug25903_v5_squashed'
by nickm＠torproject.org 14 May '18

14 May '18

commit a394a2dd86579495797432114f62ea3265ed5ed0 Merge: a9ef335c1 fd504587d Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 14 14:25:07 2018 -0400 Merge branch 'bug25903_v5_squashed' changes/ticket25903 | 6 ++ src/or/circuituse.c | 38 ++++++++ src/or/circuituse.h | 2 + src/or/connection_edge.c | 9 +- src/or/control.c | 15 ++- src/or/or.h | 24 ++++- src/or/relay.c | 45 ++++++++- src/or/relay.h | 4 + src/or/rendcommon.c | 6 ++ src/test/test_relaycell.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++ 10 files changed, 377 insertions(+), 11 deletions(-) diff --cc src/or/rendcommon.c index 3a7dfe28f,2a77b2ece..f3fa2f64d --- a/src/or/rendcommon.c +++ b/src/or/rendcommon.c @@@ -12,14 -12,10 +12,15 @@@ #include "or.h" #include "circuitbuild.h" + #include "circuituse.h" #include "config.h" #include "control.h" +#include "crypto_rand.h" +#include "crypto_util.h" +#include "hs_client.h" #include "hs_common.h" +#include "hs_intropoint.h" +#include "networkstatus.h" #include "rendclient.h" #include "rendcommon.h" #include "rendmid.h"

1 0

[tor/master] Bug 25903: Changes file.
by nickm＠torproject.org 14 May '18

14 May '18

commit 08bfb87d1b049d6454e03eb056c94d96e833e782 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Mon Apr 23 20:18:10 2018 +0000 Bug 25903: Changes file. --- changes/ticket25903 | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/changes/ticket25903 b/changes/ticket25903 new file mode 100644 index 000000000..8d358c8a8 --- /dev/null +++ b/changes/ticket25903 @@ -0,0 +1,6 @@ + o Minor features (control port): + - Introduce new fields to the CIRC_BW event. There are two new fields in + each of the read and written directions. The DELIVERED fields report the + total valid data on the circuit, as measured by the payload sizes of + verified and error-checked relay command cells. The OVERHEAD fields + report the total unused bytes in each of these cells. Closes ticket 25903.

1 0

[tor/master] Bug 25903: Report new CIRC_BW fields to control port.
by nickm＠torproject.org 14 May '18

14 May '18

commit 1268baee9e182afd764bc0e2bb68e205349f4376 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Mon Apr 23 20:14:48 2018 +0000 Bug 25903: Report new CIRC_BW fields to control port. --- src/or/control.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/src/or/control.c b/src/or/control.c index dda887218..ca2271b76 100644 --- a/src/or/control.c +++ b/src/or/control.c @@ -257,6 +257,8 @@ clear_circ_bw_fields(void) continue; ocirc = TO_ORIGIN_CIRCUIT(circ); ocirc->n_written_circ_bw = ocirc->n_read_circ_bw = 0; + ocirc->n_overhead_written_circ_bw = ocirc->n_overhead_read_circ_bw = 0; + ocirc->n_delivered_written_circ_bw = ocirc->n_delivered_read_circ_bw = 0; } SMARTLIST_FOREACH_END(circ); } @@ -5908,13 +5910,20 @@ control_event_circ_bandwidth_used(void) tor_gettimeofday(&now); format_iso_time_nospace_usec(tbuf, &now); send_control_event(EVENT_CIRC_BANDWIDTH_USED, - "650 CIRC_BW ID=%d READ=%lu WRITTEN=%lu " - "TIME=%s\r\n", + "650 CIRC_BW ID=%d READ=%lu WRITTEN=%lu TIME=%s " + "DELIVERED_READ=%lu OVERHEAD_READ=%lu " + "DELIVERED_WRITTEN=%lu OVERHEAD_WRITTEN=%lu\r\n", ocirc->global_identifier, (unsigned long)ocirc->n_read_circ_bw, (unsigned long)ocirc->n_written_circ_bw, - tbuf); + tbuf, + (unsigned long)ocirc->n_delivered_read_circ_bw, + (unsigned long)ocirc->n_overhead_read_circ_bw, + (unsigned long)ocirc->n_delivered_written_circ_bw, + (unsigned long)ocirc->n_overhead_written_circ_bw); ocirc->n_written_circ_bw = ocirc->n_read_circ_bw = 0; + ocirc->n_overhead_written_circ_bw = ocirc->n_overhead_read_circ_bw = 0; + ocirc->n_delivered_written_circ_bw = ocirc->n_delivered_read_circ_bw = 0; } SMARTLIST_FOREACH_END(circ);

1 0

[tor/master] Bug 25903: Perform accounting for new CIRC_BW fields.
by nickm＠torproject.org 14 May '18

14 May '18

commit e07e95edd35f1608ecba94b281818d27f63812e3 Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Mon Apr 23 20:06:34 2018 +0000 Bug 25903: Perform accounting for new CIRC_BW fields. Two new values in each direction. DELIVERED counts valid end-to-end circuit data that is accepted by our end and OVERHEAD counts the slack unused data in each of the relay command cells for those accepted cells. Control port changes are in the next commit. --- src/or/circuituse.c | 38 ++++++++++++++++++++++++++++++++++++++ src/or/circuituse.h | 2 ++ src/or/connection_edge.c | 9 ++++++++- src/or/or.h | 24 ++++++++++++++++++++++-- src/or/relay.c | 40 +++++++++++++++++++++++++++++++++++++++- src/or/rendcommon.c | 6 ++++++ 6 files changed, 115 insertions(+), 4 deletions(-) diff --git a/src/or/circuituse.c b/src/or/circuituse.c index 47e29c28d..a584e3eca 100644 --- a/src/or/circuituse.c +++ b/src/or/circuituse.c @@ -3083,3 +3083,41 @@ mark_circuit_unusable_for_new_conns(origin_circuit_t *circ) circ->unusable_for_new_conns = 1; } +/** + * Add relay_body_len and RELAY_PAYLOAD_SIZE-relay_body_len to + * the valid delivered written fields and the overhead field, + * respectively. + */ +void +circuit_sent_valid_data(origin_circuit_t *circ, uint16_t relay_body_len) +{ + if (!circ) return; + + tor_assert_nonfatal(relay_body_len <= RELAY_PAYLOAD_SIZE); + + circ->n_delivered_written_circ_bw = + tor_add_u32_nowrap(circ->n_delivered_written_circ_bw, relay_body_len); + circ->n_overhead_written_circ_bw = + tor_add_u32_nowrap(circ->n_overhead_written_circ_bw, + RELAY_PAYLOAD_SIZE-relay_body_len); +} + +/** + * Add relay_body_len and RELAY_PAYLOAD_SIZE-relay_body_len to + * the valid delivered read field and the overhead field, + * respectively. + */ +void +circuit_read_valid_data(origin_circuit_t *circ, uint16_t relay_body_len) +{ + if (!circ) return; + + tor_assert_nonfatal(relay_body_len <= RELAY_PAYLOAD_SIZE); + + circ->n_delivered_read_circ_bw = + tor_add_u32_nowrap(circ->n_delivered_read_circ_bw, relay_body_len); + circ->n_overhead_read_circ_bw = + tor_add_u32_nowrap(circ->n_overhead_read_circ_bw, + RELAY_PAYLOAD_SIZE-relay_body_len); +} + diff --git a/src/or/circuituse.h b/src/or/circuituse.h index 71c818b97..6458bd690 100644 --- a/src/or/circuituse.h +++ b/src/or/circuituse.h @@ -65,6 +65,8 @@ void mark_circuit_unusable_for_new_conns(origin_circuit_t *circ); int circuit_purpose_is_hidden_service(uint8_t); int circuit_should_use_vanguards(uint8_t); +void circuit_sent_valid_data(origin_circuit_t *circ, uint16_t relay_body_len); +void circuit_read_valid_data(origin_circuit_t *circ, uint16_t relay_body_len); #ifdef TOR_UNIT_TESTS /* Used only by circuituse.c and test_circuituse.c */ diff --git a/src/or/connection_edge.c b/src/or/connection_edge.c index 955f942c5..02f1680ee 100644 --- a/src/or/connection_edge.c +++ b/src/or/connection_edge.c @@ -3526,10 +3526,17 @@ connection_exit_begin_conn(cell_t *cell, circuit_t *circ) n_stream->deliver_window = STREAMWINDOW_START; if (circ->purpose == CIRCUIT_PURPOSE_S_REND_JOINED) { + int ret; tor_free(address); /* We handle this circuit and stream in this function for all supported * hidden service version. */ - return handle_hs_exit_conn(circ, n_stream); + ret = handle_hs_exit_conn(circ, n_stream); + + if (ret == 0) { + /* This was a valid cell. Count it as delivered + overhead. */ + circuit_read_valid_data(origin_circ, rh.length); + } + return ret; } tor_strlower(address); n_stream->base_.address = address; diff --git a/src/or/or.h b/src/or/or.h index e27f25197..fc72badb4 100644 --- a/src/or/or.h +++ b/src/or/or.h @@ -3250,16 +3250,36 @@ typedef struct origin_circuit_t { * associated with this circuit. */ edge_connection_t *p_streams; - /** Bytes read from any attached stream since last call to + /** Bytes read on this circuit since last call to * control_event_circ_bandwidth_used(). Only used if we're configured * to emit CIRC_BW events. */ uint32_t n_read_circ_bw; - /** Bytes written to any attached stream since last call to + /** Bytes written to on this circuit since last call to * control_event_circ_bandwidth_used(). Only used if we're configured * to emit CIRC_BW events. */ uint32_t n_written_circ_bw; + /** Total known-valid relay cell bytes since last call to + * control_event_circ_bandwidth_used(). Only used if we're configured + * to emit CIRC_BW events. */ + uint32_t n_delivered_read_circ_bw; + + /** Total written relay cell bytes since last call to + * control_event_circ_bandwidth_used(). Only used if we're configured + * to emit CIRC_BW events. */ + uint32_t n_delivered_written_circ_bw; + + /** Total overhead data in all known-valid relay data cells since last + * call to control_event_circ_bandwidth_used(). Only used if we're + * configured to emit CIRC_BW events. */ + uint32_t n_overhead_read_circ_bw; + + /** Total written overhead data in all relay data cells since last call to + * control_event_circ_bandwidth_used(). Only used if we're configured + * to emit CIRC_BW events. */ + uint32_t n_overhead_written_circ_bw; + /** Build state for this circuit. It includes the intended path * length, the chosen exit router, rendezvous information, etc. */ diff --git a/src/or/relay.c b/src/or/relay.c index 8c248e6d9..05f71efea 100644 --- a/src/or/relay.c +++ b/src/or/relay.c @@ -611,6 +611,10 @@ relay_send_command_from_edge_,(streamid_t stream_id, circuit_t *circ, tor_free(commands); smartlist_free(commands_list); } + + /* Let's assume we're well-behaved: Anything that we decide to send is + * valid, delivered data. */ + circuit_sent_valid_data(origin_circ, rh.length); } if (circuit_package_relay_cell(&cell, circ, cell_direction, cpath_layer, @@ -740,6 +744,9 @@ connection_ap_process_end_not_open( } } + /* This end cell is now valid. */ + circuit_read_valid_data(circ, rh->length); + if (rh->length == 0) { reason = END_STREAM_REASON_MISC; } @@ -1230,6 +1237,12 @@ connection_edge_process_resolved_cell(edge_connection_t *conn, } } + /* This is valid data at this point. Count it */ + if (conn->on_circuit && CIRCUIT_IS_ORIGIN(conn->on_circuit)) { + circuit_read_valid_data(TO_ORIGIN_CIRCUIT(conn->on_circuit), + rh->length); + } + connection_ap_handshake_socks_got_resolved_cell(entry_conn, errcode, resolved_addresses); @@ -1326,6 +1339,9 @@ connection_edge_process_relay_cell_not_open( entry_conn->chosen_exit_name, ttl); remap_event_helper(entry_conn, &addr); + + /* This is valid data at this point. Count it */ + circuit_read_valid_data(TO_ORIGIN_CIRCUIT(circ), rh->length); } circuit_log_path(LOG_INFO,LD_APP,TO_ORIGIN_CIRCUIT(circ)); /* don't send a socks reply to transparent conns */ @@ -1496,7 +1512,6 @@ connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, circ->dirreq_id = ++next_id; TO_OR_CIRCUIT(circ)->p_chan->dirreq_id = circ->dirreq_id; } - return connection_exit_begin_conn(cell, circ); case RELAY_COMMAND_DATA: ++stats_n_data_cells_received; @@ -1532,6 +1547,10 @@ connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, "(relay data) conn deliver_window below 0. Killing."); return -END_CIRC_REASON_TORPROTOCOL; } + /* Total all valid application bytes delivered */ + if (CIRCUIT_IS_ORIGIN(circ)) { + circuit_read_valid_data(TO_ORIGIN_CIRCUIT(circ), rh.length); + } stats_n_data_bytes_received += rh.length; connection_buf_add((char*)(cell->payload + RELAY_HEADER_SIZE), @@ -1584,6 +1603,11 @@ connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, /* only mark it if not already marked. it's possible to * get the 'end' right around when the client hangs up on us. */ connection_mark_and_flush(TO_CONN(conn)); + + /* Total all valid application bytes delivered */ + if (CIRCUIT_IS_ORIGIN(circ)) { + circuit_read_valid_data(TO_ORIGIN_CIRCUIT(circ), rh.length); + } } return 0; case RELAY_COMMAND_EXTEND: @@ -1649,6 +1673,10 @@ connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, log_info(domain,"circuit_send_next_onion_skin() failed."); return reason; } + /* Total all valid bytes delivered. */ + if (CIRCUIT_IS_ORIGIN(circ)) { + circuit_read_valid_data(TO_ORIGIN_CIRCUIT(circ), rh.length); + } return 0; case RELAY_COMMAND_TRUNCATE: if (layer_hint) { @@ -1714,6 +1742,16 @@ connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, log_debug(LD_APP,"circ-level sendme at origin, packagewindow %d.", layer_hint->package_window); circuit_resume_edge_reading(circ, layer_hint); + + /* We count circuit-level sendme's as valid delivered data because + * they are rate limited. Note that we cannot count stream + * sendme's because the other end could send as many as they like. + */ + if (CIRCUIT_IS_ORIGIN(circ)) { + circuit_read_valid_data(TO_ORIGIN_CIRCUIT(circ), + rh.length); + } + } else { if (circ->package_window + CIRCWINDOW_INCREMENT > CIRCWINDOW_START_MAX) { diff --git a/src/or/rendcommon.c b/src/or/rendcommon.c index 230da4be5..2a77b2ece 100644 --- a/src/or/rendcommon.c +++ b/src/or/rendcommon.c @@ -12,6 +12,7 @@ #include "or.h" #include "circuitbuild.h" +#include "circuituse.h" #include "config.h" #include "control.h" #include "hs_common.h" @@ -807,6 +808,11 @@ rend_process_relay_cell(circuit_t *circ, const crypt_path_t *layer_hint, tor_fragile_assert(); } + if (r == 0 && origin_circ) { + /* This was a valid cell. Count it as delivered + overhead. */ + circuit_read_valid_data(origin_circ, length); + } + if (r == -2) log_info(LD_PROTOCOL, "Dropping cell (type %d) for wrong circuit type.", command);

1 0

[tor/master] Bug 25903: Tests
by nickm＠torproject.org 14 May '18

14 May '18

commit fd504587d56d4062bccb0d815f4fd925fb8eb33c Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Wed May 9 04:00:59 2018 +0000 Bug 25903: Tests --- src/or/relay.c | 5 +- src/or/relay.h | 4 + src/test/test_relaycell.c | 239 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 244 insertions(+), 4 deletions(-) diff --git a/src/or/relay.c b/src/or/relay.c index 05f71efea..095d48fa5 100644 --- a/src/or/relay.c +++ b/src/or/relay.c @@ -83,9 +83,6 @@ static edge_connection_t *relay_lookup_conn(circuit_t *circ, cell_t *cell, cell_direction_t cell_direction, crypt_path_t *layer_hint); -static int connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, - edge_connection_t *conn, - crypt_path_t *layer_hint); static void circuit_consider_sending_sendme(circuit_t *circ, crypt_path_t *layer_hint); static void circuit_resume_edge_reading(circuit_t *circ, @@ -1412,7 +1409,7 @@ connection_edge_process_relay_cell_not_open( * * Return -reason if you want to warn and tear down the circuit, else 0. */ -static int +STATIC int connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, edge_connection_t *conn, crypt_path_t *layer_hint) diff --git a/src/or/relay.h b/src/or/relay.h index f304af684..ce0969b46 100644 --- a/src/or/relay.h +++ b/src/or/relay.h @@ -114,6 +114,10 @@ STATIC packed_cell_t *packed_cell_new(void); STATIC packed_cell_t *cell_queue_pop(cell_queue_t *queue); STATIC destroy_cell_t *destroy_cell_queue_pop(destroy_cell_queue_t *queue); STATIC int cell_queues_check_size(void); +STATIC int connection_edge_process_relay_cell(cell_t *cell, circuit_t *circ, + edge_connection_t *conn, + crypt_path_t *layer_hint); + #endif /* defined(RELAY_PRIVATE) */ #endif /* !defined(TOR_RELAY_H) */ diff --git a/src/test/test_relaycell.c b/src/test/test_relaycell.c index eea1f5dc8..1db520273 100644 --- a/src/test/test_relaycell.c +++ b/src/test/test_relaycell.c @@ -4,9 +4,14 @@ /* Unit tests for handling different kinds of relay cell */ #define RELAY_PRIVATE +#define CIRCUITLIST_PRIVATE #include "or.h" +#include "main.h" #include "config.h" #include "connection.h" +#include "crypto.h" +#include "circuitbuild.h" +#include "circuitlist.h" #include "connection_edge.h" #include "relay.h" #include "test.h" @@ -20,6 +25,11 @@ static uint8_t srm_answer[512]; static int srm_ttl; static time_t srm_expires; +void connection_free_minimal(connection_t*); +int connected_cell_format_payload(uint8_t *payload_out, + const tor_addr_t *addr, + uint32_t ttl); + /* Mock replacement for connection_ap_hannshake_socks_resolved() */ static void socks_resolved_mock(entry_connection_t *conn, @@ -60,6 +70,234 @@ mark_unattached_mock(entry_connection_t *conn, int endreason, (void) file; } +/* Helper: Return a newly allocated and initialized origin circuit with + * purpose and flags. A default HS identifier is set to an ed25519 + * authentication key for introduction point. */ +static origin_circuit_t * +helper_create_origin_circuit(int purpose, int flags) +{ + origin_circuit_t *circ = NULL; + + circ = origin_circuit_init(purpose, flags); + tor_assert(circ); + circ->cpath = tor_malloc_zero(sizeof(crypt_path_t)); + circ->cpath->magic = CRYPT_PATH_MAGIC; + circ->cpath->state = CPATH_STATE_OPEN; + circ->cpath->package_window = circuit_initial_package_window(); + circ->cpath->deliver_window = CIRCWINDOW_START; + circ->cpath->prev = circ->cpath; + /* Create a default HS identifier. */ + circ->hs_ident = tor_malloc_zero(sizeof(hs_ident_circuit_t)); + + return circ; +} + +static void +mock_connection_mark_unattached_ap_(entry_connection_t *conn, int endreason, + int line, const char *file) +{ + (void) line; + (void) file; + conn->edge_.end_reason = endreason; +} + +static void +mock_mark_for_close(connection_t *conn, + int line, const char *file) +{ + (void)line; + (void)file; + + conn->marked_for_close = 1; + return; +} + +static void +mock_start_reading(connection_t *conn) +{ + (void)conn; + return; +} + +static void +test_circbw_relay(void *arg) +{ + cell_t cell; + relay_header_t rh; + tor_addr_t addr; + edge_connection_t *edgeconn; + entry_connection_t *entryconn; + origin_circuit_t *circ; + int delivered = 0; + int overhead = 0; + + (void)arg; + +#define PACK_CELL(id, cmd, body_s) do { \ + memset(&cell, 0, sizeof(cell)); \ + memset(&rh, 0, sizeof(rh)); \ + memcpy(cell.payload+RELAY_HEADER_SIZE, (body_s), sizeof((body_s))-1); \ + rh.length = sizeof((body_s))-1; \ + rh.command = (cmd); \ + rh.stream_id = (id); \ + relay_header_pack((uint8_t*)&cell.payload, &rh); \ + } while (0) +#define ASSERT_COUNTED_BW() do { \ + tt_int_op(circ->n_delivered_read_circ_bw, OP_EQ, delivered+rh.length); \ + tt_int_op(circ->n_overhead_read_circ_bw, OP_EQ, \ + overhead+RELAY_PAYLOAD_SIZE-rh.length); \ + delivered = circ->n_delivered_read_circ_bw; \ + overhead = circ->n_overhead_read_circ_bw; \ + } while (0) +#define ASSERT_UNCOUNTED_BW() do { \ + tt_int_op(circ->n_delivered_read_circ_bw, OP_EQ, delivered); \ + tt_int_op(circ->n_overhead_read_circ_bw, OP_EQ, overhead); \ + } while (0) + + MOCK(connection_mark_unattached_ap_, mock_connection_mark_unattached_ap_); + MOCK(connection_start_reading, mock_start_reading); + MOCK(connection_mark_for_close_internal_, mock_mark_for_close); + + entryconn = entry_connection_new(CONN_TYPE_AP, AF_INET); + edgeconn = ENTRY_TO_EDGE_CONN(entryconn); + edgeconn->base_.state = AP_CONN_STATE_CONNECT_WAIT; + edgeconn->deliver_window = 1000; + circ = helper_create_origin_circuit(CIRCUIT_PURPOSE_C_GENERAL, 0); + edgeconn->cpath_layer = circ->cpath; + circ->cpath->state = CPATH_STATE_AWAITING_KEYS; + circ->cpath->deliver_window = 1000; + + /* Stream id 0: Not counted */ + PACK_CELL(0, RELAY_COMMAND_END, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Stream id 1: Counted */ + PACK_CELL(1, RELAY_COMMAND_END, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_COUNTED_BW(); + + /* Properly formatted connect cell: counted */ + PACK_CELL(1, RELAY_COMMAND_CONNECTED, "Data1234"); + tor_addr_parse(&addr, "30.40.50.60"); + rh.length = connected_cell_format_payload(cell.payload+RELAY_HEADER_SIZE, + &addr, 1024); + relay_header_pack((uint8_t*)&cell.payload, &rh); \ + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_COUNTED_BW(); + + /* Properly formatted resolved cell in correct state: counted */ + edgeconn->base_.state = AP_CONN_STATE_RESOLVE_WAIT; + entryconn->socks_request->command = SOCKS_COMMAND_RESOLVE; + edgeconn->on_circuit = TO_CIRCUIT(circ); + PACK_CELL(1, RELAY_COMMAND_RESOLVED, + "\x04\x04\x12\x00\x00\x01\x00\x00\x02\x00"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_COUNTED_BW(); + + edgeconn->base_.state = AP_CONN_STATE_OPEN; + entryconn->socks_request->has_finished = 1; + + /* Connected cell after open: not counted */ + PACK_CELL(1, RELAY_COMMAND_CONNECTED, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Resolved cell after open: not counted */ + PACK_CELL(1, RELAY_COMMAND_RESOLVED, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Drop cell: not counted */ + PACK_CELL(1, RELAY_COMMAND_DROP, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Data cell on stream 0: not counted */ + PACK_CELL(0, RELAY_COMMAND_DATA, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Data cell on open connection: counted */ + ENTRY_TO_CONN(entryconn)->marked_for_close = 0; + PACK_CELL(1, RELAY_COMMAND_DATA, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_COUNTED_BW(); + + /* Sendme on stream: not counted */ + ENTRY_TO_CONN(entryconn)->outbuf_flushlen = 0; + PACK_CELL(1, RELAY_COMMAND_SENDME, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Sendme on circuit with full window: not counted */ + PACK_CELL(0, RELAY_COMMAND_SENDME, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Sendme on circuit with non-full window: counted */ + PACK_CELL(0, RELAY_COMMAND_SENDME, "Data1234"); + circ->cpath->package_window = 900; + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_COUNTED_BW(); + + /* End cell on non-closed connection: counted */ + PACK_CELL(1, RELAY_COMMAND_END, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), edgeconn, + circ->cpath); + ASSERT_COUNTED_BW(); + + /* End cell on connection that already got one: not counted */ + PACK_CELL(1, RELAY_COMMAND_END, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), NULL, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Invalid extended cell: not counted */ + PACK_CELL(1, RELAY_COMMAND_EXTENDED2, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), NULL, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Invalid extended cell: not counted */ + PACK_CELL(1, RELAY_COMMAND_EXTENDED, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), NULL, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* Invalid HS cell: not counted */ + PACK_CELL(1, RELAY_COMMAND_ESTABLISH_INTRO, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), NULL, + circ->cpath); + ASSERT_UNCOUNTED_BW(); + + /* "Valid" HS cell in expected state: counted */ + TO_CIRCUIT(circ)->purpose = CIRCUIT_PURPOSE_C_ESTABLISH_REND; + PACK_CELL(1, RELAY_COMMAND_RENDEZVOUS_ESTABLISHED, "Data1234"); + connection_edge_process_relay_cell(&cell, TO_CIRCUIT(circ), NULL, + circ->cpath); + ASSERT_COUNTED_BW(); + + done: + UNMOCK(connection_start_reading); + UNMOCK(connection_mark_unattached_ap_); + UNMOCK(connection_mark_for_close_internal_); + circuit_free_(TO_CIRCUIT(circ)); + connection_free_minimal(ENTRY_TO_CONN(entryconn)); +} + /* Tests for connection_edge_process_resolved_cell(). The point of ..process_resolved_cell() is to handle an incoming cell @@ -244,6 +482,7 @@ test_relaycell_resolved(void *arg) struct testcase_t relaycell_tests[] = { { "resolved", test_relaycell_resolved, TT_FORK, NULL, NULL }, + { "circbw", test_circbw_relay, TT_FORK, NULL, NULL }, END_OF_TESTCASES };

1 0

[torspec/master] Merge remote-tracking branch 'mikeperry/bug25903_v3'
by nickm＠torproject.org 14 May '18

14 May '18

commit 436d08b49fb84aa62d7bc96013002a0c27534bbb Merge: 1f7657e fbb38ec Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 14 14:26:06 2018 -0400 Merge remote-tracking branch 'mikeperry/bug25903_v3' control-spec.txt | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)

1 0

[torspec/master] Document new CIRC_BW fields.
by nickm＠torproject.org 14 May '18

14 May '18

commit fbb38ec724e595ffc3a8b7ec9a4eac26e7c9f6ab Author: Mike Perry <mikeperry-git(a)torproject.org> Date: Mon Apr 23 20:28:12 2018 +0000 Document new CIRC_BW fields. --- control-spec.txt | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/control-spec.txt b/control-spec.txt index e2f749e..fb693e5 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -2926,14 +2926,31 @@ The syntax is: "650" SP "CIRC_BW" SP "ID=" CircuitID SP "READ=" BytesRead SP - "WRITTEN=" BytesWritten SP "TIME=" Time CRLF + "WRITTEN=" BytesWritten SP "TIME=" Time SP + "DELIVERED_READ=" DeliveredBytesRead SP + "OVERHEAD_READ=" OverheadBytesRead SP + "DELIVERED_WRITTEN=" DeliveredBytesWritten CRLF + "OVERHEAD_WRITTEN=" OverheadBytesWritten SP BytesRead = 1*DIGIT BytesWritten = 1*DIGIT + OverheadBytesRead = 1*DIGIT + OverheadBytesWritten = 1*DIGIT + DeliveredBytesRead = 1*DIGIT + DeliveredBytesWritten = 1*DIGIT Time = ISOTime2Frac - BytesRead and BytesWritten are the number of bytes read and written by - all applications with streams attached to this circuit since the last - CIRC_BW event. + BytesRead and BytesWritten are the number of bytes read and written + on this circuit since the last CIRC_BW event. + + DeliveredBytesRead and DeliveredBytesWritten are the total length of + all valid relay cell payloads read and written on this circuit since + the last CIRC_BW event. These fields are provided only in Tor + 3.4.0-alpha and later. + + OverheadBytesRead and OverheadBytesWritten are the total length of + the unused overhead portion of valid relay cells read and written on + this circuit since the last CIRC_BW event. These fields are provided + only in Tor 0.3.4.0-alpha and later. The Time field is provided only in versions 0.3.2.1-alpha and later. It records when Tor created the bandwidth event.

1 0

[torspec/master] Document error handling for #25852
by nickm＠torproject.org 14 May '18

14 May '18

commit c5453a02aefa084868a5bda3972d20de4a5031b4 Author: rl1987 <rl1987(a)sdf.lonestar.org> Date: Sat May 5 16:05:40 2018 +0200 Document error handling for #25852 --- control-spec.txt | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/control-spec.txt b/control-spec.txt index e2f749e..510dc29 100644 --- a/control-spec.txt +++ b/control-spec.txt @@ -563,12 +563,15 @@ on the current values of ExitPolicyRejectPrivate and ExitPolicyRejectLocalInterfaces. These lines are based on the public addresses configured in the torrc and present on the relay's - interfaces. + interfaces. Will send 552 error if the server is not running as + onion router or if there's non-transient internal error. Will send + 551 on transient internal error. "exit-policy/ipv4" "exit-policy/ipv6" "exit-policy/full" -- This OR's exit policy, in IPv4-only, IPv6-only, or - all-entries flavors. + all-entries flavors. Handles errors in the same way as "exit-policy/ + reject-private/relay" does. "desc/id/<OR identity>" or "desc/name/<OR nickname>" -- the latest server descriptor for a given OR. (Note that modern Tor clients

1 0

[torspec/master] Merge remote-tracking branch 'rl1987/bug25852_take2'
by nickm＠torproject.org 14 May '18

14 May '18

commit 1f7657ebfc8dfee85670331086e48784bb8ef5c0 Merge: 2058ddd c5453a0 Author: Nick Mathewson <nickm(a)torproject.org> Date: Mon May 14 14:15:40 2018 -0400 Merge remote-tracking branch 'rl1987/bug25852_take2' control-spec.txt | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-)

1 0