- tor-dev - lists.torproject.org

Bine - Go library for using and embedding Tor into applications
by Chad Retz 21 May '18

21 May '18

I posted it on /r/TOR but figured I'd post it here too: https://github.com/cretz/bine I built it because I have plans to use it in another project. In general it does what Stem does. I did incorporate V3 onion service generation in there and am happy to say it publishes much faster. Also, I have the option to statically compile Tor using the fairly new embedding API (i.e. tor_api.h). I've only tested it on Windows (see https://github.com/cretz/tor-static) but I figure that's the hardest of them to get working anyways. Thoughts/questions welcome. Chad

2 1

Tor Bandwidth List Format specification
by juga 21 May '18

21 May '18

Hi, after nick, irl and teor reviewed the last version i sent [1], i paste below a new version of the specification versions 1.1.0 and 1.0.0. It's the same version as commit https://github.com/juga0/torspec/commit/c7cdfd4fcb4b5623e1407e2bec38e9fdf7b…. The main question that came up was whether we should create a backwards incompatible specification version 2.0.0. Since right now it's faster to implement the version 1.1.0 of this specification, and assuming that we can add the specification and the code later, i'd propose to continue with 1.1.0. I've asked dirauths about their opinion about this. Thanks, juga [1] https://lists.torproject.org/pipermail/tor-dev/2018-May/013141.html [2] https://lists.torproject.org/pipermail/tor-dev/2018-May/013154.html ----------------------------------------------------------------------- Tor Bandwidth List Format juga teor 1. Scope and preliminaries This document describes the format of Tor's Bandwidth List, version 1.0.0, 1.1.0 and later. It is new specification for the existing format 1.0.0. Describes a new format 1.1.0, which is backwards compatible with 1.0.0 parsers. Since Tor version 0.2.4.12-alpha the directory authorities use the Bandwidth List file called "V3BandwidthsFile" generated by Torflow [1]. The format is described in Torflow's README.spec.txt and is considered to be version 1.0.0. The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 1.2. Acknowledgements The original bandwidth generator (Torflow) and format was created by mike. Teor suggested to write this specification while contributing on pastly's new bandwidth generator implementation. This specification was revised after feedback from: Nick Mathewson (nickm) Iain Learmonth (irl) 1.3 Outline The Tor directory protocol (dir-spec.txt [3]) sections 3.4.1 and 3.4.2, use the term bandwidth measurements, to refer to what here is called Bandwidth List. A Bandwidth List file contains information on relays' bandwidth capacities and is produced by bandwidth generators, previously known as bandwidth scanners. 1.4. Format Versions 1.0.0 - The legacy fallback Bandwidth List format 1.1.0 - Adds KeyValue Lines to the Header List section, add KeyValues to RelayLines and format versions. All Tor versions can consume format version 1.0.0. All Tor versions can consume format version 1.1.0, but they warn on additional header Lines. [TODO: this might be fixed, and if it is fixed should be said which version of Tor] 2. Format details The Bandwidth List MUST contain the following sections: - Header List (exactly once) - Relays' Bandwidth List (zero or more times) If it does not contain these sections, parsers SHOULD ignore the file. 2.1. Definitions The following nonterminals are defined in Tor directory protocol sections 1.2., 2.1.1., 2.1.3.: Int SP (space) NL (newline) Keyword ArgumentChar nickname hexdigest (a '$', followed by 40 hexadecimal characters ([A-Fa-f0-9])) Nonterminal defined section 2 of version-spec.txt [4]: version_number We define the following nonterminals: Line ::= ArgumentChar* NL RelayLine ::= KeyValue (SP KeyValue)* NL KeyValue ::= Keyword "=" Value Value ::= ArgumentCharValue+ ArgumentCharValue ::= any printing ASCII character except NL and SP. Terminator ::= "=====" Timestamp ::= Int Bandwidth ::= Int MasterKey ::= a base64-encoded Ed25519 public key, with padding characters omitted. DateTime ::= "YYYY-MM-DDTHH:MM:SS", as in ISO 8601 Note that key_value and value are defined in Tor directory protocol with different formats to KeyValue and Value here. All Lines in the file MUST be 510 characters or less, to allow for the trailing newline and NULL characters. The previous limit was 254 characters in Tor 0.2.6.2-alpha and earlier. The parser MAY ignore longer Lines. [TODO: Change this restriction in 1.1.0 or later] 2.2. Header List format Some header Lines MUST appear in specific positions, as documented below. All other Lines can appear in any order. If a parser does not recognize any extra material in a header Line, the Line MUST be ignored. If a header Line does not conform to this format, the Line SHOULD be ignored by parsers. It consists of: Timestamp NL [At start, exactly once.] The Unix Epoch time in seconds when the file was created. It does not follow the KeyValue format for backwards compatibility with version 1.0.0. "version=" version_number NL [In second position, zero or one time.] The specification document format version. It uses semantic versioning [5]. This Line has been added in version 1.1.0 of this specification. Version 1.0.0 documents do not contain this Line, and the version_number is considered to be "1.0.0". "software=" Value NL [Zero or one time.] The name of the software that created the document. This Line has been added in version 1.1.0 of this specification. Version 1.0.0 documents do not contain this Line, and the software is considered to be "torflow". "software_version=" Value NL [Zero or one time.] The version of the software that created the document. The version may be a version_number, a git commit, or some other version scheme. This Line has been added in version 1.1.0 of this specification. "generator_started=" DateTime NL [Zero or one time.] The date and time timestamp in ISO 8601 format and UTC time zone when the generator started. This Line has been added in version 1.1.0 of this specification. "earliest_bandwidth=" DateTime NL [Zero or one time.] The date and time timestamp in ISO 8601 format and UTC time zone when the first relay bandwidth was obtained. This Line has been added in version 1.1.0 of this specification. KeyValue NL [Zero or more times.] There MUST NOT be multiple KeyValue header Lines with the same key. If there are, the parser SHOULD choose an arbitrary Line. If a parser does not recognize a Keyword in a KeyValue Line, it MUST be ignored. Future format versions may include additional KeyValue header Lines. Additional header Lines will be accompanied by a minor version increment. Implementations MAY add additional header Lines as needed. This specification SHOULD be updated to avoid conflicting meanings for the same header keys. Parsers MUST NOT rely on the order of these additional Lines. Additional header Lines MUST NOT use any keywords specified in the relay measurements format. If there are, the parser MAY ignore conflicting keywords. Terminator NL [Zero or one time.] The Header List section ends with this Terminator. In version 1.0.0, Header List ends when the first relay bandwidth is found conforming to the next section. Implementations of version 1.1.0 SHOULD include this Line. 2.3. Relays' Bandwidth List format It consists of zero or more RelayLines with the relays' bandwidth in arbitrary order. There MUST NOT be multiple KeyValue pairs with the same key in the same RelayLine. If there are, the parser SHOULD choose an arbitrary Value. There MUST NOT be multiple RelayLine per relay identity (node_id or master_key_ed25519). If there are, parsers SHOULD issue a warning and MAY choose an arbitrary value or ignore both values. If a parser does not recognize any extra material in a RelayLine, the extra material MUST be ignored. Each RelayLine MUST include the following KeyValue pairs: In version 1.0.0, node_id MUST NOT be at the end of the Line. In version 1.1.0, the KeyValue can be in any arbitrary order. [TODO: list of Tor version that support it, when it's done] "node_id=" hexdigest [Exactly once.] The fingerprint for the relay's RSA identity key. "master_key_ed25519=" MasterKey [Zero or one time.] The relays's master Ed25519 key, base64 encoded, without trailing "="s, to avoid ambiguity with KeyValue "=" character. Implementations of version 1.1.0 SHOULD include both node_id and master_key_ed25519. Parsers SHOULD accept Lines that contain at least one of them. "bw=" Bandwidth [Exactly once.] The measured bandwidth of this relay. Tor accepts zero bandwidths, but they trigger bugs in older Tor implementations. Therefore, implementations SHOULD NOT produce zero bandwidths. Instead, they SHOULD use one as their minimum bandwidth. If there are zero bandwidths, the parser MAY ignore them. Multiple measurements can be aggregated using an averaging scheme, such as a mean, median, or decaying average. Torflow scales bandwidths to kilobytes per second. Other implementations SHOULD use kilobytes per second for their initial bandwidth scaling. If different implementations or configurations are used in votes for the same network, their measurements MAY need further scaling. See Appendix B for information about scaling, and one possible scaling method. KeyValue [Zero or more times.] Future format versions may include additional KeyValue pairs on a RelayLine. Additional KeyValue pairs will be accompanied by a minor version increment. Implementations MAY add additional relay KeyValue pairs as needed. This specification SHOULD be updated to avoid conflicting meanings for the same Keywords. Parsers MUST NOT rely on the order of these additional KeyValue pairs. Additional KeyValue pairs MUST NOT use any keywords specified in the header format. If there are, the parser MAY ignore conflicting keywords. 2.4. Implementation notes KeyValue pairs in RelayLines that current implementations generate. 2.4.1. Simple Bandwidth Scanner Every RelayLine in sbws version 0.1.0 consists of: "node_id=" hexdigest SP As above. "bw=" Bandwidth SP As above. "nick=" nickname SP [Exactly once.] The relay nickname. "rtt=" Int SP [Exactly once.] The Round Trip Time in milliseconds to obtain 1 byte of data. "time=" DateTime NL [Exactly once.] The date and time timestamp in ISO 8601 format and UTC time zone when the last bandwidth was obtained. 2.4.2. Torflow Torflow RelayLines include node_id and bw, and other KeyValue pairs [2]. References: 1. https://gitweb.torproject.org/torflow.git 2. https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/… 3. https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt 4. https://gitweb.torproject.org/torspec.git/tree/version-spec.txt 5. https://semver.org/ A. Sample data The following has not been obtained from any real measurement. A.1. Generated by Torflow This an example version 1.0.0 document: 1523911758 node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test measured_at=1523911725 updated_at=1523911725 pid_error=4.11374090719 pid_error_sum=4.11374090719 pid_bw=57136645 pid_delta=2.12168374577 circ_fail=0.2 scanner=/filepath node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 measured_at=1523911623 updated_at=1523911623 pid_error=3.96703337994 pid_error_sum=3.96703337994 pid_bw=47422125 pid_delta=2.65469736988 circ_fail=0.0 scanner=/filepath A.2. Generated by sbws version 0.1.X [TODO: this needs to be implemented when this spec is finished] 1523911758 version=1.1.0 software=sbws software_version=0.1.0 generator_started=2018-05-08T16:13:25 earliest_bandwidth=2018-05-08T16:13:26 ==== node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 master_key_ed25519=YaqV4vbvPYKucElk297eVdNArDz9HtIwUoIeo0+cVIpQ bw=760 nick=Test rtt=380 time=2018-05-08T16:13:26 node_id=$96C15995F30895689291F455587BD94CA427B6FC master_key_ed25519=a6a+dZadrQBtfSbmQkP7j2ardCmLnm5NJ4ZzkvDxbo0I bw=189 nick=Test2 rtt=378 time=2018-05-08T16:13:36 B. Scaling bandwidths B.1. Scaling requirements Tor accepts zero bandwidths, but they trigger bugs in older Tor implementations. Therefore, scaling methods SHOULD perform the following checks: * If the total bandwidth is zero, all relays should be given equal bandwidths. * If the scaled bandwidth is zero, it should be rounded up to one. Initial experiments indicate that scaling may not be needed for torflow and sbws, because their measured bandwidths are similar enough already. B.2. A linear scaling method If scaling is required, here is a simple linear bandwith scaling method, which ensures that all bandwidth votes contain approximately the same total bandwidth: 1. Calculate the relay quota by dividing the total measured bandwidth in all votes, by the number of relays with measured bandwidth votes. In the public tor network, this is approximately 7500 as of April 2018. The quota should be a consensus parameter, so it can be adjusted for all generators on the network. 2. Calculate a vote quota by multiplying the relay quota by the number of relays this bandwidth authority has measured bandwidths for. 3. Calculate a scaling factor by dividing the vote quota by the total unscaled measured bandwidth in this bandwidth authority's upcoming vote. 4. Multiply each unscaled measured bandwidth by the scaling factor. Now, the total scaled bandwidth in the upcoming vote is approximately equal to the quota. B.3. Quota changes If all generators are using scaling, the quota can be gradually reduced or increased as needed. Smaller quotas decrease the size of uncompressed consensuses, and may decrease the size of consensus diffs and compressed consensuses. But if the relay quota is too small, some relays may be over- or under-weighted.

2 3

Looking for an open project to contribute in TOR
by Saurav Malani 17 May '18

17 May '18

Hello Everyone, I am Saurav Malani, 3rd year computer science student at IIIT-Hyderabad, India. I applied for a project in TOR-SOP, but sadly my proposal was not selected. But, still I would like to contribute to TOR. So, is there any open project in which I can start contributing under the guidance of appropriate mentor. Thank you, Saurav Malani

2 1

BandwidthRate minimal values (was: Re: [tor-relays] Report on the FOSDEM relays meetup)
by nusenu 13 May '18

13 May '18

(from tor-relays) >> Content-wise, there were some pretty fruitful discussion, but we spent quite a bit >> of time on questions that recurr at other meetups too: >> >> - What's the most useful thing I can bring to the network? >> (TL;DR: Ideally, fast relays (if possible exits); if you can't contribute >> more than 1MB/s, make it a bridge) > > I like 1MByte/s (8MBit/s) as the "limit" between relay vs. bridge, currently we mention > 16MBit/s and 2MBit/s in the documentation: > > https://www.torproject.org/docs/faq.html.en#HowDoIDecide > https://trac.torproject.org/projects/tor/wiki/TorRelayGuide#RelayRequiremen… > > I'll go ahead and make 8 MBit/s the lower limit (and keep "It is recommended that a relay have at least 16 MBit/s (Mbps)") > unless there are strong opinions against it. Hi, to have all places consistent, are you open to increase the minimal values in the tor man page for relays to 8 MBit/s (and bridges to 1 MBit/s) or other greater than the current values? currently it says: BandwidthRate N [...] If you want to run a relay in the public network, this needs to be at the very least 75 KBytes for a relay (that is, 600 kbits) or 50 KBytes for a bridge (400 kbits) but of course, more is better; we recommend at least 250 KBytes (2 mbits) if possible. If you are open to it I'll submit a patch via trac. -- https://mastodon.social/@nusenu twitter: @nusenu_

1 1

Proposal #291 (two guards) IRC meeting Wed Apr 18, 17:00 UTC
by Mike Perry 09 May '18

09 May '18

Heyo. We're going to have a meeting to discuss Proposal 291. See this thread: https://lists.torproject.org/pipermail/tor-dev/2018-April/013053.html The meeting will be at 17:00 UTC, on Wednesday, April 18th, in #tor-meeting on irc.oftc.net. (That's 10:00 left coast, 12:00 middle coast, 13:00 right coast, and 19:00 in several socialist paradises that strangely do not have public water fountains.) https://www.timeanddate.com/worldclock/fixedtime.html?iso=20180415T1700 Things we need to decide: 1. Do we abandon Tor's path restrictions? 2. Do we use two guards? At the end of this meeting, we should commit to one or both of these things long-term. (Surprise twist: we're already doing #2!) Each of these choices is a nuanced thing. And just picking one or the other doesn't solve everything. I think it's best to think of them as a commitment to a plan over some timescale, based on the information we have available today. People who mos def should attend: George Kadianakis, Roger, Nick, Me People who probably maybe should attend: Aaron Johnson, Isis (and others concerned about guard fingerprinting), You? -- Mike Perry

3 11

Tor Messenger recent IRC re-topic.
by Beard CryptoWorld 07 May '18

07 May '18

Hello Everyone, We briefly talked about this idea of taking over Tor Messenger, a long with our own ideas and were asked to mail in some Documentation/Roadmaps. I'm asking in what form would you guys like to see this in? Basically: Would you like us to add a .txt as an attachment in a follow up email? Or would it be better to host the files on our site, and allow people to read without subject to downloading anything? Thank you, ~ Beard | https://twitter.com/beardlyness

2 2

Re: [tor-dev] Proposal: Tor bandwidth measurements document format
by juga 06 May '18

06 May '18

Hi, after teor's revision, second version pasted below. Changes can be seen: in https://github.com/juga0/torspec/commits/bandwidth-file-spec Best, juga ================================================================= Tor Bandwidth Measurements Document Format juga teor 1. Scope and preliminaries This document describes the format of Tor's bandwidth measurements document, version 1.0.0 and later. Since Tor version 0.2.4.12-alpha the directory authorities use the bandwidth measurements document called "V3BandwidthsFile" and produced by Torflow [1] (format described in README.spec.txt [2]). The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. 1.2. Acknowledgements The original bandwidth measurement scanner (Torflow) and format was created by mike. Teor suggested to write this specification while contributing on pastly's new bandwidth scanner implementation. This specification was revised after feedback from: XXX 1.3 Outline The bandwidth measurements mentioned in sections 3.4.1 and 3.4.2 of "Tor directory protocol" (dir-spec.txt) [3] are obtained by bandwidth authorities, which generate a file storing information on relays' measured bandwidth capacities. 1.4. Format Versions 1.0.0 - The legacy fallback bandwidth measurements document format 1.1.0 - Adds key_value lines to the header, format version, optional ones and section separator. 2. Format details Bandwidth measurements MUST contain the following sections: - Header (exactly once) - Relays measurements (zero or more times) 2.1. Definitions The following nonterminals are defined in dir-spec.txt, sections 1.2., 2.1.1., 2.1.3.: Int SP (space) NL (newline) Keyword ArgumentChar fingerprint (hexdigest) nickname Nonterminals defined in "Tor Directory List Format" (dir-list-spec.txt), section 2.2.1.: version_number We define the following nonterminals: value ::= ArgumentChar+ key_value ::= Keyword "=" value line ::= ArgumentChar* NL timestamp ::= Int bandwidth ::= Int relay_line ::= key_value (SP key_value)* NL 2.2. Header format Some header lines MUST appear in specific positions, as documented below. All other lines can appear in any order. There MUST NOT be multiple key_value header lines with the same key. It consists of: timestamp NL [At start, exactly once.] The Unix Epoch time in seconds when the file was created. "version=" version_number NL [In second position, zero or one time.] The specification document format version. It uses semantic versioning [5]. This line has been added in version 1.1.0 of this specification. Version 1.0.0 documents do not contain this line, and the version_number is considered to be "1.0.0". "software=" value NL [Zero or one time.] The name of the software that created the document. This line has been added in version 1.1.0 of this specification. Version 1.0.0 documents do not contain this line, and the software is considered to be "torflow". "software_version=" value NL [Zero or one time.] The version of the software that created the document. The version may be a version_number, a git commit, or some other version scheme. This line has been added in version 1.1.0 of this specification. "scanner_started=" timestamp NL [Zero or one time.] The Unix Epoch time in seconds when the scanner that generates the measurements document started. This line has been added in version 1.1.0 of this specification. "earliest_measurement=" timestamp NL [Zero or one time.] The Unix Epoch time in seconds when the first relay measurement was obtained. This line has been added in version 1.1.0 of this specification. key_value NL [Zero or more times.] Future format versions may include additional key_value header lines. Additional header lines will be accompanied by a minor version increment. Implementations MAY add additional header lines as needed. This specification SHOULD be updated to avoid conflicting meanings for the same header keys. Parsers MUST NOT rely on the order of these additional lines. Additional header lines MUST NOT use any keywords specified in the relay measurements format. If a header line does not conform to this format, the line SHOULD be ignored by parsers. NL [Zero or one time.] The header ends. This line has been added in version 1.1.0 of this specification. For version 1.0.0 documents, the header ends when the first relay measurement line is found conforming to the next section. 2.3. Relay measurements format It consists of zero or more relay_line with the measurement results of relays in arbitrary order. There can be at most one relay_line per relay identity (fingerprint). There MUST NOT be multiple key_value pairs with the same key in the same relay_line. Each relay_line MUST include the following key_value in arbitrary order: "node_id=" fingerprint [Exactly once.] The fingerprint of the relay being measured. "bw=" bandwidth [Exactly once.] The measured bandwidth of this relay. Tor accepts zero bandwidths, but they trigger bugs in older Tor implementations. Therefore, implementations SHOULD NOT produce zero bandwidths. Instead, they SHOULD use one as their minimum bandwidth. Multiple measurements can be aggregated using an averaging scheme, such as a mean, median, or decaying average. Torflow scales bandwidths to kilobytes per second. Other implementations SHOULD use kilobytes per second for their initial bandwidth scaling. If different implementations or configurations are used in votes for the same network, their measurements MAY need further scaling. See Appendix B for information about scaling, and one possible scaling method. key_value [Zero or more times.] Future format versions may include additional key_value pairs on a relay_line. Additional key_value pairs will be accompanied by a minor version increment. Implementations MAY add additional relay key_value pairs as needed. This specification SHOULD be updated to avoid conflicting meanings for the same relay keys. Parsers MUST NOT rely on the order of these additional key_value pairs. Additional key_value pairs MUST NOT use any keywords specified in the header format. If a relay line does not conform to this format, the line SHOULD be ignored by parsers. 2.4. Implementation notes 2.4.1. Simple Bandwidth Scanner Every relay measurement in sbws version 0.1.0 consists of: "node_id=" fingerprint SP As above. "bw=" bandwidth SP As above. "nick=" nickname SP [Exactly once.] The relay nickname. "rtt=" Int SP [Exactly once.] The Round Trip Time in milliseconds to obtain 1 byte of data. "time=" timestamp NL [Exactly once.] The Unix Epoch time in seconds when the last measurement was performed. 2.4.2. Torflow Torflow relay lines include node_id and bw, and other key_value pairs [2]. References: 1. https://gitweb.torproject.org/torflow.git 2. https://gitweb.torproject.org/torflow.git/tree/NetworkScanners/BwAuthority/… 3. https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt 4. https://metrics.torproject.org/onionoo.html#details 5. https://semver.org/ A. Sample data The following has not been obtained from any real measurement. A.1. Generated by Torflow This an example version 1.0.0 document: 1523911758 node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test measured_at=1523911725 updated_at=1523911725 pid_error=4.11374090719 pid_error_sum=4.11374090719 pid_bw=57136645 pid_delta=2.12168374577 circ_fail=0.2 scanner=/filepath node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 measured_at=1523911623 updated_at=1523911623 pid_error=3.96703337994 pid_error_sum=3.96703337994 pid_bw=47422125 pid_delta=2.65469736988 circ_fail=0.0 scanner=/filepath A.2. Generated by sbws version 0.1.0 1523911758 version=1.1.0 software=sbws software_version=0.1.0 scanner_started=1523911756 earliest_measurement=1523911757 node_id=$68A483E05A2ABDCA6DA5A3EF8DB5177638A27F80 bw=760 nick=Test rtt=380 time=1523911725 node_id=$96C15995F30895689291F455587BD94CA427B6FC bw=189 nick=Test2 rtt=378 time=1523911623 B. Scaling bandwidths B.1. Scaling requirements Tor accepts zero bandwidths, but they trigger bugs in older Tor implementations. Therefore, scaling methods SHOULD perform the following checks: * If the total bandwidth is zero, all relays should be given equal bandwidths. * If the scaled bandwidth is zero, it should be rounded up to one. Initial experiments indicate that scaling may not be needed for torflow and sbws, because their measured bandwidths are similar enough already. B.2. A linear scaling method If scaling is required, here is a simple linear bandwith scaling method, which ensures that all bandwidth votes contain approximately the same total bandwidth: 1. Calculate the relay quota by dividing the total measured bandwidth in all votes, by the number of relays with measured bandwidth votes. In the public tor network, this is approximately 7500 as of April 2018. The quota should be a consensus parameter, so it can be adjusted for all scanners on the network. 2. Calculate a vote quota by multiplying the relay quota by the number of relays this bandwidth authority has measured bandwidths for. 3. Calculate a scaling factor by dividing the vote quota by the total unscaled measured bandwidth in this bandwidth authority's upcoming vote. 4. Multiply each unscaled measured bandwidth by the scaling factor. Now, the total scaled bandwidth in the upcoming vote is approximately equal to the quota. B.3. Quota changes If all scanners are using scaling, the quota can be gradually reduced or increased as needed. Smaller quotas decrease the size of uncompressed consensuses, and may decrease the size of consensus diffs and compressed consensuses. But if the relay quota is too small, some relays may be over- or under-weighted.

5 15

Bandwidth Scanning Meeting Notes 3 May 2018
by Matt Traudt 03 May '18

03 May '18

Next meeting is 10 May 2018 at 1200 UTC for 30 minutes... maybe. While we should find a standard time, next week is a special case for pastly and teor. Notes are https://pad.riseup.net/p/ioYq89yZSx1t and copy/pasted below. --------------------------------------------------------------------- This pad: https://pad.riseup.net/p/ioYq89yZSx1t Meetbot log: http://meetbot.debian.net/tor-meeting/2018/tor-meeting.2018-05-03-09.29.html Last week https://lists.torproject.org/pipermail/tor-dev/2018-April/013108.html Next Milestone: 1st Status Update: May 25th All Milestones: https://trac.torproject.org/projects/tor/wiki/doc/gsoc ######################## Updates pastly: - Make significant strides towards the switch to HTTP/S - sbws launches Tor for itself - started tagging and icrementing and signing versions teor: - reviewed the bandwidth format spec - feedback on tor bwfile tests and bug fixes - feedback on sbws / tor launching - the sbws vs torflow averages in the testnet are stable and consistent - fixed a buffer read-past-the-end when the file can't be read in the bwfile parsing code juga: Last week: * sbws: * Worked on "Add what to include in a source distribution" (issues/132) * Discussed with pastly about "Include file where to write ``generate`` results in the config?" * Created "Fix version, prepare for future release" (issues/131): pastly started to tag versions, what is needed for packaging/distributing sbws (whatever the method will be) * Worked on "Include software and sofware_version headers" (issues/96), waiting to finish the spec so that we don't keep switch format * Worked on "Add useful information in sbws header lines" (issues/119): PR 130 waiting to finish spec * spec: * Worked with teor on the new version sent to @tor-dev * little-tor: * Worked on "Allow additional header lines" (#25960): also waiting for the specs * Started to write tests for the previous and "Create unit test for dirserv_read_measured_bandwidths" (#25947) Next week: * spec: wait for comments and change according to it * little-tor: * continue with #25960 (when spec ready) * finish #25947, tests for current code, additional header lines and possible refactor * sbws: work on 96, 119 when spec is ready ######################## Discussions -------------- sbws logo offer?? Is this something we care about? Decision: not really, but ux might like the help -------------- Meeting time 0930 UTC is terribly early for pastly, but he can do it in the name of collab. 1130 UTC would actually be easier for teor, but that's about as late as they can go For juga both times are fine ^ Decision: from 12:00 to 12:30 UTC Decision: 1200 UTC and only 30m long next week -------------- Bandwidth spec Bugfixes and incremental improvements, or a rewrite? Should i create a new version, include the changes or wait for more comments? Decision: make minor changes if there is time -------------- Bandwidth file parsing code in Tor unit tests are ok bugfixes are ok What this mean? ^ we can always do tests and bugfixes If we implement the format, we might have to change it -------------- Bandwidth file generation If we implement the format, we might have to change it pastly thinks he/we can easily handle whatever crazy thing you all come up with for the Tor side of code as long as it follows the general idea of one-line-per-relay, simple integers, and maybe a header with some metadata juga thinks there is not crazy thing, and parsing strings in python is way easier than c :)

1 0

connectivity failure for top 100 relays
by dawuud 02 May '18

02 May '18

Out of 9900 possible two hop tor circuits among the top 100 tor relays only 935 circuit builds have succeeded. This is way worse than the last time I sent a report 6 months ago during the Montreal tor dev meeting. Here's the scanner I use: https://github.com/david415/tor_partition_scanner (I was planning on improving this testing methodology in collaboration with Katharina Kohls but was unable to travel to Bochum University because of visa limitations. It was either go to tor-dev meeting or Bochum but not both.) Here's the gist of my simple testing methodology: https://gist.github.com/david415/9875821652018431dd6d6c4407bb90c0#file-dete… Here's exactly how I performed the scan to get those results: wget https://collector.torproject.org/recent/relay-descriptors/consensuses/2018-… ./helpers/query_fingerprints_from_consensus_file.py 2018-03-1 3-01-00-00-consensus > top100.relays detect_partitions.py --tor-control tcp:127.0.0.1:9051 --log-dir ./ --status-log ./status_log \ --relay-list top100.relays --secret secretTorEmpireOfRelays --partitions 1 --this-partition 0 \ --build-duration .25 --circuit-timeout 60 --log-chunk-size 1000 --max-concurrency 100 echo "select first_hop, second_hop from scan_log where status = 'failure';" | sqlite3 scan1.db | wc -l 8942 echo "select first_hop, second_hop from scan_log where status = 'timeout';" | sqlite3 scan1.db | wc -l 23 echo "select first_hop, second_hop from scan_log where status = 'success';" | sqlite3 scan1.db | wc -l 935

5 14

Tor Terminology + torspec
by Iain Learmonth 02 May '18

02 May '18

Hi All, Looking at the recent work on the Tor bandwidth measurements document format, I've noticed there are a few places where language can be ambiguous [0]. This morning, I noticed more ambiguity in some metrics tools [1]. We could do with a glossary. The problem though is not the lack of a glossary, but that we have at least 3 [2][3][4]. I've just been discussing this with juga in IRC. For the Metrics glossary, I think (but please correct me if I'm wrong) that the Metrics team would be happy to have our glossary's definitions match up with torspec. I also think it would be cool if we only add new terms to the Metrics glossary if they have a corresponding term in torspec's glossary. Would the torspec maintainers be happy to review and merge patches for new terms to facilitate that? The community glossary is perhaps more broad than the Metrics or torspec glossaries, and so having all those terms in torspec would probably not be a useful thing to do. Would it be agreeable with the community team that terms that are already defined in torspec should not be overloaded and that new terms shouldn't be defined in torspec if they would have conflicting meanings with terms defined in the community glossary? If you have other ideas, then please do also suggest them. Thanks, Iain. [0] https://lists.torproject.org/pipermail/tor-dev/2018-May/013145.html [1] https://lists.torproject.org/pipermail/tor-relays/2018-May/015132.html [2] https://gitweb.torproject.org/torspec.git/tree/glossary.txt [3] https://metrics.torproject.org/glossary.html [4] https://trac.torproject.org/projects/tor/wiki/doc/community/glossary

2 1

onion v2 deprecation plan?
by nusenu 30 Apr '18

30 Apr '18

Hi, even though you are probably years away from deprecating onion v2 services it is certainly good to have a clear plan. I'm asking because the sooner onion v2 are deprecated the sooner some people can stop worrying about malicious HSDirs. thanks, nusenu -- https://mastodon.social/@nusenu twitter: @nusenu_

8 18

TBB + Privoxy
by procmem 30 Apr '18

30 Apr '18

Hi. We are trying to get Tor Browser to work with privoxy to allow users to optionally connect to other networks including I2P, Zeronet... Here's what we tried: * Disabling network.proxy.no_proxies_on * Setting network.proxy.http_port to 8118 (privoxy’s port number) network.proxy.http 127.0.0.1 network.proxy.http_port 8118 network.proxy.ssl 127.0.0.1 network.proxy.ssl_port 8118 * We know a transition to unix sockets for localhost comms is planned but not currently enforced. We have workarounds for that. Can someone please tell me what's missing? NB This is a separate copy of Tor Browser to prevent fingerprinting problems with normal surfing in general. [0] https://forums.whonix.org/t/i2p-integration/4981

2 1

Closing old Trac Milestones
by teor 30 Apr '18

30 Apr '18

Hi, Tor's trac has a lot of milestones. Sometimes users select the wrong one, because there are so many. I've just closed some old milestones from 5+ years ago: Deliverable-Sep2010 Deliverable-December2010 Deliverable-Mar2011 Deliverable-May2011 Deliverable-Nov2011 Vidalia: 0.2.15 Vidalia: 0.3.x (Assuming they were no longer relevant, and belonged to Core Tor.) Here is a list of milestones we could close, sorted by team: HTTPSE Cr 2013.4.2x HTTPS-E 3.3 HTTPS-E 3.2.1 HTTPS-E 3.2.2 HTTPS-E 3.2.3 HTTPS-E 4.0dev8 HTTPS-E 4.0dev9 HTTPS-E 4.0dev15 Website Donations Enhancement Chronos: phase one Chronos: phase two TorBrowserBundle 2.2.x-stable TorBrowserBundle 2.3.x-stable Torbutton: 1.2.5 Torbutton: 1.3 Torbutton: 1.4 ExoneraTor 1.0.0 Metrics 1.0.0 Metrics 2.0.0 Onionoo 1.14.0 ttdnsd: 0.8 Source: https://trac.torproject.org/projects/tor/roadmap If any of the milestones on this list belong to you, and you'll never put any more tickets in them, please close them. (Or let me know, and I can close them.) Tickets in closed milestones will still appear in searches. We can always re-open a milestone if we decide we want to put more tickets in it. T -- teor PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B ricochet:ekmygaiu4rzgsk6n ------------------------------------------------------------------------

2 2

Re: [tor-dev] Standardizing the 'onion service' name
by Paul Syverson 26 Apr '18

26 Apr '18

Thanks teor for spelling things out in moderate detail and Steph for commenting. My tl;dr is to basically concur with everything they both said: The further from user-facing and the more embedded down in the codebase, variables, code comments, controller commands, etc. the less important to spend effort eliminating such vestiges from existing text. Going forward, certainly any code comments and e.g. any commands that won't break things should use current terminology. aloha, Paul On Thu, Apr 26, 2018 at 11:07:27AM -0400, Stephanie A. Whited wrote: > Hi! > > Thanks for adding me to the thread. > > > On 4/26/18 3:34 AM, teor wrote: > > Hi All, > > > > There seems to be some confusion in this thread about the > > current state of the hidden service to onion service name transition. > > > > So I'm going to describe the state of things as they are, then try to > > describe what would need to be done. > > > > I'd also appreciate feedback from Steph and others on our priorities for > > transitioning to the onion service name. I think we have been prioritising > > user-facing text. (The blog, website, Tor Browser, metrics, etc.) > Yes, user, funder, and press facing text has consistently been using > onion services at least since I've been on board. When I first joined, > the message to me was that this was changed a couple years prior but > changeover had been slow. I realize now a lot of folks didn't get that > message. > > We have already, with the help of Kat, updated hidden to onion services > where possible on the website. I think the exception to this is older > blog posts. > > I realize it is not as easy to change some of the other elements, like > the directory protocol, but I think it's important to make what changes > are possible. I think it'd have a very positive impact to see these > through. > > Being aligned will help us build trust with the press, new users, etc. > There are several reasons the name changed, if it is helpful to share > more about that lmk. > > > > > Is this a sensible way of prioritising things? > > > > On 26 Apr 2018, at 16:42, Paul Syverson <paul.syverson(a)nrl.navy.mil > > <mailto:paul.syverson@nrl.navy.mil>> wrote: > > > >>> Have OnionService aliases for controller commands, events, > > > > These are current called "hidden service" or an abbreviation. > > > > Tor could add an alias mechanism for controller commands, events, > > and fields, and use it to do the rename: > > > > https://trac.torproject.org/projects/tor/ticket/25922 > > <https://trac.torproject.org/projects/tor/ticket/25922#ticket> > > > > I don't think they are as high a priority as the torrc options and man > > page. > > > >>> descriptor > >>> fields > > > > These are currently called "hidden service", or an abbreviation. > > > > Descriptor fields are part of the directory specification and > > implementation, and they are highly technical. So I'm not sure we gain > > much from aliasing them or renaming them. > > > > Similar arguments might apply to other codebases: > > * Onionoo > > * stem > > * consensus health > > * Tor (network daemon) > > > > But the following user-facing applications should add documentation or > > change names, if they haven't already: > > * Relay Search / metrics website > > * uses HSDir for relay search, because that's what it's called in the > > directory protocol > > * uses "onion service" for statistics > > * Tor Browser > > * uses "onion site" > > * the Tor website > > * new tor blog posts > Website and new posts are covered. > > > >>> and anything else referencing 'HS' or 'HiddenService'. > > > > We considered adding OnionService* torrc option aliases for every > > HiddenService* option in 0.2.9. But we deferred that change because we > > ran out of time. > > > > All we need to do is add some new entries in the alias table, then do a > > search and replace in the tor man page: > > https://trac.torproject.org/projects/tor/ticket/17343 > > > >>> Speaking of which, how do we plan to replace abbreviations? Having an > >>> 'OSFETCH' or 'OS_CREATED' event doesn't exactly have the same ring as > >>> their HS counterparts. ;P > > > > That's a good question. > > > > OS conflicts with "operating system", so we could use: > > * Onion > > * OnSrv > > * no abbreviations > > Or any other colour you want to paint the bikeshed. > > > > To avoid an endless discussion, let's leave the decision to > > the people who write, review, and merge the code. > > > >>> Adjust all our docs to be consistent about the name. > >> > >> Right, anything v3 should be consistently calling these 'onion > >> services'. Variable names, etc. particularly those still in use in > >> code shared with v2, don't need to be changed. It is OK if such > >> vestiges of older usage remain in abbreviations, as long as the > >> description of them, e.g., in any new Tor proposal, describes them > >> with appropriately current terminology. > > > > torrc options, the tor man page, and the v3 onion service code > > mainly use "hidden service", and sometimes use "onion service". > > > > I don't see much value in changing the code. > > > > If we decide there is value in changing the torrc options and man page, > > we need to allocate a few days of work on the roadmap to make it > > happen. > > > >>> Renaming takes work. Lesson I learned from Nyx is that it works best > >>> if you draw a line in the sand and stand by it. With Nyx, version 2.0 > >>> is called Nyx (you won't find any docs saying otherwise) and version > >>> 1.x is the legacy 'arm' project. > >>> > >>> If I was in your shoes I'd opt for the same. Either prioritize the > >>> aliases and be firm that v3 are 'Onion Services' or abort the rename. > >>> Otherwise this will live in a confusing dual-named limbo land > >>> indefinitely. ;P > >> > >> I'm prettty sure that v3 being 'onion services' has been the official > >> Tor Project position since at least half a year. We wouldn't be > >> aborting the rename, because 'abort' would imply it is not > >> complete. Anything now not using the current name is not part of an > >> incomplete process, it is simply wrong and outdated. Steph correct me > >> if I am wrong about that. > >> > >> So I think you've answered your own question. Nothing in v3 should be > >> called 'hidden services'. > Right. In our communications, all are onion services, there are just > different versions and configurations possible. > > If I've missed something I can respond to more specifically in this > thread, please lmk. > -Steph > > > > That's not the current state of the tor network daemon v3 onion service > > code, specs, options, and man page. They use a mix of terminology > > (see above). > > > >> And anything new in code and documentation > >> should be called 'onion services'. If you want to think of v2 and > >> earlier as 'hidden services' for purposes of understanding legacy > >> component and variable names, e.g., HSDir that's fine. And as such, > >> variable names, etc. in code that continues to be used for both v2 and > >> v3 can can persist. But again, any new specs, documentation, etc. > >> should call them 'onion services'. > > > > We have gradually been using onion services in new documentation and > > specs, since "single onion services". But we haven't changed existing > > code and documentation. > > > >> This acceptance of existing v2 documentation calling them 'hidden > >> services' while refusing this for anything v3 is a little misleading > >> about when and why the naming transition happened, but its close > >> enough to serve as your line in the sand if you feel one is needed. > >> I actually argued the value of essentially such a line-in-the-sand > >> position to Steph a while ago. This doesn't preclude also calling v2 > >> and earlier 'onion services'. Indeed, it's not just OK but preferable, > >> for the above mentioned reasons. > > > > It looks like we are doing a gradual transition. I think we prioritised > > the things that are seen by the most users (Tor Browser, statistics, > > blog, website). > > > > We did not take a line in the sand position in the past, but we could > > adopt one for the remaining changes. We could decide on a particular > > Tor release (and releases of other codebases). But we need to schedule > > the work on the relevant roadmaps. > > > > And maybe there are some obscure technical things (code, comments, > > descriptor fields) that just aren't worth changing. > > > > T >

1 0

sbws meeting notes 26 April 2018
by Matt Traudt 26 Apr '18

26 Apr '18

Next meeting is 3 May 2018 at 0930 UTC. http://meetbot.debian.net/tor-meeting/2018/tor-meeting.2018-04-26-09.59.html https://pad.riseup.net/p/6292yqhUOuHC (copy/pasted below) Simple Bandwidth Scanner Meeting 25 April 2018 ============================================== This pad: https://pad.riseup.net/p/6292yqhUOuHC Previous week's notes: https://lists.torproject.org/pipermail/tor-dev/2018-April/013084.html Witty quote - Unknown Updates =========================================== pastly: last 7 days: - sbws is now open source https://github.com/pastly/simple-bw-scanner https://sbws.readthedocs.io/en/latest/ - fix some bad bugs in 'sbws cleanup' command while writing tests for it - integrate with Travis CI and automatically test against python 3.4, 3.5, and 3.6 - integrate with Read the Docs - gather intel on the environments our users will be running sbws in https://github.com/pastly/simple-bw-scanner/wiki/Information-regarding-sbws's-target-user-base - did some research regarding library support for various distros. incomplete - remvoe filelock dependency - brainstorm what I think a switch to http(s) should look like https://sbws.readthedocs.io/en/latest/proposals/001-switchtohttp.html next 7 days: - start implementing the switch to http(s) - verify sbws works with library versions that aren't the absolute newest misc notes section (not udpates) juga's tor tickets: https://trac.torproject.org/projects/tor/query?status=!closed&cc=%7Ejuga juga: Not sure which week was (since we changed from Monday to Thursday): - review, comment on debug option not being used #85 - add logging conf to log to file/system #87 - review Periodically test reachability of helpers #93 - create issue Include file where to write ``generate`` results in the config #91 suggest having options both in config and cli and cli overriding conf - report about problem with init and helpers #95 - create issue about adding software to header #96 - review using kilobytes #97, #98 Past week: - fix conf issues locally and in testnet after "megacommit" - more changes on the spec, after new teor review https://github.com/torproject/torspec/pull/4/commits/54ef4d4355d2b475466bf1… - create improve deployability #99 - create issue Debian package #101 - Work on issue software and sofware_version in v3bw files #112 - create issue Include coverage in travis #117 - create issue Include travis and coverage badges in README enhancement #118 Next week: - debug what tor does to solve https://github.com/torproject/torspec/pull/4/#issuecomment-384555243 - review the software/version ticket according to results from previous (https://github.com/pastly/simple-bw-scanner/issues/96) - review spec according to results from previous - add a default output file for generate that can be overriden by cli args (https://github.com/pastly/simple-bw-scanner/issues/91) - improve deployability, this should also help to update instances running in testnet (https://github.com/pastly/simple-bw-scanner/issues/99) - update version sbws running in testnet Things to communicate: - i put as date of sbws release 1.0.0 (milestone 1) 1st May, but i'd postpone to later so that we've more time testing and fixing last details - teor, can i make public the list of tasks you've proposed for SoP?, any place you would propose? - pastly: It's fine to close issues/PRs you think aren't apropiate for any reasons, but please say so before closing (#111, #87) - i stop attending net-team meetings until we need to disscuss something there that we don't discuss in this meeting List of tasks for 3 months proposed by T. (19th Apr) - implement features we wanted in torflow in sbws instead (1 week) - bwauth code needs to be smarter about failed circuits https,//trac.torproject.org/projects/tor/ticket/16559 - assign the 10-second client timeout as the time for failed circuits? - at least publish failure rates https,//trac.torproject.org/projects/tor/ticket/7281 - Understand how accurate the bandwidth authority estimates are https,//trac.torproject.org/projects/tor/ticket/7177 - integration tests (1 week) - a practice transition in the test network (1 week, but split up) - set up torflow - set up sbws - compare and switch - work little-t tor needs (3 weeks) - report version of bwscanners in votes https,//trac.torproject.org/projects/tor/ticket/3723 - stop relays reporting zero observed bandwidths https,//trac.torproject.org/projects/tor/ticket/24250 - relays should regularly do a larger bandwidth self-test https,//trac.torproject.org/projects/tor/ticket/22453 - bandwidth testing circuits should use guards sometimes https,//trac.torproject.org/projects/tor/ticket/19009 - report bandwidth more often on test networks https,//trac.torproject.org/projects/tor/ticket/17036 - report bandwidth less often on large, public relays https,//trac.torproject.org/projects/tor/ticket/24104 - non-essential tickets like https,//github.com/pastly/simple-bw-scanner/77 and https,//github.com/pastly/simple-bw-scanner/48 (2 weeks) - if time and motivation - Have sbws client launch tor itself https,//github.com/pastly/simple-bw-scanner/77 - Evaluate using [not threads, but twisted instead (probably)] https,//github.com/pastly/simple-bw-scanner/48 teor: last week: * made my test dirauth a trust pastly's and juga's dirauths * monitored the average bandwidth on the testnet using this script: https://pad.riseup.net/p/tor-vote-bw-average * opened a ticket for getting similar features into consensus-health: https://github.com/pastly/simple-bw-scanner/issues/120 this week: * busy with deadlines * eventually try to set up sbws on the testnet Discussion ========================================= ### Topic: Switch to HTTP(S) ### (Proposed) Topic: SoP - specific topics on this? pastly: where can I find your proposed schedule again? juga: mail pastly: finally found it - personally i'm not sure on how to organize the tasks we've said i'll do for SoP, since that's in a month. Either i do now other stuff not included there (doing so so far), or i start with those tasks and then update to others during SoP

1 0

Standardizing the 'onion service' name
by Damian Johnson 26 Apr '18

26 Apr '18

Hi all, teor suggested engaging the list with #25918 so here we go! Ticket #25918 has a couple goals... 1. Provide a tracking ticket for the rename effort. 2. Come to a consensus on if we should move forward with "onion service" or revert back to "hidden service". The limbo we've been in for months is confusing for our users and we should standardize on a name. Here's the ticket... ======================================== A recent post on tor-dev@ just got me thinking about the roadblocks we have for v2 deprecation. There's a couple I don't believe we're following in trac so lets fix that. For me the biggest is its name. Renaming takes work, and we attempted to rename hidden services in v3 without investing the time to make it happen. We should either fix that or revert to to the old name. To move forward we need to... Have OnionService aliases for controller commands, events, descriptor fields, and anything else referencing 'HS' or 'HiddenService'. Speaking of which, how do we plan to replace abbreviations? Having an 'OSFETCH' or 'OS_CREATED' event doesn't exactly have the same ring as their HS counterparts. ;P Adjust all our docs to be consistent about the name. Renaming takes work. Lesson I learned from Nyx is that it works best if you draw a line in the sand and stand by it. With Nyx, version 2.0 is called Nyx (you won't find any docs saying otherwise) and version 1.x is the legacy 'arm' project. If I was in your shoes I'd opt for the same. Either prioritize the aliases and be firm that v3 are 'Onion Services' or abort the rename. Otherwise this will live in a confusing dual-named limbo land indefinitely. ;P Cheers! -Damian PS. Stem and Nyx have stuck with the old name ("hidden services") and will continue to do so until tor's standardized this.

3 2

[TSoP] stem.client - introductions
by Dave Rolek 25 Apr '18

25 Apr '18

Hi everyone! I'm really excited to announce I'll be working on stem for the 2018 Tor Summer of Privacy! The news is still sinking in. This is going to be my first major chunk of contributing code back to the Tor community. :) In particular, I'll be working on expanding stem's support to act as a Tor client, aka stem.client, speaking with Tor's ORPort. You can see more about stem in general [1], and read concepts for stem.client [2], but at the end of this email I list my high-level plan for the summer project. There's definitely more to a Tor client than one can write in a summer, so my focus is on getting basic client functionality and on fetching v2/v3 Onion Service descriptors. The official project period is May 14th through August 21. My existing work on stem is available by querying through Trac [3] (easier for context) and browsing patches on GitHub [4]. I've been pretty active on IRC recently, but I wanted to broaden this. Anyone is welcome to reach out to me with questions/comments! On OFTC and Trac, I'm dmr. I plan to attend the network team meetings [5] (at least lurk) to get a sense of Tor's protocol development and to be around for any littlet-tor/stem syncing discussions. I'll also be sending updates biweekly to the tor-dev mailing list and writing "/me status: [...]" posts in #tor-dev. Lastly, my mentors for the project are atagar and teor. Really looking forward to this! Cheers! Dave -- My high-level plan is the following in this rough order: * continue prework with stem and the tor specs * get a stronger feel for / strengthen the CI testing for stem * better understand the ecosystem of projects that use stem * begin official coding by tackling minor stem.client improvements and implementing classes for a few more Cell types * continue developing stem.client.Relay and adding further testing using more of the Cell classes already defined * work on relay authentication * move onto fetching relay descriptors from the ORPort * work on the RELAY_EXTEN[sion] cells, to build multi-hop circuits * build upon this to allow actual exit traffic, including implementing RELAY_RESOLVE[D] * work on fetching v2 Onion Service descriptors, since I already have familiarity [6] with that spec, and v2 Onion Services probably will be around for a while yet * work on fetching v3 Onion Service descriptors * work on other stem.client improvements, if time allows * culminate the project with tutorials and a blog post introducing the new user-facing features of stem [1] https://stem.torproject.org/ [2] https://www.torproject.org/getinvolved/volunteer.html.en#pythonTorClient [3] https://trac.torproject.org/projects/tor/query?cc=~dmr&or&reporter=dmr&or&o… ^ a bit noisy, sorry; look at "Core Tor" [4] https://github.com/dmr-x/stem [5] https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam#Meeting… [6] https://github.com/dmr-x/talks

2 1

ahmia ~ summer of privacy
by Stelios Barberakis 24 Apr '18

24 Apr '18

Hello all, My name is Stelios and I am a CS student at Technical University of Crete. This summer I will be working on ahmia <https://ahmia.fi> project, during the "summer of privacy". This will be the first time to be engaged with the tor-ecosystem community as a developer, and I am really glad about it. The current plan includes some updates, enhancements of the codebase and the building/deployment process (including docker), as well as some new features like search filters, but we are looking to come up with new ideas for additional features, etc. Everyone is welcomed to contribute to the brainstorming, so feel free to ask anything you would like, either here or by joining #ahmia channel in OFTC if you feel so. I will do my best to contribute this summer, and hopefully beyond that. I will be sending bi-weekly reports to @tor-dev, so that everyone could be informed about the progress. Best Regards, Stelios -- PGP key: http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xBF6EA91B7CBE3998

2 2

HS desc replay protection and ed25519 malleability
by George Kadianakis 24 Apr '18

24 Apr '18

Hello Ian, isis, and other crypto people around here! Here is an intro: In HSv3 we've been using revision counters (integers++) to do HS desc replay protection, so that bad HSDirs cannot replay old descs to other HSDirs. We recently learned that this is a bad idea from a scalability prespective (multiple sites need to track rev counter...), and also it's needless complexity in the code (tor needs to cache the counters etc.). See ticket #25552 for more details: https://trac.torproject.org/projects/tor/ticket/25552 https://gitweb.torproject.org/torspec.git/tree/rend-spec-v3.txt#n1078 In #25552 we've been making plans to ditch the rev counters and replace them with a casual replay cache. (These replay caches also don't need to be big, since descriptors are only replayable for a day before the ephemeral blinded key changes, and the cache can be reset). Anyhow, now we've been playing the game of "which part of the desc should we use in the replay cache"? The latest plan here has been to use the ed25519 descriptor signature since it's something small, simple and necessarily changes with every fresh descriptor. And this is how we entered the ed25519 malleability scene. The basic question here is, can we use the ed25519 signature in our replay cache and consider it immutable by attackers without the private key? And should we use R, or S, or both? According to RFC8032: Ed25519 and Ed448 signatures are not malleable due to the check that decoded S is smaller than l. Without this check, one can add a multiple of l into a scalar part and still pass signature verification, resulting in malleable signatures. However, neither donna or ref10 include such a check explicitly IIUC. Instead they check whether (RS[63] & 224), which basically ensures that the high 3 bits of S are zeroed, which ensures S < 2^253. Is that equivalent to the RFC check? Because if I'm counting right, for most legit S values you can still add a single l as the attacker and get an S' = S + l < 2^253 equivalent signature (you can't add 2*l tho). So what's the state of ed25519 malleability? I know that after the Bitcoin incident, people have thought about this a lot, so I doubt we are in a broken state, but I just wanted to make sure that we will not mess something up. :) Thanks for the help! :)

5 7

Tor Messenger
by Milan Damjanovic 24 Apr '18

24 Apr '18

Hello, I would like to ask is there any online documentation about how is possible to transmit text messages via tor nodes? I would like to [try to] rewrite the TorChat Messenger which is written in Python 2.7 and wxPython to Python 3x and Qt. I cant find any documents how to transmit messages between two tor nodes so anything would be helpful... Cheers, Milan

2 2

Simple Bandwidth Scanner is now open source
by Matt Traudt 23 Apr '18

23 Apr '18

https://github.com/pastly/simple-bw-scanner Simple Bandwidth Scanner (sbws) aims to replace torflow[0], the bandwidth measuring software bwauths use today. For the most up to date documentation about how it works and why it exists, see the docs at readthedocs, especially the spec[2][3]. sbws's semantic version (currently 0.1.0) should not be considered meaningful until 1.0.0. There are major planned changes to how it works (the HTTPS milestone[1]), and likely a few unplanned changes too. I haven't put much thought into whether sbws should canonically live on gitweb.tpo or on GitHub other than I like GitHub's issue tracking, milestones, Travis CI integration, and readthedocs integration. So it might move ... or it might not. Cheers Matt [0]: https://gitweb.torproject.org/torflow.git [1]: https://github.com/pastly/simple-bw-scanner/milestone/3 [2]: https://sbws.readthedocs.io/en/latest/specification.html [3]: http://d7pxflytfsmz6uh3x7i2jxzzwea6nbpmtsz5tmfkcin5edapaig5vpyd.onion/speci…

1 0

Bandwidth scanner - Summer of Privacy
by juga 23 Apr '18

23 Apr '18

Hi, I will be working on the bandwidth scanner project during the Tor Summer of Privacy. This would include implementing features to the scanner code, writing tests and documentation, deployment in a test network, comparing results, modify little-tor code. The goal is to have bandwidth scanner code that is easier to maintain, understand, run and gives better measurements. There will be more details online and with reports. I've being already contributing on this project, for instance, writing a draft specification for the bandwidth measurements format [0] or reporting/fixing bugs on bwscanner code [1]. I'm MS in computer engineering, senior developer and have worked as sysadmin and researcher. Best regards, juga. GnuPG key fp: 2DA81D01455C3A0032198850F305447AF806D46B [0] https://lists.torproject.org/pipermail/tor-dev/2018-April/013071.html [1] https://github.com/TheTorProject/bwscanner

1 0

Notes from 19 April 2018 Simple Bandwidth Scanner Meeting
by Matt Traudt 19 Apr '18

19 Apr '18

Next meeting is scheduled 26 April 2018 at 2200 UTC in #tor-meeting, **BUT** we are probably going to look into finding a time that works better across three halves of the globe. Pad is here, and is copy/pasted below. https://pad.riseup.net/p/BenwGdgpz0uS Meeting IRC log http://meetbot.debian.net/tor-meeting/2018/tor-meeting.2018-04-19-21.59.log… Simple Bandwidth Scanner meeting 19 April 2018 ============================================== Previous weeks notes: https://lists.torproject.org/pipermail/tor-dev/2018-April/013059.html Updates -------------------------------------------------------- pastly: - Started running an authority in the test net. Don't think it's working yet and/or people have started trusting it - Switched to standard python logging module - Added periodic reachability tests for helper relays - (Helped) add FAQ and glosasry to docs - Learned: no part of sbws is CPU bound. Weird results were due to a helper that is not capable of much bandwidth - Time is still passing. In less than 2w the open source paperwork should be back next week: - Put docs on an onion service - Ask dirauths what they want to do about sbws servers In the coming weeks, I think I need to be transitioning away from "big" feature type stuff that isn't absolutely necessary, and instead focus my time on docs, tests, and making sbws easy to use. I think literal next steps need to include a practice deployment and transition in the testnet. It's one thing to run an sbws scanner and server. I've done that. I want to see other people try it so we can learn what isn't obvious to anyone but me. I want to exercise getting scan results to the dirauth so it can put them in the consensus. i've tried scanner and server and reported problems :) * list tasks for next months: https://pad.riseup.net/p/bfikl6PM0YV9 * plan proposed by teor with testnet deployment: * ... * it'll be very useful to have sbws doc public! juga: last week: * 1st draft spec * setup dirauth, sbws server, sbws scanner * dirauth not yet in consensus * start doc about what we have deployed so far (started to feel a bit lost about what we have/need) * added draft faq and glossary to doc next week: * finish spec * check the status of config files vs cli args, with debug (sorry pastly, didn't follow close the changes) and see whether #91 is really needed * check the status of tests/chutney * implement actually some unit test? * think how to make easier deployment at some point and add something on it? (cron/Makefile/system services?) questions: * pastly: once sbws repo is public, can we work on new releases publicly? [pastly: says yes] teor: last week: * occasional suggestions this week: * make my test dirauth a trust pastly's and juga's dirauths * eventually try to set up sbws on the testnet

1 0

Proposal: The move to two guard nodes
by Mike Perry 18 Apr '18

18 Apr '18

In-line below for ease of comment. Also available at: https://gitweb.torproject.org/user/mikeperry/torspec.git/tree/proposals/xxx… =========================== Filename: xxx-two-guard-nodes.txt Title: The move to two guard nodes Author: Mike Perry Created: 2018-03-22 Supersedes: Proposal 236 0. Background Back in 2014, Tor moved from three guard nodes to one guard node[1,2,3]. We made this change primarily to limit points of observability of entry into the Tor network for clients and onion services, as well as to reduce the ability of an adversary to track clients as they move from one internet connection to another by their choice of guards. 1. Proposed changes 1.1. Switch to two guards per client When this proposal becomes effective, clients will switch to using two guard nodes. The guard node selection algorithms of Proposal 271 will remain unchanged. Instead of having one primary guard "in use", Tor clients will always use two. This will be accomplished by setting the guard-n-primary-guards-to-use consensus parameter to 2, as well as guard-n-primary-guards to 2. (Section 3.1 covers the reason for both parameters). This is equivalent to using the torrc option NumEntryGuards=2, which can be used for testing behavior prior to the consensus update. 1.2. Enforce Tor's path restrictions across this guard layer In order to ensure that Tor can always build circuits using two guards without resorting to a third, they must be chosen such that Tor's path restrictions could still build a path with at least one of them, regardless of the other nodes in the path. In other words, we must ensure that both guards are not chosen from the same /16 or the same node family. In this way, Tor will always be able to build a path using these guards, preventing the use of a third guard. 2. Discussion 2.1. Why two guards? The main argument for switching to two guards is that because of Tor's path restrictions, we're already using two guards, but we're using them in a suboptimal and potentially dangerous way. Tor's path restrictions enforce the condition that the same node cannot appear twice in the same circuit, nor can nodes from the same /16 subnet or node family be used in the same circuit. Tor's paths are also built such that the exit node is chosen first and held fixed during guard node choice, as are the IP, HSDIR, and RPs for onion services. This means that whenever one of these nodes happens to be the guard[4], or be in the same /16 or node family as the guard, Tor will build that circuit using a second "primary" guard, as per proposal 271[7]. Worse still, the choice of RP, IP, and exit can all be controlled by an adversary (to varying degrees), enabling them to force the use of a second guard at will. Because this happens somewhat infrequently in normal operation, a fresh TLS connection will typically be created to the second "primary" guard, and that TLS connection will be used only for the circuit for that particular request. This property makes all sorts of traffic analysis attacks easier, because this TLS connection will not benefit from any multiplexing. This is more serious than traffic injection via an already in-use guard because the lack of multiplexing means that the data retention level required to gain information from this activity is very low, and may exist for other reasons. To gain information from this behavior, an adversary needs only connection 5-tuples + timestamps, as opposed to detailed timeseries data that is polluted by other concurrent activity and padding. In the most severe form of this attack, the adversary can take a suspect list of Tor client IP addresses (or the list of all Guard node IP addresses) and observe when secondary Tor connections are made to them at the time when they cycle through all guards as RPs for connections to an onion service. This adversary does not require collusion on the part of observers beyond the ability to provide 5-tuple connection logs (which ISPs may retain for reasons such as netflow accounting, IDS, or DoS protection systems). A fully passive adversary can also make use of this behavior. Clients unlucky enough to pick guard nodes in heavily used /16s or in large node families will tend to make use of a second guard more frequently even without effort from the adversary. In these cases, the lack of multiplexing also means that observers along the path to this secondary guard gain more information per observation. 2.2. Why not MOAR guards? We do not want to increase the number of observation points for client activity into the Tor network[1]. We merely want better multiplexing for the cases where this already happens. 2.3. Can you put some numbers on that? The Changing of the Guards[13] paper studies this from a few different angles, but one of the crucially missing graphs is how long a client can expect to run with N guards before it chooses a malicious guard. However, we do have tables in section 3.2.1 of proposal 247 that cover this[14]. There are three tables there: one for a 1% adversary, one for a 5% adversary, and one for a 10% adversary. You can see the probability of adversary success for one and two guards in terms of the number of rotations needed before the adversary's node is chosen. Not surprisingly, the two guard adversary gets to compromise clients roughly twice as quickly, but the timescales are still rather large even for the 10% adversary: they only have 50% chance of success after 4 rotations, which will take about 14 months with Tor's 3.5 month guard rotation. 2.4. What about guard fingerprinting? More guards also means more fingerprinting[8]. However, even one guard may be enough to fingerprint a user who moves around in the same area, if that guard is low bandwidth or there are not many Tor users in that area. Furthermore, our use of separate directory guards (and three of them) means that we're not really changing the situation much with the addition of another regular guard. Right now, directory guard use alone is enough to track all Tor users across the entire world. While the directory guard problem could be fixed[12] (and should be fixed), it is still the case that another mechanism should be used for the general problem of guard-vs-location management[9]. 3. Alternatives There are two other solutions that also avoid the use of secondary guard in the path restriction case. 3.1. Eliminate path restrictions entirely If Tor decided to stop enforcing /16, node family, and also allowed the guard node to be chosen twice in the path, then under normal conditions, it should retain the use of its primary guard. This approach is not as extreme as it seems on face. In fact, it is hard to come up with arguments against removing these restrictions. Tor's /16 restriction is of questionable utility against monitoring, and it can be argued that since only good actors use node family, it gives influence over path selection to bad actors in ways that are worse than the benefit it provides to paths through good actors[10,11]. However, while removing path restrictions will solve the immediate problem, it will not address other instances where Tor temporarily opts use a second guard due to congestion, OOM, or failure of its primary guard, and we're still running into bugs where this can be adversarially controlled or just happen randomly[5]. While using two guards means twice the surface area for these types of bugs, it also means that instances where they happen simultaneously on both guards (thus forcing a third guard) are much less likely than with just one guard. (In the passive adversary model, consider that one guard fails at any point with probability P1. If we assume that such passive failures are independent events, both guards would fail concurrently with probability P1*P2. Even if the events are correlated, the maximum chance of concurrent failure is still MIN(P1,P2)). Note that for this analysis to hold, we have to ensure that nodes that are at RESOURCELIMIT or otherwise temporarily unresponsive do not cause us to consider other primary guards beyond than the two we have chosen. This is accomplished by setting guard-n-primary-guards to 2 (in addition to setting guard-n-primary-guards-to-use to 2). With this parameter set, the proposal 271 algorithm will avoid considering more than our two guards, unless *both* are down at once. 3.2. No Guard-flagged nodes as exit, RP, IP, or HSDIRs Similar to 3.1, we could instead forbid the use of Guard-flagged nodes for the exit, IP, RP, and HSDIR positions. This solution has two problems: First, like 3.1, it also does not handle the case where resource exhaustion could force the use of a second guard. Second, it requires clients to upgrade to the new behavior and stop using Guard flagged nodes before it can be deployed. 4. The future is confluxed An additional benefit of using a second guard is that it enables us to eventually use conflux[6]. Conflux works by giving circuits a 256bit cookie that is sent to the exit/RP, and circuits that are then built to the same exit/RP with the same cookie can then be fused together. Throughput estimates are used to balance traffic between these circuits, depending on their performance. We have unfortunately signaled to the research community that conflux is not worth pursuing, because of our insistence on a single guard. While not relevant to this proposal (indeed, conflux requires its own proposal and also concurrent research), it is worth noting that whichever way we go here, the door remains open to conflux because of its utility against similar issues. If our conflux implementation includes packet acking, then circuits can still survive the loss of one guard node due to DoS, OOM, or other failures because the second half of the path will remain open and usable (see the probability of concurrent failure arguments in Section 3.1). If exits remember this cookie for a short period of time after the last circuit is closed, the technique can be used to protect against DoS/OOM/guard downtime conditions that take down both guard nodes or destroy many circuits to confirm both guard node choices. In these cases, circuits could be rebuilt along an alternate path and resumed without end-to-end circuit connectivity loss. This same technique will also make things like ephemeral bridges (ie Snowflake/Flashproxy) more usable, because bridge uptime will no longer be so crucial to usability. It will also improve mobile usability by allowing us to resume connections after mobile Tor apps are briefly suspended, or if the user switches between cell and wifi networks. Furthermore, it is likely that conflux will also be useful against traffic analysis and congestion attacks. Since the load balancing is dynamic and hard to predict by an external observer and also increases overall traffic multiplexing, traffic correlation and website traffic fingerprinting attacks will become harder, because the adversary can no longer be sure what percentage of the traffic they have seen (depending on their position and other potential concurrent activity). Similarly, it should also help dampen congestion attacks, since traffic will automatically shift away from a congested guard. References: 1. https://blog.torproject.org/improving-tors-anonymity-changing-guard-paramet… 2. https://trac.torproject.org/projects/tor/ticket/12206 3. https://gitweb.torproject.org/torspec.git/tree/proposals/236-single-guard-n… 4. https://trac.torproject.org/projects/tor/ticket/14917 5. https://trac.torproject.org/projects/tor/ticket/25347#comment:14 6. https://www.cypherpunks.ca/~iang/pubs/conflux-pets.pdf 7. https://gitweb.torproject.org/torspec.git/tree/proposals/271-another-guard-… 8. https://trac.torproject.org/projects/tor/ticket/9273#comment:3 9. https://tails.boum.org/blueprint/persistent_Tor_state/ 10. https://trac.torproject.org/projects/tor/ticket/6676#comment:3 11. https://bugs.torproject.org/15060 12. https://trac.torproject.org/projects/tor/ticket/10969 13. https://www.freehaven.net/anonbib/cache/wpes12-cogs.pdf 14. https://gitweb.torproject.org/torspec.git/tree/proposals/247-hs-guard-disco… -- Mike Perry

4 7

[release] [protocol] Onionoo 6.0-1.13.0 and metrics-lib 2.3.0
by iwakeh 18 Apr '18

18 Apr '18

Hi there! Two new releases are available at: https://dist.torproject.org/onionoo/6.0-1.13.0/ https://dist.torproject.org/metrics-lib/2.3.0/ This Onionoo release provides the announced major version change to 6.0 Protocol change (also summarized in [0], [1]): Change the "exit_addresses" field to not exclude current OR addresses anymore. This release also includes some bug fixes, please take a look at the change log [2]. The changes are already deployed on all onionoo.torproject.org instances. The new metrics-lib release adds support for reading web server logs contained in tar-archives and a backwards compatible API change detailed in the changelog [3] and described in the javadoc documentation [4]. Please direct comments and questions to the metrics-team mailing list [5]. Cheers, iwakeh [0] https://metrics.torproject.org/onionoo.html#versions_6_0 [1] https://metrics.torproject.org/onionoo.html#details_relay_exit_addresses [2] https://gitweb.torproject.org/onionoo.git/plain/CHANGELOG.md [3] https://gitweb.torproject.org/metrics-lib.git/plain/CHANGELOG.md [4] https://metrics.torproject.org/metrics-lib/org/torproject/descriptor/Server… [5] https://lists.torproject.org/cgi-bin/mailman/listinfo/metrics-team

1 0