August 2015 - tor-dev - lists.torproject.org

Relay Database: Existing Schemas?
by nusenu 26 Feb '16

26 Feb '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, I'm planing to store relay data in a database for analysis. I assume others have done so as well, so before going ahead and designing a db schema I'd like to make sure I didn't miss pre-existing db schemas one could build on. Data to be stored: - - (most) descriptor fields - - everything that onionoo provides in a details record (geoip, asn, rdns, tordnsel, cw, ...) - - historic records I didn't find something matching so far, so I'll go ahead, but if you know of other existing relay db schemas I'd like to hear about it. thanks, nusenu "GSoC2013: Searchable Tor descriptor archive" (Kostas Jakeliunas) https://www.google-melange.com/gsoc/project/details/google/gsoc2013/wfn/ 5866452879933440 https://lists.torproject.org/pipermail/tor-dev/2013-May/004923.html https://lists.torproject.org/pipermail/tor-dev/2013-September/005357.htm l https://github.com/wfn/torsearch (btw, someone knows the license of this?) > This is true: the summary/details documents (just like in Onionoo > proper) deal with the *last* known info about relays. ernie https://gitweb.torproject.org/metrics-db.git/plain/doc/manual.pdf (didn't find db/tordir.sql mentioned in the pdf) "Instructions for setting up relay descriptor database" https://lists.torproject.org/pipermail/tor-dev/2010-March/001783.html "Set up descriptor database for other researchers" https://trac.torproject.org/projects/tor/ticket/1643 -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVLrmVAAoJEFv7XvVCELh0+ycP/0VkRWMGAhx7YwWhixtOPB0q ouitosse3SdLCMd1W7cps7nnQhcGv8E1xeUPYeizNQ+RIK9ldvnwgxhVt9ETOI3e oL6bcx3B/NSbTcGfZhIU6XCCX7+39CXz3/+/gXg4c1a5Cd9rYKuwTn7ZApPUNAK2 9yfckCwHaqh67ZDHB1A7nrHX1BJAjwhri5JRGomHSUXPwNZOyMzpgPkB+I92esqc Z4ajQo2D+Vyk42qF7apqp5ApVBLdt5ycUtW1j4s4KvyJH2BjeP4tCXvvYLKv9IvN TGHMJ111NQXbuk6H1OdaDmMZhhW6utPk8YAbNc7RZHqCdH7No40zFVlikLDrV2Xl uLt8FMlck7hGQv/4udA59Dw3PGHfrSiCbvJXb0S2jarnWOH2O+zSSuMs8jQM/x7k 6tIOlwtRDVGw3VuvNbb4MJnLzdHRxq3qy6ueDYuhmhHslxjD1Cbr3eE1RA7Da9aX kNDgjfXtdah52HWUZbIIxHQYzyCg5G3M5yy51GhUlA2Fe1sbPpNcEBeyEDK/jH3Y +7G+tgR+CdMvSFraCiYKHa1drVPJHeqNvZAqPNHKxpKwOwOoJEYrsW4wb/mJ7ybe o8xNYK691hcaMvHqwI47tw4hZfrI0f0+gQKKzENQQk8/8yUZ/0jgMRlW/Cz0yYIa FZEg+OJ/yEs8vKfYerIN =Rfn4 -----END PGP SIGNATURE-----

3 2

Proposal: Merging Hidden Service Directories and Introduction Points
by John Brooks 29 Sep '15

29 Sep '15

Hello! George and I, along with the other participants of this hidden services meeting, have written a proposal for the idea to merge hidden service directories and introduction points into the same entity along with proposal 224. Comments are encouraged, especially if there are downsides or side effects that we haven’t written about yet, or that you have a different opinion on. The intent is that we can decide to do this before implementing proposal 224, so they can be implemented together. The proposal is attached, and also available from: https://raw.githubusercontent.com/special/torspec/224-no-hsdir/proposals/id… Thanks! - John

11 19

Two new Onionoo versions 2.4 and 2.5 add new "effective_family" and "measured" fields
by Karsten Loesing 27 Sep '15

27 Sep '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi devs, I was reminded by Harmony that I said a while back [0] that I would announce new Onionoo versions on this list and not only in private message to Harmony for including them in the next TWN issue. Oops. So, version 2.4 added a new "effective_family" field to details documents, listing all the relays with which the relay in question is in an effective, mutual family relationship. The main goal here is to make it easier to detect misconfigured relay families. This can be relay operators or friendly people watching over the Tor network and reminding relay operators to fix their configurations. [1] The new version 2.5 that I deployed this week adds the optional "measured" field to details documents. The main idea behind this new field is that relay operators and Tor network debuggers can now figure out easily whether a relay is affected by not being measured by a sufficient number of bandwidth authorities and as a result has lower usage than it could handle. This field is not yet displayed by the Onionoo clients Atlas or Globe, but it's accessible via Onionoo's API [2]. More details are on ticket #16020 [3]. All the best, Karsten [0] onionoo-announce@ posting from April 27, 2015 [1] https://lists.torproject.org/pipermail/tor-news/2015-August/000109.html [2] https://onionoo.torproject.org/protocol.html#details [3] https://trac.torproject.org/projects/tor/ticket/16020 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJV1HYeAAoJEJD5dJfVqbCrFJAIAIQYIC0CiSx6GJJikQ6CTxgw +AljFxABfnLtHu58EtBLx1jRJo0u9fStCMA/YFWLiq3FmEhCb5YvoL4zwa92PwlN bIoEricgq3sXjOm6TxJ75/6OBeH8AVIsah6qxHFkVfCPwjmt0bsDyaE7VqlCv2tN tAunIZcDyhsStGOOErr5uc9FHgPquvb66Iy7YFwnJxioMRE3yp4GBTDv78WyUEuR TZOWBh/cbJYrwSu5b+vU3C+t9eAI9ErQ8BDYlwy0C6zgJO+paSUkxWkCvuI3nv2l e/Ol6M40c9syZEb5jexn3K3NTOpJCKKhjXoXtP7UHXz8ZSdhZw2gxi31ZXYrSkQ= =/rwg -----END PGP SIGNATURE-----

3 3

Draft Proposal: Random Number Generation During Tor Voting
by George Kadianakis 16 Sep '15

16 Sep '15

Hello there, we are glad to release a first draft of our proposal on distributed random generation using the Tor voting process. It specifies how Tor dirauths can generate a fresh random value every day using a commit-and-reveal protocol. The protocol piggybacks on top of the regular Tor voting procedure which happens every hour. Our proposal spends lots of effort resolving the various edge cases and engineering issues that come up when you are trying to fit such a protocol into the Tor voting system. It also introduces a new consensus flavor document which is used to host shared randomness information, so that the regular consensus does not get affected if this feature is flawed. We are especially interested in feedback from people who are familiar with the Tor voting protocol and can tell us whether we have messed something up, or whether there are attacks that we did not consider. We hope the document (and the illustration) is helpful for understanding the protocol. Let us know if you have any questions, or suggestions for improving the proposal. We believe this proposal is fundamental for the security of hidden services and for developing proposal 224, hence we invite everyone to provide feedback and improvements. Thanks! ----------------------------------------------------------------------------------- Filename: xxx-commit-reveal-consensus.txt Title: Random Number Generation During Tor Voting Authors: David Goulet, George Kadianakis Created: 2015-08-03 Status: Draft 1. Introduction 1.1. Motivation For the next generation hidden services project, we need the Tor network to produce a fresh random value every day in such a way that it cannot be predicted in advance or influenced by an attacker. Currently we need this random value to make the HSDir hash ring unpredictable (#8244), which should resolve a wide class of hidden service DoS attacks and should make it harder for people to gauge the popularity and activity of target hidden services. Furthermore, this random value can be used by other Tor-related protocols (or even non-Tor-related) like OnioNS to introduce unpredictability to the protocol. 1.2. Previous work Proposal 225 specifies a commit-and-reveal protocol that can be run as an external script and have the results be fed to the directory authorities. However, directory authority operators feel unsafe running a third-party script that opens TCP ports and accepts connections from the Internet. Hence, this proposal aims to embed the commit-and-reveal idea in the Tor voting process which should makes it smoother to deploy and maintain. Another idea proposed specifically for Tor is Nick Hopper's "A threshold signature-based proposal for a shared RNG" which was never turned into an actual Tor proposal. 2. Overview This proposal alters the Tor consensus protocol such that a random number is generated by the directory authorities during the regular voting process. The distributed random generator scheme is based on a commit-and-reveal technique. The proposal also specifies how the final shared random value is embedded in consensus documents so that clients who need it can get it. 2.1. Ten thousand feet view Our commit-and-reveal protocol aims to produce a fresh shared random value everyday at 12:00UTC. The final fresh random value is embedded in the microdescriptor consensus document at that time. Our protocol uses a *new* consensus flavor document called "shared randomness document" (SR doc). We use a new consensus document as a way to keep ground truth state and also as a way to apply the majority (see section [MAJORITY]) rule on commit and reveal values. It also allows rebooting authorities to rejoin the protocol in some cases. Our protocol has two phases and uses the hourly voting procedure of Tor. Each phase lasts 12 hours, which means that 12 voting rounds happen in between. In short, the protocol works as follows: Commit phase: Starting at 12:00UTC and for a period of 12 hours, authorities every hour send their commitments in their votes. They also include any received commitments from other authorities, if available. From those votes, a shared random document consensus is computed containing the commitments decided by the majority. Reveal phase: At 00:00UTC, the reveal phase starts and lasts till the end of the protocol at 12:00UTC. In this stage, authorities must reveal the value they committed to in the previous phase. The commitment and revealed values from other authorities, when available, are also added to the vote. Then a shared random document consensus is computed containing the commitments and the revealed values agreed on. Shared Randomness Calculation: At 12:00UTC, the shared random value is computed from the agreed revealed values located in the shared random document and finally added to the microdescriptor consensus. This concludes the commit-and-reveal procedure at 12:00UTC everyday. 2.2. Commit & Reveal Our commit-and-reveal protocol aims to produce a fresh shared random value everyday at 12:00UTC. In the beginning of that time period, each authority generates a new random value and keeps it for the whole day. The authority cryptographically hashes the random value and calls the output its "commitment" value. The original random value is called the "reveal value". Given a reveal value you can verify that it corresponds to a given commitment value. However given a commitment value you cannot derive the underlying reveal value. 2.3. Microdescriptor Consensus [MDCONS] Every hour, the microdescriptor consensus documents need to include the shared random value of the day, as well as the shared random value of the previous day. That's because either of these values might be needed at a given time for a Tor client to access a hidden service according to section [TIME-OVERLAP] of proposal 224. These means that these two values also need to be included in votes and in SR documents as well. Microdescriptor consensuses include: (a) The shared random value of the current time period. This is derived from the reveal values sent by the authorities during the voting session. (b) The shared random value of the previous time period. This is the same shared random value that was included in the votes. 2.4. Shared Random Document [SRDOC] The Shared Random document is a consensus flavor that contains the current state of our commit & reveal protocol. Since it uses the consensus mechanism of Tor, we use it as a way to enforce majority voting on the commitments and reveals without messing with the actual network status consensus. See section [REBOOT] for detail on how this document is handled when an authority reboots. During the commitment phase, the SR doc is populated with the commitments of all authorities. Then during the reveal phase, it's also used to store the reveal values as well. As discussed previously, the shared random values from the current and previous time period must be present in the document at all times if they are available. A shared random document requires 50% + 1 authority signatures to be considered valid. As this proposal is being written, there are 9 authorities thus we would need 5. 2.5. Protocol Illustration We have prepared an illustration to help you understand the protocol. You can find it here: https://people.torproject.org/~asn/hs_notes/shared_rand.jpg For every hour, it shows the authority votes and the resulting SR doc and microdescriptor consensus. The chain 'A_1 -> c_1 -> r_1' denotes that the authority committed to the value c_1 which corresponds to the reveal value r_1. The illustration depicts the first 25 hours of running the protocol. It starts with the very first commit round, then moves on to the second commit round, and then skips directly to the last commit round. Then the reveal phase starts, where we again show the first, second and last rounds. After the reveal phase is done, we generate the shared randomness (SR_1) and we start the new commit phase. The illustration finishes with the second round of this new commit phase. We advice you to revisit this after you have read the whole document. 3. Protocol In this section we give a detailed specification of the protocol. We describe the protocol participants' logic and the messages they send. The encoding of the messages is specified in the next section ([SPEC]). Now we go through the phases of the protocol: 3.1 Commitment Phase [COMMITMENTPHASE] The commit phase lasts from 12:00UTC to 00:00UTC. The goal is that at the end of this phase, the shared random document MUST contain a single commitment value from each authority (or none, if that authority did not participate in this phase). 3.1.1. Voting During Commitment Phase During the commit phase, each authority includes in its votes: - A commitment value for this consensus period. - Any commitments received from other authorities. - The two previous shared random values produced by the protocol (if any). After all votes have been received or pulled in, the authorities collectively generate the shared random document containing the commitments. 3.1.2. Shared Random Document During Commitment Phase [SRDOCCOMMIT] During the commitment phase, the shared random document contains: - The commitments received by the majority of authorities - The two previous shared random values produced by the protocol (if any). A commitment should only be transcribed to the shared random document if and only if the majority of the voting authorities agreed that a particular commitment was sent by a particular authority. Appendix section [COMMITEXAMPLE] contains an example of this procedure. The commit phase lasts for 12 hours, so authorities have multiple chances to commit their values. An authority can commit a second value during a subsequent round of the commit phase, but only the last value should be transcribed to the shared random document and only if it has been seen by the majority. Also, an authority should not be able to register a commitment value for a different authority. Hence, an authority X should only vote and place in the SR doc commitments by authority Y, iff authority Y included that commitment in its vote. 3.1.3. First & Last Round Of Commitment Phase [FIRSTLASTROUND] It's worth mentioning that during the very first round of the commitment phase at 12:00UTC, each authority votes its own commitment and is unaware of the commitments of the other authorities. For this reason, it's unlikely that a majority opinion of commitments will be created at 12:00UTC. Instead authorities are expected to form a majority opinion and transcribe commitments to the SR doc during the voting period of 13:00UTC or at least until the reveal phase. Similarly, an authority will not be able to commit to a new value during the last round of the commitment phase. That's because there won't be enough time for the other authorities to form a majority opinion about this value before the reveal phase. Hence, Tor authorities SHOULD NOT commit new values during the last round of the commitment phase at 23:00UTC. 3.2 Reveal Phase The reveal phase lasts from 00:00UTC to 12:00UTC. Now that the commitments have been agreed on, it's time for authorities to reveal their random values. 3.2.1. Voting During Reveal Phase During the reveal phase, each authority includes in its votes: - Its reveal value that was previously committed in the commit phase. - All the commitments and reveals received from other authorities. - The two previous shared random values produced by the protocol (if any). The set of commitments have been established during the commitment phase and must remain the same. If an authority tries to change its commitment during the reveal phase or introduce a new commitment, the entire vote MUST be ignored for the purposes of this system. To do so, authorities during the first reveal round MUST check that received votes contain the same commitments as the last SR document of the commitment phase. In subsequent reveal rounds, authorities check the previous hour SR document for commitment validation. After all votes have been received, authorities generate the shared random document along with the consensus. 3.2.2. Shared Random Document During Reveal Phase [SRDOCREVEAL] During the reveal phase, the shared random document contains: - The commitments agreed on during the commitment phase. - The corresponding reveal values from the majority of authorities. - The two previous shared random values produced by this system (if any). Similar to the commitment phase, authorities transcribe reveal values to the shared random document if and only if the majority of the voting authorities have voted on that particular reveal value. An example of this can be seen in section [REVEALEXAMPLE]. Section [FIRSTLASTROUND] also applies for the reveal phase. This means that Tor authorities SHOULD NOT reveal new values during the last round of the reveal phase at 11:00UTC. 3.3. Shared Random Value Calculation At 12:00UTC Finally, at 12:00UTC every day, authorities compute a fresh shared random value and this value must be added to the microdescriptor consensus so clients can use it. Authorities calculate the shared random value using the reveal values in the latest shared random document as specified in subsection [SRCALC]. If the shared random value contains reveal contributions by less than 3 directory authorities, it MUST NOT be created. Instead, the old shared random value should be used as specified in section [SRDISASTER]. Authorities at 12:00UTC start including this new shared random value in their votes, replacing the one from two protocol runs ago. Authorities also start including this new shared random value in the SR document and in the microdescriptor consensus as well. Apart from that, authorities proceed voting normally as they would in the first round of the commitment phase (section [COMMITMENTPHASE]). 3.3.1. Shared Randomness Calculation [SRCALC] An authority that wants to derive the shared random value V, should use the appropriate reveal values for that time period and calculate V as follows: V = H(ID_a | R_a | ID_b | R_b | ...) where the ID_k value is the identity fingerprint of directory authority k and R_k is its corresponding reveal value of that authority for the current period. H is sha256 for protocol version 1. XXX Should the hashing here include more elements? Like the previous random value for chaining? Or the current date? See how the NIST beacon does it in case we can steal some additional RNG security properties: http://hackaday.com/2014/12/19/nist-randomness-beacon/ 3.4. Bootstrapping Procedure As described in [MDCONS], two shared random values are required for the HSDir overlay periods to work properly as specified in proposal 224. Hence clients MUST NOT use the randomness of this system till it has bootstrapped completely; that is, until two shared random values are included in a consensus. This should happen after three 12:00UTC consensuses have been produced, which takes 48 hours. 3.5. Rebooting Directory Authorities [REBOOT] The shared randomness protocol must be able to support directory authorities who leave or join in the middle of the protocol execution. An authority that commits in the Commitment Phase and then leaves SHOULD store its reveal value on disk so that it continues participating in the protocol if it returns before or during the Reveal Phase. The reveal value MUST be stored timestamped to avoid sending it on wrong protocol runs. For this reason, other authorities should carry the commitment values of absent authorities in the shared randomness document until the end of the protocol. The shared randomness document can be used to verify that the commitment values are carried properly. An authority that misses the Commitment Phase cannot commit anymore, so it's unable to participate in the protocol for that run. Same thing for an authority that misses the Reveal phase. Authorities who do not participate in the protocol SHOULD still carry commits and reveals of others in their vote. 3.6. How we define majority [MAJORITY] The shared randomness protocol must be able to support directory authorities who participate in the consensus protocol but not in the shared randomness protocol. It must also be able to tolerate authorities who drop or join in the middle of the protocol. The security of this proposal strongly relies on forming majority opinion so it's important for the number of participants to always be well defined: In the voting session right before creating the SR doc, we define the number of active participants to be the number of directory authorities that included commit/reveal values in their votes. As specified in sections [SRDOCCOMMIT] and [SRDOCREVEAL], a commit/reveal value should be transcribed to the SR doc if and only if the majority voted for it. So for example, if there are 6 active participants, a commit value will only be transcribed if 4 or more participants agreed on it. Furthermore, as specified in section [SRDOC], the shared random document is considered valid only if it is signed by 50% + 1 authorities. XXX The number of active participants is dynamic as authorities leave and join the protocol. Since the number of active participants is dynamic , an attacker could trick some authorities believing there are N participants and some others believing there are N-1 participants, by sending different votes to different auths. Should we worry? [asn] A way to avoid a dynamic number of participants could be to set the number of participants to be the number of auths who committed during the very first commitment phase round. 3.7. Shared Randomness Disaster Recovery [SRDISASTER] If the consensus at 12:00UTC fails to be created, then there will be no new shared random value for the day. Directory authorities should keep including the previous shared random values in the consensus till the next 12:00UTC commit-and-reveal session. The time period needs to be updated to reflect the current time period even if the random value stays the same. Clients should keep on using this shared random values. 4. Specification [SPEC] 4.1 Voting This section describes how commitments, reveals and SR values are encoded in votes. We describe how to encode both the authority's own commits/reveals and also the commits/reveals received from the other authorities. Commits and reveals share the same line, but reveals are optional. 4.1.1 Encoding the authority's own commit/reveal value An authority that wants to commit (or reveal) a value during a vote, should generate a random 256-bit value REVEAL, and include its commitment COMMIT in its 12:00UTC vote as follows: "shared-rand-commitment" SP algname SP COMMIT [SP REVEAL] NL During the Reveal Phase, an authority can also optionally reveal the value REVEAL. The "algname" is the hash algorithm that should be used to compute COMMIT and REVEAL if any. It should be "sha256" for this version. The commitment value COMMIT is constructed as follows: C = base64-encode( SHA256(REVEAL) ) 4.1.2 Encoding commit/reveal values received by other authorities [COMMITOTHER] An authority puts in its vote the commitments and reveals it has seen from the other authorities. To do so, it includes the following in its votes: "shared-rand-received-commitment" SP identity SP algname SP COMMIT [SP REVEAL] NL when "identity" is the hex-encoded commitment's authority fingerprint and COMMIT is the received commitment value. Authorities can also optionally include the reveal value REVEAL. There MUST be only one line per authority else the vote is considered invalid. Finally, the "algname" is the hash algorithm that should be used to compute COMMIT and REVEAL if any. 4.1.3. Shared Random value Authorities include a shared random value in their votes using the following encoding for the previous and current value respectively: "shared-rand-previous-value" SP value NL "shared-rand-current-value" SP value NL where "value" is the actual shared random value. It's computed as specified in the section [SRCALC]. To maintain consistent ordering, the shared random values of the previous period should be listed before the values of the current period. 4.2. Shared Random Document As a way to keep ground truth state in this protocol, we introduce a new consensus flavor document. We call it the "Shared Random Document". This document is only used by directory authorities. This new consensus flavor should be signed with the sha256 signature format as documented in proposal 162. 4.2.1 Format [SRFORMAT] This document has a very strict format because authorities need to generate the exact same document. It contains a preamble, a commitment and reveal section, a list of shared random values and finally a footer. The preamble (or header) contains the following items. They MUST occur in the order given here: "shared-random-version" SP version SP flavor NL [At start, exactly once.] A document format version. For this specification, version is "1". The flavor is always "shared-random". "created" SP YYYY-MM-DD SP HH:MM:SS NL [Exactly once] The creation time of this document. "valid-until" SP YYYY-MM-DD SP HH:MM:SS NL [Exactly once] After this time, this document is invalid and shouldn't be used nor trusted. The validity time period is 3 hours. "protocol-phase" SP phase NL [Exactly once] The current protocol phase when this document is generated. The accepted values are: "commitment" and "reveal". The following details the commitment and reveal section. "shared-rand-commitment" SP algname SP identity SP commitment-value [SP revealed-value] NL [Exactly once per authority] This is the commitment or/and reveal value agreed upon by the majority from one authority. The algname is always "sha256" in version 1. The "identity" is the authority hex-encoded digest of the authority identity key of the signing authority from which the values are from. Finally, "{commitment|revealed}-value" is the value as specified in section [SPEC]. Next is the shared random value section. "shared-rand-previous-value" SP value NL [At most once] This is the previous shared random value agreed on at the previous period. The "value" is defined in section [SRCALC]. "shared-rand-current-value" SP value NL [At most once] This is the latest shared random value. The "value" is defined in section [SRCALC]. Finally, the footer of the document: "shared-random-footer" NL [Exactly once] It contains one subsection, the authority signatures. "authority-signature" SP algname SP identity SP signing-key-digest NL Signature NL [Exactly once per authority] The "identity" is the hex-encoded digest of the authority identity key and the "signing-key-digest" is the hex-encoded digest of the current authority signing key of the signing authority. The "algname" item is the algorithm used to compute the hash of the document before signing it. As proposal 162 proposed, "sha256" should be used. The authority-signature entry MUST be ignored if "algname" is unrecognized. See dir-spec.txt for a specification for the Signature item. 4.3. Shared Random Value in Consensus [SRCONSENSUS] Authorities insert the two shared random values in the consensus following the same encoding format as in [SRFORMAT]. 5. Security Analysis 5.1. Security of commit-and-reveal and future directions The security of commit-and-reveal protocols is well understood, and has certain flaws. Basically, the protocol is insecure to the extent that an adversary who controls b of the authorities gets to choose among 2^b outcomes for the result of the protocol. However, an attacker who is not a dirauth should not be able to influence the outcome at all. We believe that this system offers sufficient security especially compared to the current situation. More secure solutions require much more advanced crypto and more complex protocols so this seems like an acceptable solution for now. 5.2. Is there a need for a final agreement phase? Commit-and-reveal protocols usually also end with an agreement phase, during which participants agree on which reveal values should be used to make the shared random value. An agreement phase is needed, because if the protocol ended with the reveal phase, an evil authority could wait until the last reveal round, and reveal its value to half of the authorities. That would partition the authorities into two sets: the ones who think that the shared random value should contain this new reveal, and the rest who don't know about it. This would result in a tie and two different SR docs. However, we believe that an agreement phase is not necessary in our protocol since reveal values are transcribed in the SR document if only if the majority agrees. Hence, a tie is not enough to confuse the authorities since it's not majority and the offending value would just be discarded. That said, an attack that could still work here would be if an authority can make half of the authorities believe that the value should be discarded, and make the other half of the authorities believe that the value should be included. That could be achieved if the attacker could force honest authorities to send different votes to different authorities. We believe this should not be the case currently, but we should look more into this. XXX Needs feedback by a person who knows the voting protocol well!!! 5.3. Predicting the shared random value during reveal phase The reveal phase lasts 12 hours, and most authorities will send their reveal value on the first round of the reveal phase. This means that an attacker can predict the final shared random value about 12 hours before it's generated. This does not pose a problem for the HSDir hash ring, since we impose an uptime restriction on HSDir nodes, so 12 hours predictability is not an issue. Any other protocols using the shared random value from this system should be aware of this property. 6. Discussion 6.1. Why the added complexity from proposal 225? The complexity difference between this proposal and prop225 is in part because prop225 doesn't specify how the shared random value gets to the clients. This proposal spends lots of effort specifying how the two shared random values can always be readily accessible to clients. 6.2. Why do you do a commit-and-reveal protocol in 24 rounds? The reader might be wondering why we span the protocol over the course of a whole day (24 hours), when only 3 rounds would be sufficient to generate a shared random value. We decided to do it this way, because we piggyback on the Tor voting protocol which also happens every hour. We could instead only do the shared randomness protocol from 21:00 to 00:00 every day. Or to do it multiple times a day. However, we decided that since the shared random value needs to be in every consensus anyway, carrying the commitments/reveals as well will not be a big problem. Also, this way we give more chances for a failing dirauth to recover and rejoin the protocol. 6.3. Why can't we recover if we fail to do a consensus at 12:00UTC? Section [SRDISASTER] specifies that if the 12:00UTC consensus or SR doc fails to be created, we fall back to the random value of the previous day meaning authorities will carry the last valid SR values from the previous microdescriptor consensus to the new one. Theoretically, we could recover by calculating the shared randomness of the day at 13:00UTC instead. However, adding such fallback logic would complicate the protocol even further, so we have not yet considered it. 7. Appendix 7.1. Example commitment majority [COMMITEXAMPLE] Here is an example of voting during the commitment phase. The table below represents the votes of 6 individual authorities A_i (one vote per column). Since it's the commitment phase, votes include the authorities commitments and all commitments received. For example, below all authorities believe that A_1 has registered the value 7 as its commitment. +------------+------------+-------------+-------------+-------------+-----------+ | A_1 vote | A_2 vote | A_3 vote | A_4 vote | A_5 vote | A_6 vote | +------------+------------+-------------+-------------+-------------+-----------+ | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | A_1 -> 7 | | A_2 -> 66 | A_2 -> 66 | A_2 -> 42 | A_2 -> 42 | A_2 -> 42 | A_2 -> 42 | | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | A_3 -> 16 | | A_4 -> 22 | A_4 -> 22 | A_4 -> 22 | BLANK | A_4 -> 22 | BLANK | | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | A_5 -> 9 | | A_6 -> 33 | A_6 -> 33 | A_6 -> 33 | A_6 -> 33 | A_6 -> 33 | BLANK | +------------+------------+-------------+-------------+-------------+-----------+ In this case, following the majority rule, the final SR doc will contain: +-------------+ | SR Document | +-------------+ | A_1 -> 7 | | A_2 -> 42 | | A_3 -> 16 | | A_4 -> 22 | | A_5 -> 9 | | A_6 -> 33 | +-------------+ 7.2. Example reveal phase [REVEALEXAMPLE] Here is an example of voting during the reveal phase. The table below represents 6 votes by 6 different authorities A_i (one vote per column). Since it's the reveal phase, votes include all reveals received (commitments have been hidden for simplicity). For example, below all authorities believe that A_1 has revealed the value 444. Let's say that a malicious dirauth is trying to partition the group into two sets, by sending different votes to different auths. The attacker has splitted the group into two sets, the auths who think that A_6 has revealed the value 123, and the rest who have not seen a reveal from A_6. +------------+------------+-------------+-------------+-------------+------------+ | A_1 vote | A_2 vote | A_3 vote | A_4 vote | A_5 vote | A_6 vote | +------------+------------+-------------+-------------+-------------+------------+ | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | A_1 -> 444 | | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | A_2 -> 110 | | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | A_3 -> 420 | | BLANK | BLANK | A_4 -> 980 | BLANK | A_4 -> 980 | BLANK | | A_5 -> 666 | A_5 -> 555 | A_5 -> 555 | A_5 -> 555 | A_5 -> 555 | A_5 -> 555 | | A_6 -> 123 | A_6 -> 123 | A_6 -> 123 | BLANK | BLANK | BLANK | +------------+------------+-------------+-------------+-------------+------------+ Following the rules of the reveal phase, the reveal of A_4 should be ignored since it was not voted by > 3 authorities. The reveal from A_6 should also be ignored since it was only seen by half of the auths (3/6) which is not majority (it would require at least 4/6 votes). Hence, the final shared random document should contain: +-------------+ | SR Document | +-------------+ | A_1 -> 444 | | A_2 -> 110 | | A_3 -> 420 | | BLANK | | A_5 -> 555 | | BLANK | +-------------+

10 26

[RFC] On new guard algorithms and data structures
by George Kadianakis 09 Sep '15

09 Sep '15

Hello there, recently we've been busy specifying various important improvements to entry guard security. For instance see proposals 250, 241 and ticket #16861. Unfortunately, the current guard codebase is dusty and full of problems (see #12466, #12450). We believe that refactoring and cleaning up the entry guard code is essential before we proceed to more advanced security improvements. We've been working on new algorithms and data structures for guard nodes as part of ticket #12595. In this mail I include some pseudocode for this new algorithm with the hope that it will act as a draft for implementing these changes. You can find the pseucode here: https://gitweb.torproject.org/user/asn/tor.git/tree/src/or/guardlist.c?h=bu… A short description of the algorithm is included on top, and then various methods and functions are prototyped underneath to make the logic more concrete. Apart from the comments and XXXs on the code, here are some more thoughts on this work: - This new design focuses on protecting against path bias attacks, by slightly damaging our reachability. Specifically, the old design is better at recovering in filtered networks, because it will keep on adding new nodes till one succeeds. In this new design, we will not try more than 80 relays per time. So if none of them passes the filtered network, bad luck no Tor. While this failure mode should not happen much, it's bad news for users behind FascistFirewalls which are actually quite frequent. A quick fix here would be to always add an 80/443 guard on our list, however as it stands only 30% of the guards are 80/443 guards, so this has bad anonymity consequences. - To improve our algorithm and make it more robust we need to understand further what kind of path bias attacks are relevant here. The adversary here is a network adversary (like a gateway) that can block our connections to certain guards. What nasty attacks can this adversary do? If we can't find bad attacks here, then maybe we should stop worrying about those path bias attacks so much. For example a threat here with the old guard logic, is that if we used this evil gateway just for 10 minutes (in an airport), the adversary could launch a path bias attack and force us to connect to her guard node. Then even after we left that airport, we would still stick to the evil guard node which is bad. Also, an adversary that manages to own our guard using path bias attacks, then has further possibilites for biasing the rest of the circuit. What can this adversary do? - Notice that the pseudocode contains no logic about bridges. I'm not sure how bridges should be handled here. - I tried to keep the dirguard logic very simple, hoping that we can eventually forget about dirguards entirely when #12538 is done. The main dirguard feature is that we assume that populate_live_entry_guards() and add_an_entry_guard() will return dirguards when the circuit is a directory circuit. Maybe we should consider introducing the "primary dirguard" concept as well. And maybe also add some logic where Tor will move on to the next dirguard if it failed to receive a document from the current dirguard. - I used the ATTEMPTED_THRESHOLD concept of prop241, but did not use the NET_THRESHOLD and CONNECTED_THRESHOLD ideas. I removed NET_THRESHOLD because I increased the value of ATTEMPTED_THRESHOLD to the point that it can also be used as a network down indicator. Also, I was not sure what CONNECTED_THRESHOLD was useful for, and there were certain engineering issues with it (Like, if that threshold is hit, we need a logic that will *only retry the successfully connected guards*, and not all guards). - There is no log message warning the user of path bias attacks or bad network or anything. That's because there is no way to figure out what's the problem, and issuing an alarming log message here would confuse and panic the user. If we want to inform the user anyhow, maybe if the user is *actively* trying to visit a destination, and we've been cycling through our guard list for ages, maybe we should then issue a log message telling the user that something is wrong with the network. - In general, I tried to keep the number of heuristics and kludges to the minimum to keep the logic simple. Unfortunately, it seems that without a "network down" indicator (#16120) there is no way to avoid edge cases and false positives here. We should try to fix all problems here that can occur frequently or have security consequences, but there will always be scenarios where Tor will end up thinking there is no network while it's actually on a filternet. For this reason, we should give plenty of testing to this feature before we ship it to real users! - Finally, all the constants & parameters in the pseudocode are subject to change. I tried to motivate some of them, but others are just arbitrary. Feedback is very welcome and please let me know of any issues with security or reachability that you find! Or of how the pseudocode should be altered to make it more useful for implementors. Cheers!

9 19

Donncha's SoP Status Report 7 - Final
by Donncha O'Cearbhaill 03 Sep '15

03 Sep '15

Hello all, This is my seventh, and final biweekly status report for my hidden service scaling / load-balancing SoP project. Since my last status report: * I have been revising my patch for #14846 with the aim of getting it merged before the 0.2.7 feature freeze. * I've been working with Ivan Markin on getting smart cards supported for signing in OnionBalance. Thank you Ivan for your code and enthusiasm. Although the Summer of Privacy program is now over, I plan to continue maintaining and developing this project into the future. My immediate objectives are to: * Package OnionBalance in Debian * Clarifying the documentation (installation, use cases, storing master hidden service keys on an isolated machine). * Implementing smartcard / HSM support master service key storage and signing. Work on the Complex Mode is advancing but I am not yet satisfied with the reliability of that mode. I will also further investigate any potential attacks or risks with that mode. Proposals #224 (next-gen hidden services) and #246 (merging hidden service directories and introduction points) are still developing. I'll continue developing OnionBalance so that if possible, it can facilitate some form of load balancing and redundancy with next-gen hidden services. I'd like to thank my mentors David Goulet and George Kadianakis for their invaluable guidance and feedback during the Summer of Privacy. Other members of the Tor community, in particular s7r, have been exceptional helpful with testing OnionBalance and offering feedback from a hidden service operator perspective. Finally I'd like to thank all in the wider Tor community for the welcome and support you have shown me over the past months. I hope that I can continue making contributions to the Tor community into the future. Regards, Donncha

2 1

Should cloud-hosted relays be rejected?
by Philipp Winter 01 Sep '15

01 Sep '15

We sometimes see attacks from relays that are hosted on cloud platforms. I have been wondering if the benefit of having cloud-hosted relays outweighs the abuse we see from them. To get an idea of the benefit, I analysed the bandwidth that is contributed by cloud-hosted relays. I first obtained the network blocks owned by three cloud providers (Amazon AWS, Google Cloud, Microsoft Azure), and determined the percent of bandwidth they contributed in July 2015. The results show that there were typically ~200 cloud-hosted relays online: <https://nymity.ch/sybilhunting/png/cloud-hosted_relays_2015-07.png> The spike shortly after hour 200 was caused by a lot of Amazon relays named "DenkoNet". The spike at the very beginning was caused by a number of relays that might very well belong together, too, based on their uptime pattern. What counts, however, is bandwidth. Here's the total bandwidth fraction contributed by cloud-hosted relays over July 2015: <https://nymity.ch/sybilhunting/png/cloud-hosted_bandwidth_2015-07.png> There were no Google Cloud relays to contribute any bandwidth. Amazon AWS-powered relays contributed the majority of bandwidth, followed by Microsoft Azure-powered relays. Here's a summary of the time series in percent: Min. Mean Median Max. 0.2% 0.8% 0.79% 1.5% In an average consensus in July 2015, cloud-hosted relays contributed only around 0.8% of bandwidth. Note, however, that this is just a lower bound. The netblocks I used for the analysis could have changed, and I didn't consider providers other than Google, Amazon, and Microsoft. There are also cloud-hosted bridges. Tor Cloud, however, has shut down, and the number of EC2 bridges is declining: <https://metrics.torproject.org/cloudbridges.html?graph=cloudbridges&start=2…> The harm caused by cloud-hosted relays is more difficult to quantify. Getting rid of them also wouldn't mean getting rid of any attacks. At best, attackers would have to jump through more hoops. If we were to decide to permanently reject cloud-hosted relays, we would have to obtain the netblocks that are periodically published by all three (and perhaps more) cloud providers: <https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html> <https://msdn.microsoft.com/en-us/library/azure/Dn175718.aspx> <https://cloud.google.com/appengine/kb/general?hl=en#static-ip> Note that this should be done periodically because the netblocks are subject to change. Cheers, Philipp

6 7

Where are the GetTor downloads (e.g. GitHub)?
by David Fifield 01 Sep '15

01 Sep '15

Someone IRL asked me about mirrored downloads of Tor Browser on services like GitHub. I know that such services are planned (or already implemented?) for GetTor, and I know about these tickets: Integrate cloud services that are not blocked in mainland China https://bugs.torproject.org/14114 Automate upload of latest Tor Browser to cloud services https://bugs.torproject.org/14744 Script to upload Tor Browser to Github https://bugs.torproject.org/14835 My question is, is there a static URL on GitHub or similar that has the latest downloads? That is, one that people can access even without having used GetTor? Such a URL would be more useful than a typical mirror for many users. I was referred to Lantern's GitHub download page for comparison: https://github.com/getlantern/lantern/releases/tag/latest It has direct links to downloads under githubusercontent.com: https://raw.githubusercontent.com/getlantern/lantern-binaries/master/lanter… https://raw.githubusercontent.com/getlantern/lantern-binaries/master/lanter… https://raw.githubusercontent.com/getlantern/lantern-binaries/master/lanter…

2 2

New Onionoo version 2.6 adds new "alleged_family" and "indirect_family" fields
by Karsten Loesing 31 Aug '15

31 Aug '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Onionoo users, I just merged and deployed Onionoo protocol version 2.6 which adds two new fields related to relay families and marks the current "family" field as deprecated. The idea of adding three fields for relay families is that we want to support different use cases: - Atlas and Globe want to list all relays that a relay has configured as its family ("effective_family" + "alleged_family"). - Atlas and Globe may further want to mark ineffective family relationships in red ("alleged_family") and effective family relationships in blue ("effective_family"). - Roster needs a family definition that includes family members that are either in a direct family relationship with a relay ("effective_family") or that can be reached by following effective family relationships ("indirect_family"). The current "family" field will stay available until Atlas and Globe are updated. If I should also wait for other clients to be updated, please let me know. We shouldn't wait too long, because all those fingerprints increase the size of Onionoo results for everyone. All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJV5JJAAAoJEJD5dJfVqbCrlw0IAJ3NxbTE00M72EN/DLcJw2FD /KMV2vpyQqi6tharTNGtZmZrm7HMZ/RTRhR5qlkmcKE18UDKxF0sX5ezT6+1R0QV dNmUhYspP5cKcF6nqReMdd26dp4am6I5PSWqEmsMiC5vmxZbvp+B3st/9MNfYExE qTQV8rkzsWIGTGSojxXLAjzjQwy9EBXdLrMrchH+jL0igbWmWLeshDSou/N0te1g B6TBnA2LlRLKhLoAtwjjF1TREZN6Tkk9djQ8zHRjppZzu004I8bqPWiJ36qLDEqi UVmuo5zrJUQxQEQ/n+AkRBoVNblVDY7/xeGh35ENhCzs+Ls75jCQzrkzx+7upPw= =yTnY -----END PGP SIGNATURE-----

2 2

[Measurement Team] Next IRC meeting happens on Wednesday, Sep 2, 14:00 UTC in #tor-project
by Karsten Loesing 31 Aug '15

31 Aug '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, This is an announcement/reminder that there will be an IRC meeting of the Measurement Team on Wednesday, September 2, 2015, 14:00 UTC in #tor-project https://www.timeanddate.com/worldclock/fixedtime.html?iso=20150902T14 Talk to you on Wednesday! All the best, Karsten -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJV5JhmAAoJEJD5dJfVqbCr69YH/3bpRZnJTfeavkfKKQDog8oq J6uqlFyEKpSl4S2uSc6jT/gwDfUfUuEwWMpXrOUrRzrFSCSQQcMQF8Hiu6Mk5+SD JN398wHGp2gTYp+y6xb6z8TzZfNmEEnJTcSAZtctQrB6kaj3jT29abMqIlsjQW9i F91uCJLJVJWrBhYvqvvFhUbdDhn3S8HYCEx0u6B8Gdd8q1p/Swi5IuSvkpStBHZJ diFGUsrj8HE73RKwOSl1gJcdLIGti3fayDhJZVVuTdV5weS4QF/I96XPJxnPwNqR ek2azn2RteYSMX3vhglkp86z7jKFM9V4sgDXU72S8jp0wWaXXYmbgpY4+v3G7EY= =9+9n -----END PGP SIGNATURE-----

1 0