- tor-dev - lists.torproject.org

Two major improvements to the Tor Metrics back-end
by Karsten Loesing 16 Feb '16

16 Feb '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello list, as some of you may have noticed, Tor Metrics had some trouble updating its aggregate data lately, and the issues there are by no means completely gone now. But I managed to tweak two of the aggregations that previously took many hours. Here's a graph: https://people.torproject.org/~karsten/volatile/metrics-modules-2016-01-07.… The green dots at the end are from the latest update. Changes affecting the 70-run-hidserv-stats.sh module: https://gitweb.torproject.org/metrics-web.git/commit/?id=c0f4c76 https://gitweb.torproject.org/metrics-web.git/commit/?id=6054ddd Changes affecting the 80-run-clients-stats.sh module: https://gitweb.torproject.org/metrics-web.git/commit/?id=44acfe5 Boom! All the best, Karsten -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJWjprFAAoJEJD5dJfVqbCrPT0IALX6dNrL3PK2gbbwAGmm4OBg 4kuz7qFMshTNFp8iT6HB5H9G9/wldgAPanZn+zLlFM0sOBoY1d16RoL2+ay+VKXQ xTeY85tgqMAnbaS2lQcBRZy5j8RXPui94g/ZcKnw9Bg3TpwLFO2+cMXqh1nhMLtb V+Lj0EE+y7qjdtOQO/IbOQzeUr/S9xtze44dRkGlZ0NmeKQsHXaR3m8w8ReIP32k vZekymZkUgbfzYeRoB+jI+3M3l0CF5agwIelhhHlnWwkJ70jPPVrAGX3rhb8X9SX iSNMUJTG62AC3WLqdRkhyY2YJ5OHSMLtpvQwOaOzqSs1yHWylHDVSLTZx+XCQmM= =tvfK -----END PGP SIGNATURE-----

3 4

Re: [tor-dev] Existing Tor Guard Selection Algorithm
by George Kadianakis 15 Feb '16

15 Feb '16

Reinaldo Junior <rjunior(a)thoughtworks.com> writes: > On Thu, Feb 11, 2016 at 7:55 AM, George Kadianakis <desnacked(a)riseup.net> > wrote: > >> Chelsea Komlo <ckomlo(a)thoughtworks.com> writes: >> >> > Hi George, >> > >> > Thanks for your help with this! >> > >> > We wrote up our high-level understanding of the current Tor guard >> selection >> > algorithm here: >> > >> > https://gist.github.com/chelseakomlo/2acbe15314b5a809c6f4 >> > >> > This has more than our python simulation, but less than the actual Tor >> > implementation. For example, it is missing conditions like prioritization >> > by uptime, capacity, etc. >> > >> > >> https://github.com/twstrike/tor_guardsim/blob/develop/lib/original_client.py >> > >> > If you wouldn't mind taking a look at this and letting us know anything >> > that is missing/should change, that would be really helpful. >> > >> >> Your pseudocode looks pretty accurate based on what I remember from the >> guard >> algorithm and from quickly skimming the code. >> >> One inaccuracy might be that if IIRC in RECEIVE_NEW_CONSENSUS we don't >> remove >> guards that are not listed in the latest consensus. Instead we mark them >> as 'bad' >> which rules them out of retries etc. till they reappear on the consensus. >> See entry_guard_set_status() and its caller for how this works. >> > > <snip> > > See: > https://github.com/twstrike/tor_guardsim/commit/1fb7400c0c8e167508753ede15b… Hmm, looks pretty good! One note: I feel that you are hiding information behind the simple phrase "Build LIVE_GUARDS". Interesting things happen in populate_live_entry_guards() that are not documented by your algo. For example, in entry_is_live() we call entry_is_time_to_retry(), which periodically retries guards based on time: struct guard_retry_period_s periods[] = { { 6*60*60, 60*60 }, /* For first 6 hrs., retry hourly; */ { 3*24*60*60, 4*60*60 }, /* Then retry every 4 hrs. until the 3-day mark; */ { 7*24*60*60, 18*60*60 }, /* After 3 days, retry every 18 hours until 1 week mark. */ { TIME_MAX, 36*60*60 } /* After 1 week, retry every 36 hours. */ }; Also, another function that I'm not sure if you have seen (and whether it fits with the algorithm) is entries_retry_all(). BTW, I like the way you have splitted the algo into various events (RECEIVE_NEW_CONSENSUS, BUILD_NEW_CIRCUIT, CHOOSE_A_GUARD, CONNECT_ENTRY_GUARD, etc.). We should maybe try to shape Ola's new algorithm to fit this event structure exactly since it resembles the actual Tor networking API. Cheers!

1 0

Possibly rescheduling monday meeting for prop#265
by Nick Mathewson 15 Feb '16

15 Feb '16

Hi! Looking at the schedule at https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/Meeting… , it does not appear that there is a time when Mike (the proposal author) could make it. I'll check my email in the morning, but it's possible we'll have to discuss prop#265 on another day.

1 0

Applications team prep for Valencia
by Isabela 14 Feb '16

14 Feb '16

Hello there! As we approach our 2016 Winter Tor's Dev Meeting in Valencia [1], I am trying to help the different teams prep for discussions we will have there. I hope the Applications team take some time to review your retrospective notes[2] to see what the team have done from that list and what it haven't. Also, to think about other points that could help improve the team organization process. We hope to have another retrospective session in Valencia and that will help prep for it. Another exercise I would like each dev team that plans to be in Valencia to do is to review the roadmaps they created in Berlin. We will be planning another 6 months roadmap (from Winter Dev meeting till Summer Dev meeting), it will be useful for the meeting to look at what you planned, what got done, what you want to carry on for this next 6months we will be planning and what you think its better to drop or leave for latter. I added Tor Browser and Tor Messenger roadmaps to the Applications wiki page [3]. I plan on adding other Applications dev teams stuff there too, if you need help finding the roadmap you did you let me know. Or if you need help with anything else. I am available on IRC as well if you have questions. Cheers, Isabela [1] https://trac.torproject.org/projects/tor/wiki/org/meetings/2016WinterDevMee… [2] https://trac.torproject.org/projects/tor/wiki/org/meetings/2015SummerDevMee… [3] https://trac.torproject.org/projects/tor/wiki/org/teams/ApplicationsTeam -- PM at TorProject.org gpg fingerprint = 8F2A F9B6 D4A1 4D03 FDF1 B298 3224 4994 1506 4C7B @isa

1 0

Proposal: Rendezvous Single Onion Services
by Tim Wilson-Brown - teor 12 Feb '16

12 Feb '16

Hi All, Please find below and attached a proposal: Rendezvous Single Onion Services. This is an updated and expanded version of "Direct Onion Services: Fast-but-not-hidden services”. It also borrows heavily from "Single Onion Services" (Proposal #252). The proposal is available in the branch “feature-17178-rsos” at https://github.com/teor2345/torspec.git <https://github.com/teor2345/torspec.git> as torspec/proposals/ideas/xxx-rend-single-onion.txt Work on this proposal is being tracked in trac ticket #17178 at https://trac.torproject.org/projects/tor/ticket/17178 <https://trac.torproject.org/projects/tor/ticket/17178> There is a basic implementation in the branch “feature-17178-rsos” at https://github.com/teor2345/tor.git <https://github.com/teor2345/tor.git> This can be tested with the chutney branch "feature-17178-rsos” at https://github.com/teor2345/chutney.git <https://github.com/teor2345/chutney.git> using the command: src/test/test-network.sh --flavor rsos-min Regards, Tim ----- Filename: xxx-rend-single-onion.txt Title: Rendezvous Single Onion Services Author: Tim Wilson-Brown, John Brooks, Aaron Johnson, Rob Jansen, George Kadianakis, Paul Syverson, Roger Dingledine Created: 2015-10-17 Status: Draft 1. Overview Rendezvous single onion services are an alternative design for single onion services, which trade service-side location privacy for improved performance, reliability, and scalability. Rendezvous single onion services have a .onion address identical to any other onion service. The descriptor contains the same information as the existing double onion (hidden) service descriptors. The introduction point and rendezvous protocols occur as in double onion services, with one modification: one-hop connections are made from the onion server to the introduction and rendezvous points. This proposal is a revision of the unnumbered proposal Direct Onion Services: Fast-but-not-hidden services by Roger Dingledine, and George Kadianakis at https://lists.torproject.org/pipermail/tor-dev/2015-April/008625.html It incorporates much of the discussion around hidden services since April 2015, including content from Single Onion Services (Proposal #252) by John Brooks, Paul Syverson, and Roger Dingledine. 2. Motivation Rendezvous single onion services are best used by sites which: * Don’t require location anonymity * Would appreciate lower latency or self-authenticated addresses * Would like to work with existing tor clients and relays * Can’t accept connections to an open ORPort Rendezvous single onion services have a few benefits over double onion services: * Connection latency is lower, as one-hop circuits are built to the introduction and rendezvous points, rather than three-hop circuits * Stream latency is reduced on a four-hop circuit * Less Tor network capacity is consumed by the service, as there are fewer hops (4 rather than 6) between the client and server via the rendezvous point Rendezvous single onion services have a few benefits over single onion services: * A rendezvous single onion service can load-balance over multiple rendezvous backends (see proposal #255) * A rendezvous single onion service doesn't need an accessible ORPort (it works behind a NAT, and in server enclaves that only allow outward connections) * A rendezvous single onion service is compatible with existing tor clients, hidden service directories, introduction points, and rendezvous points Rendezvous single onion services have a few drawbacks over single onion services: * Connection latency is higher, as one-hop circuits are built to the introduction and rendezvous points. Single onion services perform one extend to the single onion service’s ORPort only It should also be noted that, while single onion services receive many incoming connections from different relays, rendezvous single onion services make many outgoing connections to different relays. This should be taken into account when planning the connection capacity of the infrastructure supporting the onion service. Rendezvous single onion services are not location hidden on the service side, but clients retain all of the benefits and privacy of onion services. (The rationale for the 'single' and 'double' nomenclature is described in section 7.4 of proposal #252.) We believe that it is important for the Tor community to be aware of the alternative single onion service designs, so that we can reach consensus on the features and tradeoffs of each design. However, we recognise that each additional flavour of onion service splits the anonymity set of onion service users. Therefore, it may be best for user anonymity that not all designs are adopted, or that mitigations are implemented along with each additional flavour. (See sections 8 & 9 for a further discussion.) 3. Onion descriptors The rendezvous single onion descriptor format is identical to the double onion descriptor format. 4. Reaching a rendezvous single onion service as a client Clients reach rendezvous single onion services in an identical fashion to double onion services. The rendezvous design means that clients do not know whether they are talking to a double or rendezvous single onion service, unless that service tells them. (This may be a security issue.) However, the use of a four-hop path between client and rendezvous single onion service may be statistically distinguishable. (See section 8 for further discussion of security issues.) (Please note that this proposal follows the hop counting conventions in the tor source code. A circuit with a single connections between the client and the endpoint is one-hop, a circuit with 4 connections (and 3 nodes) between the client and endpoint is four-hop.) 5. Publishing a rendezvous single onion service To act as a rendezvous single onion service, a tor instance (or cooperating group of tor instances) must: * Publish onion descriptors in the same manner as any onion service, using three-hop circuits. This avoids service blocking by IP address, proposal #224 (next-generation hidden services) avoids blocking by onion address. * Perform the rendezvous protocol in the same manner as a double onion service, but make the intro and rendezvous connections one-hop. (This may allow intro and rendezvous points to block the service.) 5.1. Configuration options 5.1.1 RendezvousSingleOnionServiceNonAnonymousServer The tor instance operating a rendezvous single onion service must make one-hop circuits to the introduction and rendezvous points: RendezvousSingleOnionServiceNonAnonymousServer 0|1 If set, make one-hop circuits between the Rendezvous Single Onion Service server, and the introduction and rendezvous points. This option makes every onion service instance hosted by this tor instance a Rendezvous Single Onion Service. (Default: 0) Because of the grave consequences of misconfiguration here, we have added ‘NonAnonymous’ to the name of the torrc option. Furthermore, Tor MUST issue a startup warning message to operators of the onion service if this feature is enabled. [Should the name start with ‘NonAnonymous’ instead?] As RendezvousSingleOnionServiceNonAnonymousServer modifies the behaviour of every onion service on a tor instance, it is impossible to run hidden (double onion) services and rendezvous single onion services on the same tor instance. This is considered a feature, as it prevents hidden services from being discovered via rendezvous single onion services on the same tor instance. 5.1.2 Recommended Additional Options: Correctness Based on the experiences of Tor2Web with one-hop paths, operators should consider using the following options with every rendezvous single onion service, and every single onion service: UseEntryGuards 0 One-hop paths do not use entry guards. This also deactivates the entry guard pathbias code, which is not compatible with one-hop paths. Entry guards are a security measure against Sybil attacks. Unfortunately, they also act as the bottleneck of busy onion services and overload those Tor relays. LearnCircuitBuildTimeout 0 Learning circuit build timeouts is incompatible with one-hop paths. It also creates additional, unnecessary connections. Perhaps these options should be set automatically on (rendezvous) single onion services. Tor2Web sets these options automatically: UseEntryGuards 0 LearnCircuitBuildTimeout 0 5.1.3 Recommended Additional Options: Performance LongLivedPorts The default LongLivedPorts setting creates additional, unnecessary connections. This specifies no long-lived ports (the empty list). PredictedPortsRelevanceTime 0 seconds The default PredictedPortsRelevanceTime setting creates additional, unnecessary connections. RendPostPeriod 0 seconds This option typically hides the startup time of a hidden service by randomly posting over a 2 hour period. Since single onion services value speed over anonymity, they can post descriptors straight away. (Actually, 30 seconds after they bootstrap, for descriptor stability.) However, we do not recommend setting the following option to 1, unless bug #17359 is resolved so tor onion services can bootstrap without predicted circuits. __DisablePredictedCircuits 0 This option disables all predicted circuits. It is equivalent to: LearnCircuitBuildTimeout 0 LongLivedPorts PredictedPortsRelevanceTime 0 seconds And turning off hidden service server preemptive circuits, which is currently unimplemented (#17360) 5.1.3 Recommended Additional Options: Security We recommend that no other services are run on a rendezvous single onion service tor instance. Since tor runs as a client (and not a relay) by default, rendezvous single onion service operators should set: SocksPort 0 Disallow connections from client applications to the tor network via this tor instance. ClientOnly 1 Even if the defaults file configures this instance to be a relay, never relay any traffic or serve any descriptors. 5.2. Publishing descriptors A single onion service must publish descriptors in the same manner as any onion service, as defined by rend-spec. 5.3. Authorization Client authorization for a rendezvous single onion service is possible via the same methods used for double onion services. 6. Related Proposals, Tools, and Features 6.1. Load balancing High capacity services can distribute load and implement failover by: * running multiple instances that publish to the same onion service directories, * publishing descriptors containing multiple introduction points (OnionBalance), * publishing different introduction points to different onion service directories (OnionBalance upcoming(?) feature), * handing off rendezvous to a different tor instance via control port messages (proposal #255), or by a combination of these methods. 6.2. Ephemeral single onion services (ADD_ONION) The ADD_ONION control port command could be extended to support ephemerally configured rendezvous single onion services. Given that RendezvousSingleOnionServiceNonAnonymousServer modifies the behaviour of all onion services on a tor instance, if it is set, any ephemerally configured onion service should become a rendezvous single onion service. 6.3. Proposal 224 ("Next-Generation Hidden Services") This proposal is compatible with proposal 224, with onion services acting just like a next-generation hidden service, but making one-hop paths to the introduction and rendezvous points. 6.4. Proposal 246 ("Merging Hidden Service Directories and Intro Points") This proposal is compatible with proposal 246. The onion service will publish its descriptor to the introduction points in the same manner as any other onion service. Clients will use the merged hidden service directory and introduction point just as they do for other onion services. 6.5. Proposal 252 ("Single Onion Services") This proposal is compatible with proposal 252. The onion service will publish its descriptor to the introduction points in the same manner as any other onion service. Clients can then choose to extend to the single onion service, or continue with the rendezvous protocol. Running a rendezvous single onion service and single onion service allows older clients to connect via rendezvous, and newer clients to connenct via extend. This is useful for the transition period where not all clients support single onion services. 6.5. Proposal 255 ("Hidden Service Load Balancing") This proposal is compatible with proposal 255. The onion service will perform the rendezvous protocol in the same manner as any other onion service. Controllers can then choose to handoff the rendezvous point connection to another tor instance, which should also be configured as a rendezvous single onion service. 7. Considerations 7.1 Modifying RendezvousSingleOnionServiceNonAnonymousServer at runtime Implementations should not reuse introduction points or introduction point circuits if the value of RendezvousSingleOnionServiceNonAnonymousServer is different than it was when the introduction point was selected. This is because these circuits will have an undesirable length. There is specific code in tor that preserves introduction points on a HUP, if RendezvousSingleOnionServiceNonAnonymousServer has changed, all circuits should be closed, and all introduction points must be discarded. 7.2 Delaying connection expiry Tor clients typically expire connections much faster than tor relays [citation needed]. (Rendezvous) single onion service operators may find that keeping connections open saves on connection latency. However, it may also place an additional load on the service. (This could be implemented by increasing the configured connection expiry time.) 7.3. (No) Benefit to also running a Tor relay In tor Trac ticket #8742, running a relay and hidden onion service on the same tor instance was disabled for security reasons. While there may be benefits to running a relay on the same instance as a rendezvous single onion service (existing connections mean lower latency, it helps the tor network overall), a security analysis of this configuration has not yet been performed. In addition, a potential drawback is overloading a busy single onion service. 6.4 Predicted circuits We should look whether we can optimize further the predicted circuits that Tor makes as a onion service for this mode. 8. Security Implications 8.1 Splitting the Anonymity Set Each additional flavour of onion service, and each additional externally visible onion service feature, provides oportunities for fingerprinting. Also, each additional type of onion service shrinks the anonymity set for users of double onion (hidden) services who require server location anonymity. These users benefit from the cover provided by current users of onion services, who use them for client anonymity, self-authentication, NAT-punching, or other benefits. For this reason, features that shrink the double onion service anonymity set should be carefully considered. The benefits and drawbacks of additional features also often depend on a particular threat model. It may be that a significant number of users and sites adopt (rendezvous) single onion services due to their benefits. This could increase the traffic on the tor network, therefore increasing anonymity overall. However, the unique behaviour of each type of onion service may still be distinguishable from both the client and server ends of the connection. 8.2 Hidden Service Designs can potentially be more secure As a side-effect, by optimizing for performance in this feature, it allows us to lean more heavily towards security decisions for regular onion services. 8.3 One-hop onion service paths may encourage more attacks There's a possible second-order effect here since both encrypted services and hidden services will have foo.onion addresses and it's not clear based on the address whether the service will be hidden -- if *some* .onion addresses are easy to track down, are we encouraging adversaries to attack all rendezvous points just in case? 9. Further Work Further proposals or research could attempt to mitigate the anonymity-set splitting described in section 8. Here are some initial ideas. 9.1 Making Client Exit connections look like Client Onion Service Connections A mitigation to this fingerprinting is to make each (or some) exit connections look like onion service connections. This provides cover for particular types of onion service connections. Unfortunately, it is not possible to make onion service connections look like exit connections, as there are no suitable dummy servers to exit to on the Internet. 9.1.1 Making Client Exit connections perform Descriptor Downloads (Some) exit connections could perform a dummy descriptor download. (However, descriptors for recently accessed onion services are cached, so dummy downloads should only be performed occasionally.) Exit connections already involve a four-hop "circuit" to the server (including the connection between the exit and the server on the Internet). The server on the Internet is not included in the consensus. Therefore, this mitigation would effectively cover single onion services which are not relays. 9.1.2 Making Client Exit connections perform the Rendezvous Protocol (Some) exit connections could perform a dummy rendezvous protocol. Exit connections already involve a four-hop "circuit" to the server (including the connection between the exit and the server on the Internet). Therefore, this mitigation would effectively cover rendezvous single onion services, as long as a dummy descriptor download was also performed occasionally. 9.1.3 Making Single Onion Service rendezvous points perform name resolution Currently, Exits perform DNS name resolution, and changing this behaviour would cause unacceptable connection latency. Therefore, we could make onion service connections look like exit connections by making the rendezvous point do name resolution (that is, descriptor fetching), and, if needed, the introduction part of the protocol. This could potentially *reduce* the latency of single onion service connections, depending on the length of the paths used by the rendezvous point. However, this change makes rendezvous points almost as powerful as Exits, a careful security analysis will need to be performed before this is implemented. There is also a design issue with rendezvous name resolution: a client wants to leave resolution (descriptor download) to the RP, but it doesn't know whether it can use the exit-like protocol with an RP until it has downloaded the descriptor. This might mean that single onion services of both flavours need a different address style or address namespace. We could use .single.onion or something. (This would require an update to the HSDir code.) 9.2 Performing automated and common queries over onion services Tor could create cover traffic for a flavour of onion service by performing automated or common queries via an onion service of that type. In addition, onion service-based checks have security benefits over DNS-based checks. See Genuine Onion, Syverson and Boyce, 2015, at http://www.nrl.navy.mil/itd/chacs/syverson-genuine-onion-simple-fast-flexib… Here are some examples of automated queries that could be performed over an onion service: 9.2.1 torcheck over onion service torcheck ("Congratulations! This browser is configured to use Tor.") could be retrieved from an onion service. Incidentally, this would resolve the exitmap issues in #17297, but it would also fail to check that exit connections work, which is important for many Tor Browser users. 9.2.2 Tor Browser version checks over onion service Running tor browser version checks over an onion service seems to be an excellent use-case for onion services. It would also have the Tor Project "eating its own dogfood", that is, using onion services for its essential services. 9.2.3 Tor Browser downloads over onion service Running tor browser downloads over an onion service might require some work on the onion service codebase to support high loads, load-balancing, and failover. It is a good use case for a (rendezvous) single onion service, as the traffic over the tor network is only slightly higher than for Tor Browser downloads over tor. (4 hops for [R]SOS, 3 hops for Exit.) 9.2.4 SSL Observatory submissions over onion service HTTPS certificates could be submitted to HTTPS Everywhere's SSL Observatory over an onion service. This option is disabled in Tor Browser by default. Perhaps some users would be more comfortable enabling submission over an onion service, due to the additional security benefits. -----

2 2

Proposal: Load-balancing hidden services by splitting introduction from rendezvous
by Tom van der Woerdt 12 Feb '16

12 Feb '16

Hey all, I'd like your thoughts and comments on this proposal. Tom PS: If you want to deliver them in person, I'm in Berlin. Filename: xxx-intro-rendezvous-controlsocket.txt Title: Load-balancing hidden services by splitting introduction from rendezvous Author: Tom van der Woerdt Created: 2015-09-30 Status: draft 1. Overview and motivation To address scaling concerns with the onion web, we want to be able to spread the load of hidden services across multiple machines. OnionBalance is a great stab at this, and it can currently give us 60x the capacity by publishing 6 separate descriptors, each with 10 introduction points, but more is better. This proposal aims to address hidden service scaling up to a point where we can handle millions of concurrent connections. The basic idea involves splitting the 'introduce' from the 'rendezvous', in the tor implementation, and adding new events and commands to the control specification to allow intercepting introductions and transmitting them to different nodes, which will then take care of the actual rendezvous. External controller code could relay the data to another node or a pool of nodes, all which are run by the hidden service operator, effectively distributing the load of hidden services over multiple processes. By cleverly utilizing the current descriptor methods, we could publish up to sixty unique introduction points, which could translate to many thousands of parallel tor workers. This should allow hidden services to go multi-threaded, with a few small changes. 2. Specification We propose two additions to the control specification, of which one is an event and the other is a new command. We also introduce a new configuration option. 2.1. DisableAutomaticRendezvous configuration option The syntax is: "DisableAutomaticRendezvous" SP [1|0] CRLF This configuration option is defined to be a boolean toggle which, if set, stops the tor implementation from automatically doing a rendezvous when an INTRODUCE2 cell is received. Instead, an event will be sent to the controllers. If no controllers are present, the introduction cell should be dropped, as acting on it instead of dropping it could open a window for a DoS. For security reasons, the configuration should be made available only in the configuration files, and not as an option settable by the controller. 2.2. The "INTRODUCE" event The syntax is: "650" SP "INTRODUCE" SP RendezvousData CRLF RendezvousData = implementation-specific, but must not contain whitespace, must only contain human-readable characters, and should be no longer than 512 bytes The INTRODUCE event should contain sufficient data to allow continuing the rendezvous from another Tor instance. The exact format is left unspecified and left up to the implementation. From this follows that only matching versions can be used safely to coordinate the rendezvous of hidden service connections. 2.3. "PERFORM-RENDEZVOUS" command The syntax is: "PERFORM-RENDEZVOUS" SP RendezvousData CRLF This command allows a controller to perform a rendezvous using data received through an INTRODUCE event. The format of RendezvousData is not specified other than that it must not contain whitespace, and should be no longer than 512 bytes. 3. Compatibility and security The implementation of these methods should, ideally, not change anything in the network, and all control changes are opt-in, so this proposal is fully backwards compatible. Controllers handling this data must be careful to not leak rendezvous data to untrusted parties, as it could be used to intercept and manipulate hidden services traffic. 4. Example Let's take an example where a client (Alice) tries to contact Bob's hidden service. To do this, Bob follows the normal hidden service specification, except he sets up ten servers to do this. One of these publishes the descriptor, the others have this desabled. When the INTRODUCE2 cell arrives at the node which published the descriptor, it does not immediately try to perform the rendezvous, but instead outputs this to the controller. Through an out-of-band process this message is relayed to a controller of another node of Bob's, and this transmits the "PERFORM-RENDEZVOUS" command to that node. This node finally performs the rendezvous, and will continue to serve data to Alice, whose client will now not have to talk to the introduction point anymore. 5. Other considerations We have left the actual format of the rendezvous data in the control protocol unspecified, so that controllers do not need to worry about the various types of hidden service connections, most notably proposal 224. The decision to not implement the actual cell relaying in the tor implementation itself was taken to allow more advanced configurations, and to leave the actual load-balancing algorithm to the implementor of the controller. The developer of the tor implementation should not have to choose between a round-robin algorithm and something that could pull CPU load averages from a centralized monitoring system.

5 9

Existing Tor Guard Selection Algorithm
by Chelsea Komlo 11 Feb '16

11 Feb '16

Hi George, Thanks for your help with this! We wrote up our high-level understanding of the current Tor guard selection algorithm here: https://gist.github.com/chelseakomlo/2acbe15314b5a809c6f4 This has more than our python simulation, but less than the actual Tor implementation. For example, it is missing conditions like prioritization by uptime, capacity, etc. https://github.com/twstrike/tor_guardsim/blob/develop/lib/original_client.py If you wouldn't mind taking a look at this and letting us know anything that is missing/should change, that would be really helpful. Thanks! Chelsea

2 1

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 11 Feb '16

11 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hi, > >> So maybe the simple answer here is that if prop247 is enabled (this could be a >> NumGuards=N argument to our algorithm), instead of always returning the first >> reachable guard, we instead build a list of the first N reachable guards, and >> randomly choose one of them. Could this work? (see #12466 for a weird behavior >> that will make us skip bugs if NumEntryGuards != 1) > Ah, I understand. We could definitely do that. I think the easiest way > is to just have a parameter to the existing algorithm with > ExcludeNodes. We can then repeat it N times sending in the previous > nodes chosen as ExcludeNodes. This will generate the N reachable first > guards. Then, we can just choose randomly from them. > > Does that sound reasonable? > Yes, that sounds like a reasonable way to achieve this. Maybe N_PRIMARY_GUARDS should also scale up in that case, so that N_PRIMARY_GUARDS >= N.

2 1

Re: [tor-dev] Detailed algorithm
by George Kadianakis 10 Feb '16

10 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > OK, with your feedback and thinking a bit more about it, here is a > revision of the algorithm from yesterday. I think we are starting to > get close so we will rip out the original simulation code and > implement something that matches this now. Hopefully, the changes will > be smaller. > > - Start of algorithm (arguments: USED_GUARDS, EXCLUDE_NODES) > - If selecting directory guards, GUARDS is all guards from the consensus with the V2Flag, > otherwise GUARDS is all guards from the consensus > - Set UTOPIC_GUARDS to be all guards to use under utopic conditions from GUARDS > - Set DYSTOPIC_GUARDS to be all guards to use under dystopic conditions from GUARDS > - Set REMAINING_UTOPIC_GUARDS to be UTOPIC_GUARDS without EXCLUDE_NODES > - Set REMAINING_DYSTOPIC_GUARDS to be DYSTOPIC_GUARDS without EXCLUDE_NODES > - Create a list of PRIMARY_GUARDS that contain N_PRIMARY_GUARDS that are not bad by: > - Taking the next entry from USED_GUARDS > - If USED_GUARDS is empty: > - randomly select an entry from UTOPIC_GUARDS, weighted by bandwidth > - Set TRIED_GUARDS to be an empty set > - Set TRIED_DYSTOPIC_GUARDS to be an empty set > - Set state = STATE_PRIMARY_GUARDS > > - Each iteration of algorithm > - If a new consensus has arrived: > - Update all guard profiles with new bad/non-bad information > - If any PRIMARY_GUARDS have become bad: > - re-add to the list of PRIMARY_GUARDS using the same procedure > - If any USED_GUARDS have become non-bad: > - add it back to PRIMARY_GUARDS at the place it would have been if > it was non-bad when running the start of the algorithm. If this > results in PRIMARY_GUARDS being larger than N_PRIMARY_GUARDS, > remove from the end of the list until the list is N_PRIMARY_GUARDS long > - Ensure that UTOPIC_GUARDS and DYSTOPIC_GUARDS are updated with the changes > from the consensus > > - If it was at least 3 minutes since we tried the primary guards and we are not in STATE_PRIMARY_GUARDS: > - save previous state > - set state = STATE_PRIMARY_GUARDS > > - STATE_PRIMARY_GUARDS: > - return each entry in PRIMARY_GUARDS in turn > - mark each entry as "unreachable" if algorithm doesn't terminate > - restore previous state (or STATE_TRY_UTOPIC if no previous) > > - STATE_TRY_UTOPIC: > - for each entry in TRIED_GUARDS that was marked as unreachable more than 20 minutes ago > - add it back to REMAINING_UTOPIC_GUARDS > - return each remaining entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > - return each entry from REMAINING_UTOPIC_GUARDS randomly selected, weighted by bandwidth > - remove the returned entry from REMAINING_UTOPIC_GUARDS > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > > - STATE_TRY_DYSTOPIC: > - for each entry in TRIED_DYSTOPIC_GUARDS that was marked as unreachable more than 20 minutes ago > - add it back to REMAINING_DYSTOPIC_GUARDS > - return each remaining DYSTOPIC entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_DYSTOPIC_GUARDS > - if the number of entries in TRIED_GUARDS+TRIED_DYSTOPIC_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_DYSTOPIC_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of DYSTOPIC_GUARDS: > - mark all guards in PRIMARY_GUARDS, TRIED_GUARDS and TRIED_DYSTOPIC_GUARDS as not "unreachable" > - return failure from the algorithm > - return each entry from REMAINING_DYSTOPIC_GUARDS randomly selected, weighted by bandwidth > - remove the returned entry from REMAINING_DYSTOPIC_GUARDS > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_DYSTOPIC_GUARDS > - if the number of entries in TRIED_GUARDS+TRIED_DYSTOPIC_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_DYSTOPIC_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of DYSTOPIC_GUARDS: > - mark all guards in PRIMARY_GUARDS, TRIED_GUARDS and TRIED_DYSTOPIC_GUARDS as not "unreachable" > - return failure from the algorithm > > - End of algorithm > - If circuit is set up correctly, let algorithm know > - Algorithm marks the guard chosen as used and makes sure it is in USED_GUARDS > - Otherwise do another run of the algorithm > Ah great! Good improvements. I think we are going the right way. Here is another quick review. I include a second copy of the algorithm and comment inline: > > - Start of algorithm (arguments: USED_GUARDS, EXCLUDE_NODES) > - If selecting directory guards, GUARDS is all guards from the consensus with the V2Flag, > otherwise GUARDS is all guards from the consensus > - Set UTOPIC_GUARDS to be all guards to use under utopic conditions from GUARDS > - Set DYSTOPIC_GUARDS to be all guards to use under dystopic conditions from GUARDS > - Set REMAINING_UTOPIC_GUARDS to be UTOPIC_GUARDS without EXCLUDE_NODES > - Set REMAINING_DYSTOPIC_GUARDS to be DYSTOPIC_GUARDS without EXCLUDE_NODES Maybe we also need to exclude USED_GUARDS from these two lists? > - Create a list of PRIMARY_GUARDS that contain N_PRIMARY_GUARDS that are not bad by: > - Taking the next entry from USED_GUARDS > - If USED_GUARDS is empty: > - randomly select an entry from UTOPIC_GUARDS, weighted by bandwidth > - Set TRIED_GUARDS to be an empty set > - Set TRIED_DYSTOPIC_GUARDS to be an empty set > - Set state = STATE_PRIMARY_GUARDS > > - Each iteration of algorithm > - If a new consensus has arrived: > - Update all guard profiles with new bad/non-bad information > - If any PRIMARY_GUARDS have become bad: > - re-add to the list of PRIMARY_GUARDS using the same procedure > - If any USED_GUARDS have become non-bad: > - add it back to PRIMARY_GUARDS at the place it would have been if > it was non-bad when running the start of the algorithm. If this > results in PRIMARY_GUARDS being larger than N_PRIMARY_GUARDS, > remove from the end of the list until the list is N_PRIMARY_GUARDS long > - Ensure that UTOPIC_GUARDS and DYSTOPIC_GUARDS are updated with the changes > from the consensus > Not sure if this is part of this algorithm, or it's actually another helper algorithm that is called when a consensus arrives. I feel it might be cleaner if we do it as a separate algo, but we can proceed like this as well since it's not too confusing. > - If it was at least 3 minutes since we tried the primary guards and we are not in STATE_PRIMARY_GUARDS: > - save previous state > - set state = STATE_PRIMARY_GUARDS > > - STATE_PRIMARY_GUARDS: > - return each entry in PRIMARY_GUARDS in turn > - mark each entry as "unreachable" if algorithm doesn't terminate So IIUC the "algorithm" referenced here is _not_ the algorithm that we are describing right now (let's call it ALGO_CHOOSE_GUARD). Instead the "algorithm" here is the _caller of ALGO_CHOOSE_GUARD_, which is the algorithm responsible for creating circuits, testing whether they work and reporting the results back to CHOOSE_GUARD_ALGO (let's call this other algorithm ALGO_BUILD_CIRCUIT). Maybe we can clarify this? Also, do PRIMARY_GUARDS go into TRIED_GUARDS and count against our thresholds? > - restore previous state (or STATE_TRY_UTOPIC if no previous) > > - STATE_TRY_UTOPIC: > - for each entry in TRIED_GUARDS that was marked as unreachable more than 20 minutes ago > - add it back to REMAINING_UTOPIC_GUARDS > - return each remaining entry from USED_GUARDS in turn > - for each entry, if algorithm doesn't terminate In general, I think we should make the context of this algorithm (ALGO_CHOOSE_GUARD) a bit clearer, so that we know when "Start of algorithm" is supposed to run, and when "End of algorithm" is supposed to run. Because for example one could think here that in an asynchronous setting, when we try a entry from USED_GUARDS and find out whether the circuit succeeded or not, we need to run the algorithm from the beginning including the "Start of algorithm" step. Whereas if I understand correctly you assume that we will just drop in and continue exactly from this point. I feel that this confusion is caused by the ALGO_CHOOSE_GUARD algorithm being pseudo-asynchronous. To address this I suggested additional states for when we are waiting for the results of a circuit construction, but I agree that this might complicate things too much. Is there a way we can make this cleaner? > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > - return each entry from REMAINING_UTOPIC_GUARDS randomly selected, weighted by bandwidth > - remove the returned entry from REMAINING_UTOPIC_GUARDS > - for each entry, if algorithm doesn't terminate > - mark entry as "unreachable" > - add entry to TRIED_GUARDS > - if the number of entries in TRIED_GUARDS that were tried within GUARDS_TRY_THRESHOLD_TIME > is larger than GUARDS_TRY_THRESHOLD, return failure from the algorithm > - if the number of entries in TRIED_GUARDS is larger than a GUARDS_FAILOVER_THRESHOLD > proportion of UTOPIC_GUARDS, set state = STATE_TRY_DYSTOPIC > > <snip> Finally, what kind of statistics and measurements does the simulator conduct? For example, I can think of reachability related stats like: * Time (or number of guards) till we manage to build first circuit * Time till we manage to recover from flaky network and I can also think of security related stats like: * Number of guards we tried before succeeding first circuit * Number of guards we exposed ourselves to after time t etc.

2 1

Re: [tor-dev] Entry guards, primary guards, dir guards
by George Kadianakis 10 Feb '16

10 Feb '16

Ola Bini <obini(a)thoughtworks.com> writes: > Hey, > > Maybe I misunderstood the hard part - I thought the problem was to > choose the NUM longlived vanguards - since there are only ever NUM > possible guards at each level, not to choose which one to use among > the NUM guards. For the first, it felt like using 259 would work, and > the for the second use case, just randomly choose one of the NUM > guards. > Hmm, let's try to formalize this slightly. Maybe it's helpful. I'm doing this a bit rushed, so maybe I'm wrong. So far we've been designing an algorithm that: a) populates and manages our guardlist b) picks the right guard to use from our guardlist We could in theory decouple these two steps, but so far we've been mixing them together. Step (b) has been easy so far because Tor uses a single guard node for general circuits, so we always pick the very first reachable guard from our guardlist every time. However, when we consider prop247 or directory guards we don't always want to pick the very first available guard. For example, when considering the layer-3 guards of prop247, we want to pick amongst N guards everytime we build a circuit so that we load balance our traffic amongst the guards. When considering directory guards, we want to pick randomly amongst the first 3 directory guards every time, so that if the first guard lied to us, we will also try the second one who might not lie. So maybe the simple answer here is that if prop247 is enabled (this could be a NumGuards=N argument to our algorithm), instead of always returning the first reachable guard, we instead build a list of the first N reachable guards, and randomly choose one of them. Could this work? (see #12466 for a weird behavior that will make us skip bugs if NumEntryGuards != 1) > The other alternative is to simply initialize USED_GUARDS to be > layer-2 guardlist and run the algorithm, then init USED_GUARDS to be > layer-3 guardlist and run the algorithm. Makes sense or am I missing > something? > Hmm, in this case we would just use the first reachable guard from USED_GUARDS. We would only reach other guards if the first guards are unreachable. That will not work for load balancing prop247 for example. (got to go. will be back later!)

1 0