October 2019 - tor-dev - lists.torproject.org

the consequences of deprecating debian alpha repos with every new major branch
by nusenu 21 Dec '21

21 Dec '21

Hi, tldr: - more outdated relays (that is a claim I'm making and you could easily proof me wrong by recreating the 0.3.3.x alpha repos and ship 0.3.3.7 in them and see how things evolve after a week or so) - more work for the tpo website maintainer - less happy relay operators [3][4] - more work for repo maintainers? (since a new repo needs to be created) When the tor 0.3.4 alpha repos (deb.torproject.org) first appeared on 2018-05-23 I was about to submit a PR for the website to include it in the sources.list generator [1] on tpo but didn't do it because I wanted to wait for a previous PR to be merged first. The outstanding PR got merged eventually (2018-06-28) but I still did not submit a PR to update the repo generator for 0.3.4.x nonetheless and here is why. Recently I was wondering why are there so many relays running tor version 0.3.3.5-rc? (see OrNetStats or Relay Search) (> 3.2% CW fraction) Then I realized that this was the last version the tor-experimental-0.3.3.x-* repos were shipping before they got abandoned due to the new 0.3.4.x-* repos (I can no longer verify it since they got removed by now). Peter made it clear in the past that the current way to have per-major-version debian alpha repos (i.e. tor-experimental-0.3.4.x-jessie) will not change [2]: > If you can't be bothered to change your sources.list once or twice a > year, then you probably should be running stable. but maybe someone else would be willing to invoke a "ln" commands everytime a new new alpha repo is born. tor-alpha-jessie -> tor-experimental-0.3.4.x-jessie once 0.3.5.x repos are created the link would point to tor-alpha-jessie -> tor-experimental-0.3.5.x-jessie It is my opinion that this will help reduce the amount of relays running outdated versions of tor. It will certainly avoid having to update the tpo website, which isn't a big task and could probably be automated but it isn't done currently. "..but that would cause relay operators to jump from i.e. 0.3.3.x to 0.3.4.x alphas (and break setups)!" Yes, and I think that is better than relays stuck on an older version because the former repo no longer exists and operators still can choose the old repos which will not jump to newer major versions. [1] https://www.torproject.org/docs/debian.html.en#ubuntu [2] https://trac.torproject.org/projects/tor/ticket/14997#comment:3 [3] https://lists.torproject.org/pipermail/tor-relays/2018-June/015549.html [4] https://trac.torproject.org/projects/tor/ticket/26474 -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu

4 7

tor relay process health data for operators (controlport)
by nusenu 16 Nov '20

16 Nov '20

Hi, every now and then I'm in contact with relay operators about the "health" of their relays. Following these 1:1 discussions and the discussion on tor-relays@ I'd like to rise two issues with you (the developers) with the goal to help improve relay operations and end user experience in the long term: 1) DNS (exits only) 2) tor relay health data 1) DNS ------ Current situation: Arthur Edelstein provides public measurements to tor exit relay operators via his page at: https://arthuredelstein.net/exits/ This page is updated once daily. the process to use that data looks like this: - first they watch Arthur's measurement results - if their failure rate is non-zero they try to tweak/improve/change their setup - wait for another 24 hours (next measurement) This is a somewhat suboptimal and slow feedback loop and is probably also less accurate and less valuable data when compared to the data the tor process can provide. Suggestion for improvement: Exposes the following DNS status information via tor's controlport to help debug and detect DNS issues on exit relays: (total numbers since startup) - amount of DNS queries send to the resolver - amount of DNS queries send to the resolver due to a RESOLVE request - DNS queries send to resolver due to a reverse RESOLVE request - amount of queries that did not result in any answer from the resolver - breakdown of number of responses by response code (RCODE) https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-pa… - max amount of DNS queries send per curcuit If this causes a significant performance impact this feature should be disabled by default. 2) general relay health metrics -------------------------------- Compared to other server daemons (webserver, DNS server, ..) tor provides little data for operators to detect operational issues and anomalies. I'd suggest to provide the following stats via the control port: (most of them are already written to logfiles by default but not accessible via the controlport as far as I've seen) - total amount of memory used by the tor process - amount of currently open circuits - circuit handshake stats (TAP / NTor) DoS mitigation stats - amount of circuits killed with too many cells - amount of circuits rejected - marked addresses - amount of connections closed - amount of single hop clients refused - amount of closed/failed circuits broken down by their reason value https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n1402 https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1994 - amount of closed/failed OR connections broken down by their reason value https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n2205 If this causes a significant performance impact this feature should be disabled by default. cell stats - extra info cell stats as defined in: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n1072 This data should be useful to answer the following questions: - High level questions: Is the tor relay healthy? - is it hitting any resource limits? - is the tor process under unusual load? - why is tor using more memory? - is it slower than usual at handling circuits? - can the DNS resolver handle the amount of DNS queries tor is sending it? This data could help prevent errors from occurring or provide additional data when trying to narrow down issues. When it comes to the question: **Is it "safe" to make this data accessible via the controlport?** I assume it is safe for all information that current versions of tor writes to logfiles or even publishes as part of its extra info descriptor. Should tor provide this or similar data I'm planing to write scripts for operators to make use of that data (for example a munin plugin that connects to tor's controlport). I'm happy to help write updates for control-spec should these features seem reasonable to you. Looking forward to hearing your feedback. nusenu -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu

5 7

DNS-over-HTTPS (DOH) in Firefox/Torbrowser
by nusenu 01 Jul '20

01 Jul '20

Hi, since Mozilla did tests [0] on DOH [1] in Firefox I was wondering if Torbrowser developers have put any thought into that as well? Note: I'm _not_ suggesting to use DOH in torbrowser I'm just asking because the answer probably matters for exit documentation in the relay guide if clients do DNS themselves over TCP connections instead of relying on the exit (even if torbrowser is not the only tor client). thanks, nusenu [0] https://www.ghacks.net/2018/03/20/firefox-dns-over-https-and-a-worrying-shi… [1] https://datatracker.ietf.org/doc/draft-hoffman-dns-over-https/ -- https://mastodon.social/@nusenu twitter: @nusenu_

4 4

Lets give every circuit its own exit IP?
by nusenu 18 Mar '20

18 Mar '20

The unbearable situation with Google's reCAPTCHA motivated this email (but it is not limited to this specific case). This idea came up when seeing a similar functionality in unbound (which has it for a different reason). Assumption: There are systems that block some tor exit IP addresses (most likely the bigger once), but they are not blocked due to the fact that they are tor exits. It just occurred that the IP got flagged because of "automated / malicious" requests and IP reputation systems. What if every circuit had its "own" IP address at the exit relay to avoid causing collateral damage to all users of the exit if one was bad? (until the exit runs out of IPs and starts to recycle previously used IPs again) The goal is to avoid accumulating a bad "reputation" for the single used exit IP address that affects all tor users of that exit. Instead of doing it on the circuit level you could do it based on time. Change the exit IP every 5 minutes (but do _not_ change the exit IPs for _existing_ circuits even if they live longer than 5 minutes). Yes, no one has that many IPv4 addresses but with the increasing availability of IPv6 at exits and destinations, this could be feasible to a certain extend, depending on how many IPv6 addresses the exit operator has. There are exit operators that have entire /48 IPv6 blocks. problems: - will not solve anything since reputation will shift to netblocks as well (How big of a netblock are you willing to block?) - you can tell two tor users easily apart from each other even if they use the same exit (or more generally: you can tell circuits apart). There might be all kinds of bad implications that I'm not thinking off right now. - check.tpo would no longer be feasible - how can do we still provide the list of exit IPs for easy blocking? Exits could signal their used netblock via their descriptor. What if they don't? (that in turn opens new kinds of attacks where an exit claims to be /0 and the target effectively blocks everything) - more state to track and store at the exit -... some random thoughts, nusenu -- https://mastodon.social/@nusenu twitter: @nusenu_

4 6

HSv3 descriptor work in stem
by George Kadianakis 04 Dec '19

04 Dec '19

Hello atagar, I'm starting this thread to ask you questions about stem and the HSv3 work we've been doing over email so that we don't do it over IRC. Here is an initial question: I'm working on HSv3 descriptor encoding, and I'm trying to understand how `_descriptor_content()` works. In particular, I want to compute the signature of a descriptor, but I see that `descriptor_content()` fills it with random bytes in all the `content()` methods I managed to find: ('signature', _random_crypto_blob('SIGNATURE')), What's the right way to compute the signature for such objects? In particular, I would need a method that first generates the whole descriptor body, and then computes the signature of that with a given private key. Can I use `_descriptor_content()` to do that? Or should I call `_descriptor_content()` to generate the whole thing _without_ the sig, and then do the signature computation on its result and concatenate it after? Thanks! :)

3 10

Proposal 271 - improvements
by Florentin Rochet 26 Nov '19

26 Nov '19

Hello, I'm doing research on improving Tor security via path selection. This work is with Ryan Wails, Aaron Johnson, Prateek Mittal, and Olivier Pereira. In our work, we ran into load-balancing problems that affected our experiments. We identified the problem to originate in the specification and implementation of Proposal 271, "Another algorithm for guard selection", which was implemented in tor-0.3.0.1-alpha ~2 years ago and is thus now in tor stable. In short, Prop 271 causes guards to be selected with probabilities different than their weights due to the way it samples many guards and then chooses primary guards from that sample. We are suggesting a straightforward fix to the problem, which is, roughly speaking, to choose primary guards in the order in which they were sampled. We have created a patch implementing this fix for the case affecting our experiments, which would improve the current situation. We are further suggesting that Tor apply the technique throughout the guard-selection logic. In more detail, Prop 271 chooses guards via a multi-step process: 1. It chooses 20 distinct guards (and sometimes more) by sampling without replacement with probability proportional to consensus weight. 2. It produces two subsets of the sample: (1) "filtered" guards, which are guards that satisfy various torrc constraints, and (2) "confirmed" guards, which are guards through which a circuit has been constructed. 3. The "primary" guards (i.e. the actual guards used for circuits) are chosen from the confirmed and filtered subsets. I'm ignoring the additional "usable" subsets for clarity. This description is based on Section 4.6 of the specification (https://gitweb.torproject.org/torspec.git/tree/guard-spec.txt) The primary guards are selected uniformly at random from the filtered guards when no confirmed guards exist. No confirmed guards appear to exist until some primary guards have been selected, and so when Tor is started the first time the primary guards always come only from the filtered set. The uniformly-random selection causes a bias in primary-guard selection away from consensus weights and towards a more uniform selection of guards. As just an example of the problem, if there were only 20 guards in the network, the sampled set would be all guards and primary guard selection would be entirely uniformly random, ignoring weights entirely. This bias is worse the larger the sampled set is relative to the entire set of guards, and it has a significant effect on Tor simulations, which are typically on smaller networks. We believe that this issue has only a limited effect on Tor currently due to the relatively large number of guards. However, the issue is potentially worse when there exist confirmed guards. In this case, primary guards are chosen from the intersection of confirmed and filtered guards in the order they were added to the confirmed set. That order seems to be the order in which circuits were successfully created through the guards, which would thus be both somewhat non-deterministic (because it depends on when a circuit successfully finishes) and random (because circuits are only attempted through primary guards, which are initially uniformly-random filtered guards). The non-determinism especially could yield guard selection probabilities that deviate quite a bit from the consensus weights. This design has both performance and security implications. It potentially reduces the performance of the Tor network by making the load unbalanced. It also affects the correctness of performance analyses done with Shadow simulations, and the error is likely much larger due to the smaller size of simulation networks, which commonly include 100 or fewer guards. The design also reduces Tor's security by increasing the number of clients that an adversary running small relays can observe. In addition, an adversary has to wait less time than it should after it starts a malicious guard to be chosen by a client. This weakness occurs because the malicious guard only needs to enter the sampled list to have a chance to be chosen as primary, rather than having to wait until all previously-sampled guards have already expired. We propose a solution that fits well within the existing guard-selection algorithm. Our solution is to select primary guards in the order they were sampled. This ordering should be applied after the filtering and/or confirmed guard sets are constructed as normal. That is, primary guards should be selected from the filtered guards (if no guards are both confirmed and filtered) or from the set of confirmed and filtered guards (if such guards exist) in the order they were initially sampled. This solution guarantees that each primary guard is selected (without replacement) from all guards with a probability that is proportional to its consensus weight. We include a patch that applies this ordering only when primary guards are selected from filtered guards. This fixes the problem in Shadow simulations because confirmed guards never exist when primary guards are being selected. However, to solve all issues in the real Tor network, we recommend that you apply similar logic to the case that primary guards are selected from confirmed guards. Note that the issue of sampling skewing guard selection seems to have been raised in proposal discussions, which is documented in the proposal (although the proposed solution is less practical): > [Paul Syverson in a conversation at the Wilmington Meeting 2017 says that > we should look into how we're doing this sampling. Essentially, his > concern is that, since we are sampling by bandwidth at first (when we > choose the `sampled` set), then later there is another bias—when trying to > build circuits (and hence marking guards as confirmed) we select those > which completed a usable circuit first (and hence have the lowest > latency)—that this sort of "doubly skewed" selection may "snub" some > low-consensus-weight guards and leave them unused completely. Thus the > issue is primarily that we're not allocating network resources > efficiently. Mine and Nick's guard algorithm simulation code never > checked what percentage of possible guards the algorithm reasonably > allowed clients to use; this would be an interesting thing to check in > simulation at some point. If it does turn out to be a problem, Paul's > intuition for a fix is to select uniformly at random to obtain the > `sampled` set, then weight by bandwidth when trying to build circuits and > marking guards as confirmed. —isis] Best, Florentin

3 3

Onion Service Intro Point Retry Behavior
by David Goulet 01 Nov '19

01 Nov '19

Greetings everyone, After some discussions between arma, mikeperry, asn and I, it is time for a famous tor-dev@ email thread to try to get to a consensus if need be. In the past weeks, a set of HSv3 reachability issues have been found regarding *service* intro points (IP): - hs-v3: Service can keep unused intro points in its list https://bugs.torproject.org/31561 - hs-v3: Service can pick more than HiddenServiceNumIntroductionPoints intro points https://bugs.torproject.org/31548 - hs-v3: Stop using ip->circuit_established flag https://bugs.torproject.org/32094 - hs-v3: Client can re-pick bad intro points https://bugs.torproject.org/31541 - hs-v3: Service circuit retry limit should not close a valid circuit https://bugs.torproject.org/31652 Long story short, couple weeks ago we've almost merged a new behavior on the service side with #31561 that would have ditch an intro point if its circuit would time out instead of retrying it. (Today, a service always retry their intro point up to 3 times on any type of circuit failure.) And here comes the core of the discussion of this thread: Retrying intro point on failure or simply ditch it on failure and pick a new one? Some 7 years ago, this ticket was created and thus we implemented roughly 4 years ago a mechanism that makes a service retry to establish the intro point circuit up to 3 times when it collapses (except for very very specific cases for which we wouldn't): https://bugs.torproject.org/8239 HSv3 tried to be on feature parity there with v2 up until now that the above bugs have been mostly fixed. That being all said, regarding the retry feature, there are pros and cons. I'll try to organize them below based on many adhoc discussions in the past and what I can get from all the tickets up to this day (there could be more! this is just what I could recall and find in the tickets): == Pros == The primary original argument for retrying is based on the mobile use case. If a .onion is running on a cellphone and the network happens to be bad all the sudden, the service is better off to re-establish the intro circuits which would make the retry attempts of the client to finally succeed after a bit instead of having to re-fetch a descriptor and go to the new intro points. Thus, in theory, it is mostly a reachability argument. One question that can arise from this is: Will the client be able to reconnect using the old intro points by the time the service re-established? In other words, is the retry behavior of the *client* allows enough time for the service to stabilize for the mobile use case? I'm curious to learn from people with experience with this! == Cons == Recently, mikeperry raised concerns about the retry behavior all together and proposed to simply ditch each time the intro point instead of retrying. (@Mike, I do invite you to comment here as you mentionned many times rationales for this but I don't have enough IRC backlog :S). == Pros _and_ Cons at the same time == There is a possible Guard discovery attack argument against retrying. But it is nuanced on what exactly constitute a failure and when should it retry vs ditching. Quote from https://trac.torproject.org/projects/tor/ticket/8239#comment:6 FWIW, it's also worth mentioning that making HSes more stubborn towards old IPs might also allow guard discovery attacks from the IP. That is the IP kills incoming circuits, till a compromised middle node is selected, and since the HS is stubborn it will keep on establishing new circuits. This was mentioned by waldo here: https://lists.torproject.org/pipermail/tor-dev/2014-May/006843.html ... which is where the "what is the failure" is important as arma's mentions in the same ticket: That's why you should only stick to your intro point when it's your network that failed (that is, the connection between you and your guard), not the intro circuit. (This is what I meant in the body of the bug in the 'main tricky point' sentence.) We had this discussion before in Tor many times on "how to detect network failures" vs "circuit failures". In other words, if the link to your Guard fails, that would be enough to consider a network failure and thus retry the intro point. But if the circuit collapses due to let say a DESTROY or TRUNCATED cell, then it could be the IP closing it for the purpose of an attack and thus you would select a new intro point. But, it could also be that the middle node died... That one has many false positive. Soooooo, to repeat what I first said at the beginning, today an HSv3 will _always_ retry up to 3 times regardless of the reason why the circuit collapsed. Should that behavior get more refined with the network failed vs circuit close argument? Should we stop at once retrying? Should we change the retry behavior client side to better match the latency of the mobile use case? Whatever we decide, most importantly, we need to document the *why* of this retry/ditch behavior and thus this email thread is I hope a good start to keep a record of the discussions/arguments. Cheers! David -- 5qZaRu0+AqSNqiaTmTpzcIEztqeYQIq7AAfzKdg/2cs=

2 1

New python Tor client implementation
by James Brown 24 Oct '19

24 Oct '19

4 4

Comparing Proposals 295 and 308
by Tomer Ashur 23 Oct '19

23 Oct '19

Dear all, Some time ago I sent to this mailing list a proposal for using the ADL construction to solve the crypto tagging attack and it was registered as Proposal 295. Then, about a month ago Jean Paul Degabriele sent another proposal aiming for the same, which was registered as Proposal 308. We’ve now had the chance to compare both proposals and we provide our observations below. But before we discuss the pros and cons of each proposal, I’d like to re-state our goal for Proposal 295. Our aim was to build something that: 1. does not lose any security guarantees that are already in place; 2. prevents successful crypto-tagging; and 3. does not introduce new weaknesses. We *did not* consider advanced security goals such as forward secrecy and/or non-repudiation which was also mentioned earlier on this mailing list. In achieving these goals, the two proposals are almost the same: for the encryption part, both use layered encryption where the nonce is tweaked with a digest of the ciphertext, and sent in encrypted form to the next node as part of the ciphertext. The only meaningful difference I could find is that instead of using the output of the universal hash function (i.e., GHASH) as a running digest as is done in Proposal 295, Proposal 308 uses the encrypted nonce. Jean Paul made the correct observation that our security proof did not account for key-dependent input, but we believe that this can be resolved by rewriting the proof. In either case, this is a subtlety and common ground can be found. On a high level, both proposals use the same mechanism to avoid crypto-tagging. Where the proposals differ is in the authentication part. Proposal 295 makes a functional separation between the encryption part and the authentication part, cf. Lines 150-152 (authentication) and Lines 156-160 (layered encryption). Conversely, Proposal 308 does not offer such separation, and the authentication and encryption of the last node are done in a single pass (cf. Lines 227-230 and Lines 244-252). This comes with what we think are two highly unwanted side effects defeating the purpose of using the ADL construction to begin with: the authentication of the last node depends solely on the proper execution of the IF statement on Line 266. As a result, if this line is skipped for some reason (e.g., because an adversary corrupted the last node, a bug, or as a result of over-optimization), modified messages may leave the network. Moreover, the last layer is malleable which means that a difference introduced to the ciphertext entering the last node will be preserved through the final decryption (given that the IF statement on Line 266 is skipped). This is because the decryption nonce does not depend on the authentication process (in the lingo of Proposal 308 this is called a “dynamic nonce”). Comparing this to Proposal 295 we see that the same cannot happen. Any change introduced at any point (including the ciphertext entering the last node) will completely destroy the payload in an irrecoverable way (the same happens in the “static layers” of Proposal 308; only the dynamic layer is malleable). For the record, a corollary of all of this is that if Sf_I is leaked (e.g., via a side channel in the generation process of Nf_I that is used by the IF statement), the adversary now has the secret it needs to decrypt the ciphertext regardless of the authentication process. Not being able to do this is exactly what’s captured by the RUP property used in Proposal 295 in which the only way to obtain N_4 (the counterpart of Proposal 308’s Nf_I) is via a successful digest of an unmodified ciphertext. The place where Proposal 308 nicely extends over Proposal 295 is in the forward secrecy domain. In an email to this mailing list we conjectured that if certain changes are made to Proposal 295 it will provide forward secrecy in addition to its crypto-tagging resistance. Jean Paul suggested an attack against this conjecture, but I find that the attack is not very convincing. Indeed, once the keys are leaked, the last message can be recovered. But I don’t think that there’s anything surprising in the fact that the set of keys that would have normally decrypted a message will also do so if leaked to an adversary. A more reasonable definition for forward secrecy would be that no message can be decrypted *after* the state (including ephemeral secrets) that was used to generate this message was replaced. Admittedly, Proposal 308 replaces this state earlier than Proposal 295 (immediately after processing the message vs. after processing the next message), which may be desirable, but is anyway not disastrous. That being said, this discussion is theoretic in nature since of the two proposals, only Proposal 308 offers an actual mechanism. For Proposal 295 we only offer a conjecture. We also tend to somewhat agree that frequent re-keying is a better way to achieve forward secrecy. Regardless of which is the better way, both can be built on top of the encryption mechanism we offered in Proposal 295 whose goal is to resist crypto-tagging. In the interest of moving forward we propose to implement Proposal 295 as suggested or something close to it (e.g., using POLYVAL) to counter crypto-tagging, then discuss alternatives to achieving forward secrecy and add those on top of the ADL construction via Proposal 308. A few side notes: 1. Proposal 308 argues that POLYVAL is more suited than GHASH to our this use-case. This is an implementation issue. GHASH or POLYVAL or any other collision resistant hash function are all the same to us. 2. I'm pretty sure there's a typo on Line 250 in Proposal 308 and that the text should be Y_I = Tf_{I+1} ^ X_I. Otherwise, I can't see how the protocol decrypts on Line 285. 3. The lengths in Section 2.2 (marked for revision) are given in bytes, but then in Section 2.3 they are treated as bits. 4. Line 230 has unbalanced parenthesis. Tomer

2 1

Optimistic SOCKS Data
by Tom Ritter 15 Oct '19

15 Oct '19

The attached is a draft proposal for allowing tor to lie to an application about the SOCKS connection enabling it to send data optimistically. It's going to need some fleshing out in ways I am not familiar with, but I wanted to get something out to start as we think that this is probably the best path forward for bringing back Tor Browser's optimistic SOCKS behavior. -tom

6 11