February 2019 - tor-dev - lists.torproject.org

the consequences of deprecating debian alpha repos with every new major branch
by nusenu 21 Dec '21

21 Dec '21

Hi, tldr: - more outdated relays (that is a claim I'm making and you could easily proof me wrong by recreating the 0.3.3.x alpha repos and ship 0.3.3.7 in them and see how things evolve after a week or so) - more work for the tpo website maintainer - less happy relay operators [3][4] - more work for repo maintainers? (since a new repo needs to be created) When the tor 0.3.4 alpha repos (deb.torproject.org) first appeared on 2018-05-23 I was about to submit a PR for the website to include it in the sources.list generator [1] on tpo but didn't do it because I wanted to wait for a previous PR to be merged first. The outstanding PR got merged eventually (2018-06-28) but I still did not submit a PR to update the repo generator for 0.3.4.x nonetheless and here is why. Recently I was wondering why are there so many relays running tor version 0.3.3.5-rc? (see OrNetStats or Relay Search) (> 3.2% CW fraction) Then I realized that this was the last version the tor-experimental-0.3.3.x-* repos were shipping before they got abandoned due to the new 0.3.4.x-* repos (I can no longer verify it since they got removed by now). Peter made it clear in the past that the current way to have per-major-version debian alpha repos (i.e. tor-experimental-0.3.4.x-jessie) will not change [2]: > If you can't be bothered to change your sources.list once or twice a > year, then you probably should be running stable. but maybe someone else would be willing to invoke a "ln" commands everytime a new new alpha repo is born. tor-alpha-jessie -> tor-experimental-0.3.4.x-jessie once 0.3.5.x repos are created the link would point to tor-alpha-jessie -> tor-experimental-0.3.5.x-jessie It is my opinion that this will help reduce the amount of relays running outdated versions of tor. It will certainly avoid having to update the tpo website, which isn't a big task and could probably be automated but it isn't done currently. "..but that would cause relay operators to jump from i.e. 0.3.3.x to 0.3.4.x alphas (and break setups)!" Yes, and I think that is better than relays stuck on an older version because the former repo no longer exists and operators still can choose the old repos which will not jump to newer major versions. [1] https://www.torproject.org/docs/debian.html.en#ubuntu [2] https://trac.torproject.org/projects/tor/ticket/14997#comment:3 [3] https://lists.torproject.org/pipermail/tor-relays/2018-June/015549.html [4] https://trac.torproject.org/projects/tor/ticket/26474 -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu

4 7

tor relay process health data for operators (controlport)
by nusenu 16 Nov '20

16 Nov '20

Hi, every now and then I'm in contact with relay operators about the "health" of their relays. Following these 1:1 discussions and the discussion on tor-relays@ I'd like to rise two issues with you (the developers) with the goal to help improve relay operations and end user experience in the long term: 1) DNS (exits only) 2) tor relay health data 1) DNS ------ Current situation: Arthur Edelstein provides public measurements to tor exit relay operators via his page at: https://arthuredelstein.net/exits/ This page is updated once daily. the process to use that data looks like this: - first they watch Arthur's measurement results - if their failure rate is non-zero they try to tweak/improve/change their setup - wait for another 24 hours (next measurement) This is a somewhat suboptimal and slow feedback loop and is probably also less accurate and less valuable data when compared to the data the tor process can provide. Suggestion for improvement: Exposes the following DNS status information via tor's controlport to help debug and detect DNS issues on exit relays: (total numbers since startup) - amount of DNS queries send to the resolver - amount of DNS queries send to the resolver due to a RESOLVE request - DNS queries send to resolver due to a reverse RESOLVE request - amount of queries that did not result in any answer from the resolver - breakdown of number of responses by response code (RCODE) https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-pa… - max amount of DNS queries send per curcuit If this causes a significant performance impact this feature should be disabled by default. 2) general relay health metrics -------------------------------- Compared to other server daemons (webserver, DNS server, ..) tor provides little data for operators to detect operational issues and anomalies. I'd suggest to provide the following stats via the control port: (most of them are already written to logfiles by default but not accessible via the controlport as far as I've seen) - total amount of memory used by the tor process - amount of currently open circuits - circuit handshake stats (TAP / NTor) DoS mitigation stats - amount of circuits killed with too many cells - amount of circuits rejected - marked addresses - amount of connections closed - amount of single hop clients refused - amount of closed/failed circuits broken down by their reason value https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n1402 https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1994 - amount of closed/failed OR connections broken down by their reason value https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n2205 If this causes a significant performance impact this feature should be disabled by default. cell stats - extra info cell stats as defined in: https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n1072 This data should be useful to answer the following questions: - High level questions: Is the tor relay healthy? - is it hitting any resource limits? - is the tor process under unusual load? - why is tor using more memory? - is it slower than usual at handling circuits? - can the DNS resolver handle the amount of DNS queries tor is sending it? This data could help prevent errors from occurring or provide additional data when trying to narrow down issues. When it comes to the question: **Is it "safe" to make this data accessible via the controlport?** I assume it is safe for all information that current versions of tor writes to logfiles or even publishes as part of its extra info descriptor. Should tor provide this or similar data I'm planing to write scripts for operators to make use of that data (for example a munin plugin that connects to tor's controlport). I'm happy to help write updates for control-spec should these features seem reasonable to you. Looking forward to hearing your feedback. nusenu -- https://twitter.com/nusenu_ https://mastodon.social/@nusenu

5 7

DNS-over-HTTPS (DOH) in Firefox/Torbrowser
by nusenu 02 Jul '20

02 Jul '20

Hi, since Mozilla did tests [0] on DOH [1] in Firefox I was wondering if Torbrowser developers have put any thought into that as well? Note: I'm _not_ suggesting to use DOH in torbrowser I'm just asking because the answer probably matters for exit documentation in the relay guide if clients do DNS themselves over TCP connections instead of relying on the exit (even if torbrowser is not the only tor client). thanks, nusenu [0] https://www.ghacks.net/2018/03/20/firefox-dns-over-https-and-a-worrying-shi… [1] https://datatracker.ietf.org/doc/draft-hoffman-dns-over-https/ -- https://mastodon.social/@nusenu twitter: @nusenu_

4 4

Lets give every circuit its own exit IP?
by nusenu 18 Mar '20

18 Mar '20

The unbearable situation with Google's reCAPTCHA motivated this email (but it is not limited to this specific case). This idea came up when seeing a similar functionality in unbound (which has it for a different reason). Assumption: There are systems that block some tor exit IP addresses (most likely the bigger once), but they are not blocked due to the fact that they are tor exits. It just occurred that the IP got flagged because of "automated / malicious" requests and IP reputation systems. What if every circuit had its "own" IP address at the exit relay to avoid causing collateral damage to all users of the exit if one was bad? (until the exit runs out of IPs and starts to recycle previously used IPs again) The goal is to avoid accumulating a bad "reputation" for the single used exit IP address that affects all tor users of that exit. Instead of doing it on the circuit level you could do it based on time. Change the exit IP every 5 minutes (but do _not_ change the exit IPs for _existing_ circuits even if they live longer than 5 minutes). Yes, no one has that many IPv4 addresses but with the increasing availability of IPv6 at exits and destinations, this could be feasible to a certain extend, depending on how many IPv6 addresses the exit operator has. There are exit operators that have entire /48 IPv6 blocks. problems: - will not solve anything since reputation will shift to netblocks as well (How big of a netblock are you willing to block?) - you can tell two tor users easily apart from each other even if they use the same exit (or more generally: you can tell circuits apart). There might be all kinds of bad implications that I'm not thinking off right now. - check.tpo would no longer be feasible - how can do we still provide the list of exit IPs for easy blocking? Exits could signal their used netblock via their descriptor. What if they don't? (that in turn opens new kinds of attacks where an exit claims to be /0 and the target effectively blocks everything) - more state to track and store at the exit -... some random thoughts, nusenu -- https://mastodon.social/@nusenu twitter: @nusenu_

4 6

Proposal: Don't include package fingerprints in consensus documents
by Iain Learmonth 13 Mar '19

13 Mar '19

Hi All, #28465 [0] needed a proposal. Feedback is welcome and encouraged. I've not written a proposal before, so if someone could let me know if I'm following the process OK (or not) then that is useful too. Thanks, Iain. [0] https://trac.torproject.org/projects/tor/ticket/28465

3 8

#3600 tech doc
by Richard Pospesel 13 Mar '19

13 Mar '19

Hey y'all, For the past little while I've been working on a technical overview doc for #3600 (Prevent redirects from transmitting+storing cookies+identifiers) detailing the problems, scenarios and possible solutions. Please take a look and feel free to comment, edit or add! Link: https://storm.torproject.org/grain/X4nhdNqR9fGRc7sgTefFkg/ best, -Richard

3 7

Anti-censorship discussion with Briar devs
by Cecylia Bocovich 28 Feb '19

28 Feb '19

Hi, I just had a really great conversation with some of the developers at Briar about the recent work they've done in integrating some pluggable transports into their messaging application. I thought I would summarize some key points from the conversation here. In particular, this information might prove useful for both the metrics and anti-censorship teams to know how other projects are using our tools and what they would like help with. This summary is organized as follows: I. How they used metrics data II. Independent reachability tests they performed III. What they did to integrate pluggable transports into Briar IV. What they want help with I. How they used metrics data One goal the Briar project had was to automatically detect where bridges are needed so they can be enabled automatically and for Briar to "work out of the box" with minimal developer overhead as they do not currently have dedicated anti-censorship funding. Here is the related ticket with a lot of useful discussion: https://code.briarproject.org/briar/briar/issues/1266 They wrote a collection of analytics scripts that use both OONI and Tor Project metrics data to determine where bridges are being blocked. Source code for the scripts: https://code.briarproject.org/briar/tor-circumvention-analytics/ Results (outdated): https://grobox.de/tor/bridges.html They rely mostly on OONI data but also use some Tor metrics to look at the ratio of bridge users in a country. Their reasoning being that while OONI shows what doesn't work, Tor metrics data shows what is currently working. Source code for Tor metrics script: https://code.briarproject.org/briar/tor-circumvention-analytics/blob/master… The ticket linked above has more discussion that might be useful to the metrics team as well. II. Independent reachability tests they performed To validate some of the results from OONI and Tor metrics, they ran a private bridge and did some reachability tests to it from China. They found that the bridge was not blocked, however they did not perform bandwidth tests to determine whether or not obfs4 bridges are being throttled. This makes sense as briar produces probably fairly low bandwidth traffic. When I asked them about it they suggested performing the following test: Set up obfs4 to forward connections to a local http server that servers a large, static file. Have a contact in China set up a cron job that periodically downloads the file from the bridge via obfs4. Their original reachability test was not running for very long, so it's still possible that prolonged use would result in its discovery but the fact that it wasn't blocked suggests that China might be enumerating bridges through the distribution mechanism rather than by something identifiable from obfs4 itself. We should perform our own tests to verify this, and to check for throttling. III. What they did to integrate pluggable transports into Briar My understanding from the issue text and commit messages about how briar decides whether to use pluggable transports is to use the output of their reachability analysis to determine user needs by country. If Tor is not blocked, users just use vanilla Tor If it is they use obfs4. If obfs4 is blocked, then they use meek (meek lite). Right now the countries that use meek in briar are: China, Iran, Egypt, Belarus, Turkey, Syria, and Venezuela. As far as the integration, they wrote some code that makes reproducible builds of obfs4 and meek and spits out a java/android library: https://code.briarproject.org/briar/go-reproducer Briar already uses Tor, so they configure these bridges in the usual way using a torrc file and a hardcoded, shipped file of bridge information (which I believe are also the default bridges used by Tor Browser). They decided they didn't want to maintain any private bridges like the one they used for their reachability tests. One of the concerns there was the ability to fingerprint Briar traffic by bridge connection and differentiate it from other Tor traffic. IV. What they want help with Because they do not have dedicated anti-censorship funding, they mentioned a few things that would help them maintain their pluggable transport use going forward and ease the integration of pluggable transports. The main thing they would like on the metrics side is up-to-date information about which PT works in which country and where PTs are needed at all in order to make quick and easy decisions based on location about which transports to use. They started to work with OONI to expand their tests but it turned out to be too much work for their time/funding: https://code.briarproject.org/briar/briar/issues/1414 It's on our roadmap to work with OONI and other censorship measurement tools (like Censored Planet) to expand our tests so we should get into contact with them again once we have gotten farther with this. On the PT development side, they expressed a desire to transfer maintenance of their reproducible builds of obfs4 and meek to someone else (that's the go-reproducer code linked above).

5 6

Speeding up your Tor Appveyor Builds
by teor 27 Feb '19

27 Feb '19

Hi, We have made some changes to speed up Tor's Appveyor builds: https://bugs.torproject.org/29601 They should be merged soon. If you have Appveyor set up on your personal tor git repository, you can change Appveyor's configuration so that it's a bit faster. (We can't make these changes in the .appveyor.yaml config.) 0. Log in to Appveyor: https://ci.appveyor.com/ I use GitHub to log in to Appveyor. 1. Go to Appveyor's project settings page. Mine is: https://ci.appveyor.com/project/teor2345/tor/settings 2. Set "Build timeout, minutes" to 30. The default timeout is 60 minutes. Tor's jobs usually take about 10 minutes, unless they hang. 3. Set the "Rolling builds" checkbox. The default is: "build all versions of a branch/pull request". Rolling builds means: "when a branch or pull request is updated, cancel the previous build, and start a build with the latest changes." That's it. Your builds will be a bit faster! T -- teor ----------------------------------------------------------------------

1 0

Sending multiple streams throuh a single Tor circuit
by Piyush Kumar Sharma 26 Feb '19

26 Feb '19

Hello all, I am a PhD student, and am working on some measurements in Tor. I am stuck at a point where i need to send multiple applications(streams) traffic through a single circuit. I am currently using torsocks/torify to send traffic of these multiple applications through Tor. The main problem is that, despite trying many different ways to achieve the same (sending multiple streams through a single circuit), i am not successful. Things i have tried : 1.) Force Tor process to create only a single circuit at a time preventing any new circuit creation, so that any new stream would be attached to this only available circuit. To acheive this i have set the following Tor options : set __DisablePredictedCircuits to 1 set MaxClientCircuitsPending to 1 set newcircuitperiod to 999999999 set maxcircuitdirtiness to 999999999 The problem with the above method is that it seems to work sometimes randomly. But most of the times for some reason, a new circuit is still created. 2.) Next, i assumed that maybe running torify multiple times for each application is the culprit, as it may try to create new circuit for each run. So i created a new bidirectional stream using socat, which listens on a local TCP port, and forwards the data to the Tor SOCKS port assuming that it will lead to a singe connection to local SOCKS. Even this did not work and still new circuits were created randomly. 3.) Next i tried to attach streams to circuits manually, using the stem library following the link : https://stem.torproject.org/tutorials/to_russia_with_love.html#custom-path-… . This seemed to work initially, but then after every 4-5 runs, the streams seemed to detach automatically. Moreover, the original circuit crashed too. It would be great, if someone could tell a simple way to achieve the same, or would point me to any mistakes that can be improved in the above methodologies to make them work. Regards Piyush

4 8

xp + T
by Gedropi 26 Feb '19

26 Feb '19

I would like to continue to get Tor updates but intend to keep Windows XP. How can I do it? Thanks

4 4