- tor-dev - lists.torproject.org

Walking onions: week 10 update
by Nick Mathewson 11 May '20

11 May '20

Walking Onions: week 10 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html Week 5: Voting (part 2) https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html Week 6: Editing, voting rules, and client behavior (part 1) https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html Week 7: Exit policies: How do they work? https://lists.torproject.org/pipermail/tor-dev/2020-April/014232.html Week 8: Onion services, relay honesty, migration and families https://lists.torproject.org/pipermail/tor-dev/2020-April/014255.html Week 9: (There was no week 9) == Now with an introduction I left the Introduction section of the proposal till late in the game; now I have it in place as a new section 1: == Proposals into the pipeline I've also revised all of the "other proposals" that were logically separate from the rest of Walking Onions. Three of them were ready to go to tor-dev, so I've added them as: 318-limit-protovers.md Limit protover values to 0-63. 319-wide-everything.md RELAY_FRAGMENT cells 320-tap-out-again.md Removing TAP usage from v2 onion services == In the home stretch I've got till the end of the month to wrap up this proposal, so let's see where we are. We have a minimal viable proposal, I believe: if we were to sit down and start implementing, we would probably get most of the way based on what we have now. There are a few additional things I'd like to wrap up before I call the proposal done, however: * I've got all the necessary machinery written up for generating and calculating indices, but before I actually move ahead, I should fill in the details in sections 2 through 4. * One of the original goals of Walking Onions was to permit us to move to alternative topologies where not every relay connects to every other relay. That's not something that we can include in this proposal, since the research here is quite preliminary, but I still want to prepare for it by making sure that clients can handle getting SNIPs that say "this SNIP is only valid when given by a relay of type X". * I should take another pass through the CDDL format to see whether there are any opportunities to save a bunch of bytes. I should also fill in a couple of example SNIPs to see how big they are with this proposal. * There are places all over the proposal where I have marked uncertain points with XXXX. I should go through them and resolve them all before I call the proposal done. [INTRO] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/01-intro…

1 0

Proposal 315: Updating the list of fields required in directory documents
by Nick Mathewson 11 May '20

11 May '20

Filename: 315-update-dir-required-fields.txt Title: Updating the list of fields required in directory documents Author: Nick Mathewson Created: 23 April 2020 Status: Open 1. Introduction When we add a new field to a directory document, we must at first describe it as "optional", since older Tor implementations will not generate it. When those implementations are obsolete and unsupported, however, we can safely describe those fields as "required", since they are always included in practice. Making fields required is not just a matter of bookkeeping: it helps prevent bugs in two ways. First, it simplifies our code. Second, it makes our code's requirements match our assumptions about the network. Here I'll describe a general policy for making fields required when LTS versions become unsupported, and include a list of fields that should become required today. This document does not require to us to make all optional fields required -- only those which we intend that all Tor instances should always generate and expect. When we speak of making a field "required", we are talking about describing it as "required" in dir-spec.txt, so that any document missing that field is no longer considered well-formed. 2. When fields should become required We have three relevant kinds of directory documents: those generated by relays, those generated by authorities, and those generated by onion services. Relays generate extrainfo documents and routerdesc documents. For these, we can safely make a field required when it is always generated by all relay versions that the authorities allow to join the network. To avoid partitioning, authorities should start requiring the field before any relays or clients do. (If a relay field indicates the presence of a now-required feature, then instead of making the field mandatory, we may change the semantics so that the field is assumed to be present. Later we can remove the option.) Authorities generate authority certificates, votes, consensus documents, and microdescriptors. For these, we can safely make a field required once all authorities are generating it, and we are confident that we do not plan to downgrade those authorities. Onion services generate service descriptors. Because of the risk of partitioning attacks, we should not make features in service descriptors required without a phased process, described in the following section. 2.1. Phased addition of onion service descriptor changes Phase one: we add client and service support for the new field, but have this support disabled by default. By default, services should not generate the new field, and clients should not parse it when it is present. This behavior is controlled by a pair of network parameters. (If the feature is at all complex, the network parameters should describe a _minimum version_ that should enable the feature, so that we can later enable it only in the versions where the feature is not buggy.) During this phase, we can manually override the defaults on particular clients and services to test the new field. Phase two: authorities use the network parameters to enable the client support and the service support. They should only do this once enough clients and services have upgraded to a version that supports the feature. Phase three: once all versions that support the feature are obsolete and unsupported, the feature may be marked as required in the specifications, and the network parameters ignored. Phase four: once all versions that used the network parameters are obsolete and unsupported, authorities may stop including those parameters in their votes. 3. Directory fields that should become required. These fields in router descriptors should become required: * identity-ed25519 * master-key-ed25519 * onion-key-crosscert * ntor-onion-key * ntor-onion-key-crosscert * router-sig-ed25519 * proto These fields in router descriptors should become "assumed present": * hidden-service-dir These fields in extra-info documents should become required: * identity-ed25519 * router-sig-ed25519 The following fields in microdescriptors should become required: * ntor-onion-key The following fields in votes and consensus documents should become required: * pr

3 2

Is anybody using OnionPerf's measurement results in the .tpf format?
by Karsten Loesing 10 May '20

10 May '20

Hello list, if the subject line doesn't make any sense to you, this email is very likely irrelevant for you. But if it does make sense to you and you did use OnionPerf's [0] measurement in the .tpf format [1] in the past and/or plan to use this data in the future, you should comment on ticket #34141 [2] and tell us about your use case, ideally in the next couple of days. For what it's worth, the same measurements data is also available in OnionPerf's JSON format [3], and existing data from past measurements will continue to be available in the .tpf format. It's just that we want to get rid of the .tpf format for new measurements. All the best, Karsten [0] https://metrics.torproject.org/torperf.html [1] https://metrics.torproject.org/collector.html#type-torperf [2] https://trac.torproject.org/projects/tor/ticket/34141 [3] https://metrics.torproject.org/collector.html#type-onionperf

1 0

Example of how a stream of RELAY_DATA cells would work?
by Eli Vakrat 09 May '20

09 May '20

Hi again everyone! It's Eli ... So thanks to Teor and Nick's help, my python client (tor OP) is finally able to successfully establish a three-hop circuit with any TOR relays in the whole public network! Now It's on to the Data Cells... As of writing this, I can send and receive the proper RELAY_BEGIN and RELAY_CONNECTED to and from my exit node, but I'm not quite sure what to do next... Do I just start sending RELAY_DATA cells (where the "data" of the cell is literally the encoded HTTP requests)? I've tried connecting to 'www.facebook.com:443' with the RELAY_BEGIN cells as a test (I do get a Relay Connected Cell so at least I know that part works). After getting back the RELAY_CONNECTED cell, I send a RELAY_DATA cell with the data of the cell being the following 'utf-8' encoded string: * #######this is how i wrote the literal in python#### * *'GET / HTTP/1.1\r\nHost: www.facebook.com <http://www.facebook.com>\r\nUser-Agent: python-requests/2.23.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\n\r\n\r\n'.encode() * What I get back is a short couple of bytes: *\x15\x03\x03\x00\x02\x022* I had no idea what this meant but after digging around a bit I found that this seems to be some part of the TLS handshake that is used in HTTPS. So now two questions arise: 1. Is this a good TLS response? What does it mean exactly? 2. Generally speaking, is this how the RELAY_DATA cells are supposed to be sent and received? Just to clarify it would be great if I could get an exact example of how the stream of data should look. For example, if someone could maybe break down the steps of how a basic HTTP GET request would work through a TOR circuit (starting from sending a RELAY_BEGIN cell) that would help me tremendously. Thanks again to everyone who've helped me so far, and thanks in advance to anyone with an answer to any of my questions! Regards, Eli

2 1

Tor Network Expansion Question/Suggestion
by Beecher Bruno 09 May '20

09 May '20

Hello All, I don’t know much in the ways of how the actual coding works, but I’m very familiar with computers. I’m a student in Network Systems Engineering Technology in the USA. After being introduced to Tor several years ago, I loved it. I recently had a potentially good idea in making the Tor network more resilient against tracking/surveillance attempts. My basic understanding is that information from one computer gets transmitted to a node, bounced around between different nodes, and then spit out somewhere else. Or, at least, that’s what it would look like from the outside, making it extremely difficult to pin down the actual origin computer. (Please do forgive me if this is an inaccurate representation). But what about expanding the network significantly, without truly widening the load on all the nodes? My idea was essentially this: an app for mobile devices and/or computers that would ask for consent from the user. Upon consent, that device could essentially fake its identity, so that everyone outside the network thought it was a node? Data that’s actually sent between nodes could just put in a random “fake” node at the beginning and end of a packet so, as far as anyone else knows, that data went through an extra two nodes? Just in case that’s not clear, imagine a Tor node receives packet A in Washington DC. On its complicated, 3 node journey to the end node in Hongkong, random devices that have been consented to be “fake nodes” will have added themselves in. Theoretically, by the time it arrives in Hongkong, this 5 node journey (DC Hongkong and the 3 in between) will look on the outside to be a 6 or 7 node journey. I’m not sure if this is possible, but if it is, I think it could help protect people’s privacy even more. I do apologize for emailing this list, I couldn’t exactly find an “Contact Us” page, probably for good reason. Have a great day! Stay safe out there. Beecher

1 0

My Introduction_GSoC Student
by Hashik Donthineni 07 May '20

07 May '20

Hello everyone! My name is Hashik Donthineni (IRC: HashikD), I am a GSoC 2020 student. I will be working on* Snowflake Proxy on the Android *project with the help of my mentors Cecylia (Cohosh), Phillipp (phw). I am really excited to join the Tor team! Thank you all for giving me this exciting opportunity! Regards, D. Hashik

1 0

Does a design document for the DoS subsystem exist?
by Lennart Oldenburg 07 May '20

07 May '20

Hi all, We are investigating how Tor protects itself against Denial-of-Service (DoS) attacks. So far, it has been difficult to find a comprehensive top-level design document for the DoS subsystem (e.g., a torspec or proposal) that reflects the decisions that lead to the subsystem in its current form. Specifically, we are looking at the DoS mitigation subsystem code for entry guards at src/core/or/dos.{h,c} [1]. We are trying to understand the chosen countermeasures and how the default and current consensus values came to be, e.g., the decision to limit to 3 circuits per second after the initial burst. 1) Could you kindly point us in the right direction if any such document exists? 2) If it does not exist, would you mind briefly explaining how the DoS threshold values (such as DoSCircuitCreationMinConnections, DoSCircuitCreationRate, DoSCircuitCreationBurst, and DoSConnectionMaxConcurrentCount) were chosen? Thank you very much in advance. Kind regards Lennart Oldenburg KU Leuven [1] https://gitweb.torproject.org/tor.git/tree/src/core/or/dos.c

3 4

(No walking onions update for last week)
by Nick Mathewson 04 May '20

04 May '20

Hi all! Sending a short note that I won't be doing a walking onions update from the past week -- my time went to follow-on tasks for personnel transitions, and I didn't make much progress to speak of on the walking onions specs. I hope to get more done this week, and send an update then. -- Nick

1 0

Fwd: Orbot 16.2.0 RC-1 (tor-0.4.2.7)
by Nathan Freitas 04 May '20

04 May '20

Now with integrated MOTE bridge request support, just like on Onion Browser iOS, courtesy of Benjamin Erhart (aka tla). -------- Forwarded Message -------- Subject: Orbot 16.2.0 RC-1 (tor-0.4.2.7) Date: Mon, 4 May 2020 12:03:13 -0400 From: Nathan of Guardian <nathan(a)guardianproject.info> Organization: Guardian Project To: Guardian Dev <guardian-dev(a)lists.mayfirst.org> Orbot 16.2.0 RC-1 (tor-0.4.2.7) Tag: https://gitweb.torproject.org/orbot.git/tag/?h=16.2.0-RC-1-tor-0.4.2.7 APKs: - https://github.com/guardianproject/orbot/releases/tag/16.2.0-RC-1-tor-0.4.2… - Universal: https://guardianproject.info/releases/Orbot-16.2.0-RC-1-tor-0.4.2.7-1-geed7… (.asc) - ARM64: https://guardianproject.info/releases/Orbot-16.2.0-RC-1-tor-0.4.2.7-1-geed7… (.asc) - F-Droid soon... Thanks to the fantastic contributions by @bitmold @tladesignz and @Hashik-Donthineni Highlights Resolved many issues related to startup of Tor and Obfs4/Meek bridges on Android Q/10 VPN fixes resolved through unifying 2 background services (TorVPNService, TorService) into one (OrbotService) Update tor to 0.4.2.7 and obfs4proxy to 0.0.12 latest Brand new "beta" integrated bridge fetch feature through Tor's MOTE service (just like in Tor Browser) Brand new custom bridge configuration, setup UI Zulu language random switch bug FINALLY fixed.... now the language you choose remains! (sorry, Zulu!) and much more... /** 16.2.0-RC-1-tor-0.4.2.7 / 4 May 2020 / 0a3a4f7 **/ 0a3a4f7 (HEAD -> master) update version code to 16202001 a631077 (HEAD -> master, origin/master, origin/HEAD) Merge branch 'tladesignz-master' 875f18a (tladesignz-master) Merge branch 'master' of https://github.com/tladesignz/orbot into tladesignz-master 85e027c update version code cabf328 don't start tun2socks/VPN if it is already running 718bb42 Issue #324: Implemented CustomBridgesActivity analogous to Onion Browser iOS. 12a3f4a (tag: 16.2.0-BETA-4-tor-0.4.2.7) remove TorVpnService from manifest to address #327 crash 22d7ef8 (tag: 16.2.0-BETA-3-tor-0.4.2.7) update to 16201003 22bd248 update binary install scripts, to fix pdnsd install - now pdnsd is packaged as libpdsnd.so to work on newer platforms e93c8bc (tag: 16.2.0-BETA-2-tor-0.4.2.7) update version code to 16201002 d0fec8e update tor and obfs4proxy binary libs to address #326 hopefully 3ac5a70 Issue #309: Activate newly required bridges, fixed accidental reset through Tor status notification. e1425d3 Issue #309: Also disable CAPTCHA solution EditText, as long as there's no CAPTCHA to solve and while letting it check. ff7cbcf Issue #309: Error alert button should say "OK" instead of "Cancel", since there's nothing to cancel, really, just to acknowledge the error. 54f5f31 Issue #309: Hide refresh option menu item instead of disable, because Android doesn't visualize that adequately. Additionally: Fix issue where you couldn't refresh after a network error. 3bfaa8a Issue #309: Fixed potential crash, when activity went away. c41d453 (public/master) fix unneeded includes 4f4300a (tag: 16.2.0-BETA-1A-tor-0.4.2.7, tag: 16.2.0-BETA-1-tor-0.4.2.7, ghdev/master) update version to 16201001 7d7b2e9 remove app updater code to address concerns in #323 ec47151 major VPN refactor with Android Q updates for #263 #261 #151 #316 and #303 - integrate TorVPNService directly into OrbotService, so just one service now - removed VPNEnableActivity, and just have VPN activated by service now - changes address start on boot with VPN enable - also improvements to managing PDNSD daemon d64b4d0 (tag: 16.1.5-BETA-2-tor-0.4.2.7) update version code to 16150001 d5bc374 update to SDK 29 919c211 improve pdnsd pid checking e78c27b remove unneeded commands in tor config and startup 5cd6cc8 update tor to 0.4.2.7 785fa8f Merge branch 'master' of github.com:guardianproject/orbot 93a0786 Merge pull request #322 from bitmold/AboutDialogCrashFix b24750e Fixes #321 IlleglaStateException on about dialog 2e6c9b1 (tag: 16.1.5-BETA-1-tor-0.4.2.5-rc) small tweaks to request bridge strings 3c80df0 Merge branch 'bitmold-compile_erorr+refactor' 08de553 (bitmold-compile_erorr+refactor) Merge branch 'compile_erorr+refactor' of https://github.com/bitmold/orbot into bitmold-compile_erorr+refactor df847ff Merge branch 'tladesignz-master' b2646a3 update AndroidPT to latest 1.0.8 with fixes for obfs4 and meek on Android 10 feedb3b fix string variable values 155980f Change from using String resources for a bundle key in Onboarding ec0ad26 Fixes compile error on master branch right now f2c8473 Merge branch 'master' of https://github.com/tladesignz/orbot into tladesignz-master 732c153 Merge branch 'master' of https://github.com/guardianproject/orbot 09ded59 (githubgp/master) Merge pull request #311 from bitmold/email_bridges_fix b1049ca Merge branch 'master' of https://github.com/guardianproject/orbot b9804c1 Merge pull request #305 from Hashik-Donthineni/AppIntroOrientationFix 410ae03 tweaks to app data storage and control port interaction - in some cases, Orbot connection to local control port can hang. Make changes to address that. 19d5054 Color indeterminate ProgressBar. 1749167 Fixed behaviour on device turn. f94931e Don't allow refresh button to trigger while request ongoing. 7a9b9d1 Adapted to naming convention of project. 964c311 Fixed MoatActivity startup when Tor is not running. Reset old bridge configuration, when not completed. 5c99859 Finish BridgeWizardActivity on successful MOAT. a34ee22 Improved UI: show progress spinner, start request on keyboard enter. a46e463 Updated Gradle to latest version. c026e20 Check OrbotService status, start Tor, if down, switch to Meek, when started, fetch CAPTCHA, as soon as done with that. Proxy Volley correctly. Improved option menu handling. Not working, yet: Tor doesn't come up. Why? e7cfe46 Added first stab at a MoatActivity. Doesn't change bridges automatically, yet. Also, Volley needs to be proxied. 469d4cf Fixed most warnings and errors in OrbotMainActivity. 3eb76e6 Updated Gradle. 4283221 Updated BUILD info. 446dfa6 Fixes bugs in #289 pertaining to parsing bridges 8627989 Merge pull request #310 from bitmold/zulu_context_menu 29541f3 Fixes #300 Where Locale is set to Zulu when settings are opened bfbb85b Changed variable names for better understanding a09b4eb Fixed Unused Imports f4580cf Added comments for better understanding 6c09f67 Extracted String Resources ee3e054 Fixed blank screen when orintation changed in AppIntro 8a3727a Merge pull request #301 from PLNech/patch-1 0dcba0d fix FAQ URL in README 73b0b0d handle NPE since node is not yet set Assets 6 Orbot-16.2.0-RC-1-tor-0.4.2.7-1-geed732a3-fullperm-arm64-v8a-release.apk 11.2 MB Orbot-16.2.0-RC-1-tor-0.4.2.7-1-geed732a3-fullperm-arm64-v8a-release.apk.asc 833 Bytes Orbot-16.2.0-RC-1-tor-0.4.2.7-1-geed732a3-fullperm-universal-release.apk 32.2 MB Orbot-16.2.0-RC-1-tor-0.4.2.7-1-geed732a3-fullperm-universal-release.apk.asc 833 Bytes Source code (zip) Source code (tar.gz)

1 0

Proposal Idea: IPv6-Only Exit Relays
by teor 01 May '20

01 May '20

Hi all, The network team is making it easier to set up a dual-stack Tor relay. We're currently working on: * IPv6 reachability self-tests * IPv6 address auto-detection * IPv6 relay statistics We're also thinking about the next IPv6 project. One possibility is adding IPv6-only exits to the Tor network. We'd need to solve two technical problems: 1. stop requiring IPv4 in the consensus and circuit code 2. avoiding delays when using an IPv6-only exit for an IPv4-only site And one deployment problem: 3. Get more dual-stack middle relays (50% - 75%) Here's a quick sketch of a solution to the IPv4-only site issue: Clients open two exit streams for every site. One of those streams must use an exit with IPv4 support. The client uses whichever stream succeeds first. Just like the "happy eyeballs" protocol, there is a delay between launching the two streams: https://www.rfc-editor.org/rfc/rfc8305.txt Here are some benefits of a universal change: * code and protocol consistency * makes traffic analysis harder * also helps with other exit failures, for a better user experience Here are some drawbacks: * extra load We can change the delay to manage the tradeoff between load, user experience, and traffic analysis resistance. Making the delay longer reduces the load, but makes the user experience worse. Traffic analysis resistance is a bit more complex. Consistency across different clients is important, but we also don't want to create obvious traffic patterns. So there are also some benefits to randomisation. If I get time, I'd like to turn these ideas into a proposal. T -- teor ----------------------------------------------------------------------

1 0

Tor Relay wont connect to private IP address
by Eli Vakrat 29 Apr '20

29 Apr '20

Hey guys! So thanks to teor's insightful response yesterday I decided to try to run a second tor relay (my middle node) on my private network. Unfortunately, I can't do it with Chutney because my python client is running on a windows machine. But I do have 3 machines at my disposal: 1. A windows machine (the python client) 2. A mac (the guard node) 3. Another mac (the middle node) However, after connecting all three machines to my private LAN (meaning they now all have local IP addresses), the EXTEND from the guard node to the middle node fails. When my guard node tries to connect to my middle node after receiving from the client a RELAY_EXTEND cell, the guard node logs the following error: Apr 28 17:00:31.000 [info] circuit_extend: Client asked me to extend to a private address Apr 28 17:00:31.000 [info] circuit_receive_relay_cell: connection_edge_process_relay_cell (away from origin) failed. Apr 28 17:00:31.000 [info] command_process_relay_cell: circuit_receive_relay_cell (forward) failed. Closing. So regarding this, I have two questions: 1. Is there a way for me to change something in my torrc file to override this error and allow my relay to extend to private IP addresses? My torrc is currently configured as such (Notice I put some place holders for the drectories and for the ip address tha aren't actually whats written there): ContactInfo e <draftkingschaching(a)gmail.com>mail(a)example.com ControlPort 9051 DataDirectory </path/to/data/dir> ExitPolicy reject *:* ExitRelay 0 GeoIPFile </path/to/geo/ip/file> GeoIPv6File </path/to/geo/ipv6/file> Log notice file <path/to/log/dirs/>/notice.log Log debug file <path/to/log/dirs/>/debug.log Log warn file <path/to/log/dirs/>/warn.log Nickname vtoria ORPort 443 NoAdvertise ORPort Relay.Public.IP.Example:443 <http://79.183.54.194:443/> NoListen SafeLogging 0 ExtendAllowPrivateAddresses 1 2. Would there maybe be a better way to run this private tor network (without chutney)? Thanks in advance for any answers! Eli

2 1

Relay Truncated Cell Example
by Eli Vakrat 28 Apr '20

28 Apr '20

Hello again everyone! For anyone who doesn't know\remember, my name is Eli, I'm a high school student from Israel and I'm currently trying to implement a TOR Client in Python. Currently, my project is configured so that my python client (OP) has its guard node set as my local machine (which is running a downloaded version of TOR). I do this for debugging purposes so that if I send a malformed cell as the implemented client, I can read the debug log that the OR generates and see what I did wrong. Last time I sent a question to the dev-list I was stuck trying to get my CREATE cells to work, and Nick Mathewson immediately answered my question and helped me a lot! Thanks Nick! I've made pretty good progress since then, and as of writing this, my implemented OP can successfully send a CREATE cell and even an EXTEND cell to my guard node. However, I am running into a problem while trying to extend my circuit to the third node (my exit node in this case). After sending a RELAY_EXTEND cell to my middle node (which is encrypted twice with my middle node and then guard node forward key using AES128 in CTR mode), the Response I get back is a relay cell with the correct Circuit Id and command (the command I get is the RELAY command which is represented as 3), BUT the payload of the response cell is very weird. I am unable to 'recognize' it (as specified in section 6.1 of the tor-spec). and furthermore, it does not seem to be any type of cell, it just seems like a bunch of nonsense. Seeing this I initially thought that the response cell I kept getting was a RELAY_EXTENDED cell that I couldn't 'recognize' due to an error while decrypting the cell payload. But then I looked at the debug log of my guard node (remember that my guard node is on my local machine) and it said that it had received a DESTROY Cell back from the middle node and was passing on a RELAY_TRUNCATED cell to me: *Apr 26 16:11:03.014 [debug] command_process_destroy_cell: Received for circID 2297363203.* <------this is the circuit ID between my gaurd node and the middle node *Apr 26 16:11:03.014 [debug] command_process_destroy_cell: Delivering 'truncated' back.Apr 26 16:11:03.014 [debug] relay_send_command_from_edge_: delivering 9 cell backward.* To my understanding what this log means is that some part of the EXTEND cell I sent to the middle node was wrong or malformed and because of this when the middle node tried to extend the circuit, an error occurred, and the circuit needed to be torn down. This is very weird because when I send an EXTEND cell that is meant for my guard node (meaning I want to extend the circuit from one hop to two hops) everything works fine, and I can even successfully derive the shared key material for the middle node. So I have several questions regarding this: 1. First of all, I didn't quite understand the exact format of a RELAY_TRUNCATED cell. Does it contain a relay cell command + recognized+field +digest and so on? or is it just a single octet that immediately follows the cell command field? If some could show me an example of the cell, it would be much appreciated... 2. What are some common errors that would make an OR drop a RELAY EXTEND cell? I thought maybe it was a problem with my TAP handshake data, but after extensive checking that doesn't seem to be the case. 3. If someone could describe the exact steps of extending a circuit to a third node, it would greatly help me to make sure that I didn't miss a step or do something wrong. Thanks in advance for any answers, examples, or comments! I am having a lot of fun doing this project so far and I hope to hear back from anyone who has an answer :) Regards, Eli

2 1

Walking onions status update: week 8
by Nick Mathewson 27 Apr '20

27 Apr '20

Walking Onions: week 8 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html Week 5: Voting (part 2) https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html Week 6: Editing, voting rules, and client behavior (part 1) https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html Week 7: Exit policies: How do they work? https://lists.torproject.org/pipermail/tor-dev/2020-April/014232.html == Onion services: surprisingly easy! (but not?) This week I started with a look at onion services and how to make them work with Walking Onions [ONIONSPEC]. Most of my initial thought had gone into the problem of finding and using HSDirs, and so specifying this part went smoothly. I already havd a link specifier for asking for a span of several entries in an ENDIVE, and I already had a notion of truncating index values based on a network parameter. The harder part came when I wanted to talk about how to specify introduction points and rendezvous points by their SNIPs. The hard part here was managing the migration path. To avoid fingerprinting, when we make changes to the onion service protocol of this kind, we need to give a lot of onion services and clients the opportunity to start supporting the new protocol all at once, and then to stop supporting the old one all at once. This requires more network parameters, somewhat carefully specified. I'm hoping we can turn this into a more general pattern. And the most irritating part is when I ran into the issue of supporting HSv2 handshakes. Were it not for those, we wouldn't have to keep TAP keys around any more -- and TAP keys would bloat SNIPs to a pretty huge degree. The best answer here seems to be migrating our legacy HSv2 services so they can use ntor onion keys for intro and rendezvous points. That would also need to be done in a phased way. [TAP-OUT-AGAIN]. The alternative way to keep HSv2 working with walking onions would be to allow TAP keys to be distributed as part of the SNIP system. That seems like a fair amount of work, though, and unlike using ntor, it wouldn't have any positive benefit. ((Yes, HSv2 is deprecated. But I don't want to give people who still use it a reason to avoid the walking onions transition.)) == Relay honesty and ENDIVE swapping One of the issues with our current directory system is that multiple consensus network documents are valid at the same time. In practice, clients will accept fairly old documents, to avoid running into trouble if the authorities have failed to vote. But this client behavior gives hostile directory guards an opportunity to select which of several recent documents to serve. Walking Onions would make this problem worse, since every relay, when it serves a SNIP, is serving part of a directory that it knows a client is about to use. If we did nothing to mitigate the problem, then hostile relays could mount an "ENDIVE swapping" attack where they look at a client's choice of routing index, and then see whether any live ENDIVE contains a SNIP that the attacker likes at that position. In [HONESTY] I described a few responses to this problem. The first is that we should limit the rate at which the index layouts change. The second is that all parties that look up SNIPs through a relay should verify that the "published" times from each relay are monotonically increasing. The third response seems like the most useful to me: we would recognize that under ordinary circumstances, the permissible lifetime for an ENDIVE would be much shorter than it would be under emergency (consensus failed) circumstances. That is, if everything is going well, each relay should get a new ENDIVE every hour or so. Only if there is no new ENDIVE should clients accept SNIPs that are extremely old. With this in mind, I'm proposing a rule that if a client sees _any_ valid SNIP published at time `T` or later, it should not accept any SNIP published before time `T-Delta`. (Delta is a network parameter.) With this rule in place, clients will insist on fresh SNIPs when the network is working well, but will tolerate old ones if the authorities are failing to reach consensus for a while. == More questions of migration One hard step in any proposal is dealing with migration questions. I've stared to try to lay out a timeline in a new section [MIGRATION]. This part isn't looking great to me right now. It seems like it won't be safe to turn on Walking Onions for clients until it's supported by most relays. Moreover, clients that use Walking Onions won't be able to use relays (or at least, non-exit relays) without support for it. This implies a longer migration path than I'd like, but I don't see a way around it. == And what about families? Our existing family representation is quadratic in family size, since every relay needs to list every other relay in the family. This gets worse with Walking Onions, since SNIPs can't benefit from compressing the repeated family entries. At first I had hoped to use some kind of a max-clique algorithm to infer the families at the authorities, and then encode clique identifiers in the SNIPs. This isn't such a great idea, though: I wasn't able to persuade myself that an adversary couldn't force the authorities' burden to become exponential here. Instead I think we should resurrect proposal 242, in an improved version [BETTER-FAMILIES]. Instead of having to list every node in a family, each relay in a family would include a signed certificate from a "family key" attesting to their membership in the family. The proposal addresses some tricky compatibility issues. == Coming up next This week is going to be rather busy on my end: I'll be a bit startled if I get very much done with Walking Onions at all. My hope is to edit the proposal, fill in gaps, and address pending comments. [BETTER-FAMILIES] https://github.com/nmathewson/walking-onions-wip/blob/master/other-proposal… [HONESTY] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/08-track… [MIGRATION] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/09-migra… [ONIONSPEC] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/07-onion… [TAP-OUT-AGAIN] https://github.com/nmathewson/walking-onions-wip/blob/master/other-proposal…

1 0

Building a privacy-preserving "contact tracing" app
by Adrien Luxey 24 Apr '20

24 Apr '20

Hi all, The French state is making a glosing about the "privacy-preserving", "anonymous" contact tracing app they are developing with Inria (national informatics research agency). You can check about the protocol proposal, ROBERT, here: https://github.com/ROBERT-proximity-tracing/documents (in English!) As you would expect, the proposal is not privacy-preserving unless you believe the State would never ever misbehave, e.g. link IP address to identity with the help of ISPs etc. There is some relevant criticism here: https://github.com/ROBERT-proximity-tracing/documents/issues/6 I'd like to propose a really private "contact tracing" counter-proposal, which would use Tor's onion services for sender-receiver anonymity. Not that I am a proponent of the idea, but we need to come up with alternatives in the debate. My question is: /would the Tor community agree on having StopCovid's traffic go through its net?/ In my proposal, I would only broadcast a message from a declared COVID-positive person to all its recent contacts; the proximity sensing would remain Bluetooth. Still, deployed at the scale of a country, it's a lot of traffic. Thank you for your time, I will keep you posted, Adrien

4 3

Proposal 314: Allow Markdown for proposal format.
by Nick Mathewson 23 Apr '20

23 Apr '20

``` Filename: 314-allow-markdown-proposals.md Title: Allow Markdown for proposal format. Author: Nick Mathewson Created: 23 April 2020 Status: Open ``` # Introduction This document proposes a change in our proposal format: to allow Markdown. ## Motivation Many people, particularly researchers, have found it difficult to write text in the format that we prefer. Moreover, we have often wanted to add more formatting in proposals, and found it nontrivial to do so. Markdown is an emerging "standard" (albeit not actually a standardized one), and we're using it in several other places. It seems like a natural fit for our purposes here. # Details We should pick a particular Markdown dialect. "CommonMark" seems like a good choice, since it's the basis of what github and gitlab use. We should also pick a particular tool to use for validating Markdown proposals. We should continue to allow text proposals. We should continue to require headers for our proposals, and do so using the format at the head of this document: wrapping the headers inside triple backticks.

1 0

Re: [tor-dev] Chutney code refactoring
by c 22 Apr '20

22 Apr '20

On Fri, 17 Apr 2020 18:01:42 -0400 Nick Mathewson <nickm(a)freehaven.net> wrote: > If you want to work on this it would be helpful to maybe start by > listing (here or elsewhere) some places where you *don't* feel like > you could (or would want to) write documentation: those would be a > good target for devs who _have_ worked on Chutney before. Sounds good, I will work on this. > Though I think this is likely to be an ongoing change that we see > over time, since Was this sentence supposed to be longer? > * More tests for the various parts of chutney, and refactoring to > make them more testable. Internal unit testing seems like a good idea, yes. > * Making chutney comply with one or more of the python style > checking tools that are out there. I can look into this; personally I have no experience with Python style-checking or linting tools, but now's a good time as ever for me to learn what is available and how it would work for Chutney. > This is an good vision for how Chutney should go; I really support > this idea. Glad you are on board :) Caitlin

3 2

Fallback Directory Handover
by teor 21 Apr '20

21 Apr '20

Hi all, Here's a summary of the current state of fallback directory mirrors. Overall Design This repository contains a list of potential fallback directory mirrors (a fallback "offer list"), and a script that checks each mirror for speed and reliability: https://gitweb.torproject.org/fallback-scripts.git/ There is a "Fallback Scripts" component for tickets: https://trac.torproject.org/projects/tor/query?status=!closed&component=Cor… The fallback system is designed to gracefully degrade as fallback directory mirrors fail. Failures shift load to directory authorities, and cause brief delays during client bootstrap. We expect the system to operate well, even if all the fallbacks have failed. But we try to keep the fallback failure rate below 20-30%. When the failure rate gets too high, we rebuild the fallback list. Regular Tasks This ticket is the parent ticket for the next fallback rebuild: https://trac.torproject.org/projects/tor/ticket/30971 This ticket contains the "offer list" changes that relay operators have requested. I usually commit them all at once, but you should feel free to do them incrementally: https://trac.torproject.org/projects/tor/ticket/30972 Sometimes, we don't have enough relays on the offer list, and we have to ask relay operators to opt-in to the list. Ideally, we want at least 100 fallbacks, we usually have between 120-160. Future Work It's hard to verify changes to the offer list. Changes are usually sent by email or through trac tickets. There's no reliable trust path from the relay key to the email or ticket. The opt-in process is also a manual process. It can be time-consuming. To resolve these issues, I had planned to add a signed fallback offer line to relay descriptors: https://trac.torproject.org/projects/tor/ticket/24839 Instead of checking the list in the fallback-scripts repository, the script can check relay descriptors instead. (Or check both, during the transition period.) Unresolved Issues Fallbacks eventually see the entire set of clients. Clients that are active all the time may only ever contact one fallback. (Clients re-use the same fallback for authority keys, and then switch to the consensus as soon as possible.) But clients whose consensuses have expired will choose new fallbacks at random. Ideally, clients should select fallback (and maybe authority) guards. That is, they should retry previously-selected fallbacks. There are some tradeoffs here: a bad fallback guard can continue to manipulate its client's view of the network. We can avoid this issue by selecting multiple fallback guards. Clients will need persistent state to remember their guards, so transient systems like TAILS won't benefit from this change. T -- teor ----------------------------------------------------------------------

2 2

Walking Onions: week 7 update
by Nick Mathewson 17 Apr '20

17 Apr '20

Walking Onions: week 7 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html Week 5: Voting (part 2) https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html Week 6: Editing, voting rules, and client behavior (part 1) https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html == Clustering exit ports This week I tried to figure out how exits work with walking onions. A naive approach might have 65535 different routing indices in the ENDIVE (one for each TCP port), but that would give horrible performance: it would make the expanded ENDIVEs absolutely huge. Instead, we can cluster ports into "classes" based on which are always supported or not supported together. (For example, on the day I scanned the directory, every relay supporting any port in the range 49002-50000 supported all of them.) There are about 200 such classes on the current Tor network. But 200 exit indices is still probably too much: it seems like we'll need to cluster exit ports together in a more lossy way. For example, not every exit that supports port 80 supports port 443 or vice versa. But around 98% of them do -- so it we wouldn't be losing too much by using one exit index for "443 and 80", and not listing the 2% of exits that only support one of those ports. I wrote a short program [EXIT-ANALYSIS] to experiment with clustering strategies here, and settled on a greedy algorithm to get down to a smaller number of port classes by iteratively combining whichever pair of classes will cause us to lose the fewest number of (relay:port) possibilities. We can get down to 16 classes and lose only 0.07% of all the relay:port possibilities; we can get down to 8 classes and lose only 0.2%. That's not the whole story, though. Some of what we're losing would be things we'd miss. When the algorithm gives us 16 classes, it loses 75% of port 10000, 70% of port 23, and so on. Clearly not all ports are equivalent, and we should tune this approach to take that into account. We should also consider the bandwidth impact of these changes, and retool the algorithm so that it minimizes the bandwidth-per-port combination we have. We'd also likely want to hand-edit these port lists so that they make more sense semantically on their own. That way, we could take Teor's suggestion from last month and try to give exit relays the options to opt in to or out of human-intelligible partitions. Further, we could save some complexity by requiring dual-stack exits to support the same ports on IPv4 and IPv6. That way we wouldn't need to worry about divergent sets of port classes. == So what do we do with these clusters? If a client wants to connect to port 25, and uses a port-25-only index, that leaks the client's intention to the middle node. That's not so good. Instead, in the tech report [WHITEPAPER], one or both of Chelsea Komlo and Ian Goldberg came up with a mechanism called Delegated Verifiable Selection (DVS). With DVS, when the client sends a BEGIN cell, the relay can respond that it does not support the requested port, along with a SNIP from a relay that does. To prevent the relay from choosing an arbitrary SNIP, the BEGIN cell would have to include an auxiliary index value to be used when DVS is in place, and the END cell would need room for a SNIP. (Alternatively, clients could just make a 3-hop circuit and ask it for SNIPs to be used as exits on some other circuit.) == Voting on policy sets These policy clusters (and a few other things!) are too complex for the voting operations I had before. Fortunately, it's safe to treat them as opaque. I've added a new voting operation to allow authorities to vote on a bstr that is then decoded and treated as a cbor object (if it is valid and well-formed). It should be useful in other places too. == Migrating port classes Migrating from one set of port classes to another is nontrivial. Clients who have the old root document will expect the indices to correspond to one partition of ports, whereas clients with the new root document will expect another partition. To solve this, I've designed the system so that two sets of exit indices can exist concurrently while clients are moving to the newer partition of ports. Alternatively, we could give SNIPs a mechanism to contain a field saying "Your root document must be at least this new." I don't know which approach would be simpler; both seem like too much complexity for a feature that would only be used occasionally. == Next steps My next plan here is to write up onion services; I believe I've got the design there mostly figured out, and just need to write it down. After that, I'm planning to turn back and fill in all the missing subsections that I've written so far, and write a short introduction. There are a lot of missing sections, so that won't be super fast. I'm also going to need to show the whole thing to a beta reader or two to answer the question, "Would this make sense to somebody who doesn't already know what it says?" == Fun facts The script to generate exit port classes [EXIT-ANALYSIS] was written in Rust. If I'm counting right, it's only my second Rust program that actually does something useful! If you are one of those Rust enthusiasts who would like to critique it, I wouldn't mind at all -- just send me comments offline or make notes on the repository. (Only do this if you like critiquing newbie Rust that was never actually meant for production use. Don't do this if you just want to point out all the things that make it unsuitable for production use.) == Not-so-fun facts This week I've been fairly overwhelmed with the lead up to and down from the recent layoffs at Tor, in response to a downturn in funding during the COVID-19 pandemic [BLOGPOST]. Tor will continue, and this work will continue with it--but it is still a hard blow to have to lose so many excellent coworkers. (Yes, I'm still at Tor, for the record.) Do you know somebody who is interested in hiring one or more amazingly talented programmers, engineers, communications specialists, or project managers -- all with experience at productive remote work and all with dedication to improving people's lives? Please pass their information along to tor-alums(a)lists.torproject.org , and it will (after moderation) get forwarded to those affected. [BLOGPOST] https://blog.torproject.org/covid19-impact-tor [EXIT-ANALYSIS] https://github.com/nmathewson/walking-onions-wip/tree/master/exit-analysis [EXIT-PARTITION] https://lists.torproject.org/pipermail/tor-dev/2020-March/014182.html [WHITEPAPER] https://crysp.uwaterloo.ca/software/walkingonions/

1 0

Chutney code refactoring
by c 17 Apr '20

17 Apr '20

I was going to mention this on my pull request <https://github.com/torproject/chutney/pull/67> but I figured it would be much more fitting to have a more-general thread here. In the PR I mentioned how I believe that, if it quacks like a dict, it should be implemented as a dict. That way we (hopefully) run into fewer gotchas down the line, trying to reinvent the dict ourselves, and we have clearer self-docuemtnation ("oh, this class inherits dict, now I know kind of what to expect out of it"). I'm still trying to reason through chutney's code for how exactly inheritance works and how this inheritance can be expressed more clearly, for example it looks like the `Node` class is for a type of relay (authority, client, bridge, et cetera) and its `getN` method is to generate a number of those nodes. That is poorly documented and I only figured this out by seeing how `Node.getN` was being called in the tests themselves. To me, it makes more sense not to "overload" what a `Node` is; if we are using an object as a factory to create more `Node`s, then we should make that evident. The factory can then make identical nodes with similar properties, rather than one parent `Node` spawning children that refer back to their parent. For me this issue is mostly "it feels like a lot of mental overhead, and I'm sure it can be done better than it already is." But of course it's difficult to fix something without first understanding how it should behave. (Perhaps a test suite for the test suite? ;) But since we're dealing with a lot of inheritance in chutney, and since this inheritance is (in my opinion) expressed poorly, it can be quick to find oneself caught in a web of code with no clear way out. Here is what I have been considering lately, with the code overall. I hope that laying it out in points will help both myself and others try to divy up what could be improved about the code, work on it piecewise, and make sure nothing is reimplemented incorrectly. I know I'm new to the code and thus have not looked into it thoroughly enough to understand everything about it, but again, something in the back of my head is telling me that if it's this difficult to understand now, then it can stand to be changed. So please take my criticism lightly and only where it applies. I could be wrong about some things as well. But, as a testing suite, this is the place we absolutely need to be sure we get right, because with a hard-to-reason testing suite, we open ourselves up to bugs within the tests themselves. - Documentation especially for methods that the tests are supposed to call. In `Node` class, there is a comment "Users are expected to call these" with `__init__` and three other functions declared. `__init__` is fairly self-explanatory but the other three could stand to have the same docstrings that we are giving other functions. I think this should be my first task, before trying to mess with anything that moves. That way, we have a clearer overview of the testsuite's API, and then we can work the implementation around that. - Once it's clear to all of us (meaning, not only the core Chutney devs, but also outsiders such as myself) what the current interface does, we can see the strengths of that interface, maybe some design flaws if we deem any to exist, and some improvements we can make with the interface. What I mean is: here is the place to make any necessary adjustments to the API before we dive into implementation. - Now we visit the implementation. By this point I (and other non-core developers) should have a better understanding on what the tests *should* be doing, that we can then proceed to verify the correctness of the tests' implementations themselves. Here's where I would like to revisit cleaning up the code so that we make use of functions to abstract and "atomize" certain aspects of the code (like the currently-unused function isBoostrapped), where we should introduce more self-documenting names (as suggested, replace getAttribute names with isAttribute names for when we want to return bool values), et cetera. I'm outlining what *I* think the approach should be, and in hopes that everything else will line up and make more sense for us all. I do want to make sure I know what the code should be doing, because while I could inspect the code line by line and go through the logic branches myself, I'm going to be perfectly honest and say it gives me a headache :) I think it's much easier to reason about with compartmentalised, self-documenting (and docstringed) components, than to try to step through the code as it runs and take it in as one giant piece. Because, we're testing a complex piece of software (Tor). Tor itself can do a lot of things that can throw us off especially if we're less familiar with its implementation, yet. And that throws off our understanding of how Chutney is supposed to work, as well. A lot of the tests currently seem to be very conditional -- they pass or fail with little rhyme or reason, which is of course the best type of issues to debug :P -- so I don't see the test results helping me as well as refactoring the testsuite first. Again, let me know if I have anything wrong, made any unjust assumptions, showed my ignorance, et cetera. I'm going into this boldly and I hope that's the right call. Caitlin

3 2

Re: [tor-dev] Chutney code refactoring
by c 16 Apr '20

16 Apr '20

On Thu, 16 Apr 2020 20:12:18 +0400 meejah <meejah(a)meejah.ca> wrote: > As to the specific DictWrapper issue, I also found this confusing. It's > kind of tied into a custom templating system too. I would question > whether this is a good idea; there are many templating systems already > for Python -- what does Chutney's add that these don't already have? That's why I want to understand what Chutney is trying to do exactly so maybe I could find some pre-made wheels (no pun intended toward the package format) in Python's ecosystem that will fit the vehicle, rather than reinventing it. That is, of course, if such wheels make sense for Chutney to use. One thing I love about Python is its mature ecosystem. > While re-factoring I would try to reduce the amount of inheritence where > the user has to understand it to make progress. For example, any API > where a user is expected to override a class and re-write some methods. Agreed, I will keep this in mind. > I'd have to delve back into Chutney to give more specific suggestions, > but wanted to chime in and say that I also found it confusing and hard > to read Python :) So far your suggestions have been helpful, so thank you. Caitlin

1 0

Choosing a valid Circuit ID for an OR connection
by Eli Vakrat 14 Apr '20

14 Apr '20

Hello to the TOR dev team! My name is Eli, I'm a high school student from Israel and I'm currently trying to implement a TOR Client in Python. Currently, my project is configured so that the python client (OP) has its guard node set as my local machine (which is running a downloaded version of TOR). I do this for debugging purposes so that if I send a malformed cell as the implemented client, I can read the debug log that the OR generates and see what I did wrong. As I am writing this, I have successfully been able to make a v3 "in protocol" handshake by sending the proper VERSIONS and NETINFO cells that the OR has accepted. Right now, I am working on getting the CREATE cell to work (with the TAP handshake), and when I send the cell to the OR It sends a DESTROY cell with the following message in the debug log: Apr 14 12:24:51.166 [warn] Received create cell with unexpected circ_id 1. Closing. To my understanding, the problem here is that the circuit id I have chosen is not valid. I read from the tor spec that there is a range of acceptable circuit id's for each relay and it can be found in the long term 1024 bit RSA "signing key". My question to you all is, how can I extract that range for the circuit ID's (along with any other relevant information) from the signing key? In other words, what is the "signing key" made up of, and how can I get the information that its made of? The bottom line is I need to find out what valid circuit ID's I can send to OR's. I am having a lot of fun doing this project so far and I hope to hear back from anyone who has an answer :) Thanks in advance! Eli

2 1

[RFC] Proposal: A First Take at PoW Over Introduction Circuits
by George Kadianakis 14 Apr '20

14 Apr '20

Hello list, hope everyone is safe and doing well! I present you an initial draft of a proposal on PoW-based defences for onion services under DoS. The proposal is not finished yet and it needs tuning and fixing. There are many places marked with XXX and TODO around the proposal that should be addressed. The important part is that looking at the numbers it does seem like this proposal can work as a concept and serve its intended purpose. The most handwavey parts of the proposal right now are [INTRO_QUEUE] and [POW_SECURITY] and if this thing fails in the end, it's probably gonna be something that slipped over there. Hence, we should polish these sections before we proceed with any sort of engineering here. In any case, I decided to send it to the list even in premature form, so that it can serve as a stable point of reference in subsequent discussions. It can also be found in my git repo: https://github.com/asn-d6/torspec/tree/pow-over-intro Cheers and stay safe! --- Filename: xxx-pow-over-intro-v1 Title: A First Take at PoW Over Introduction Circuits Author: George Kadianakis Created: 2 April 2020 Status: Draft 0. Abstract This proposal aims to thwart introduction flooding DoS attacks by introducing a dynamic Proof-Of-Work protocol that occurs over introduction circuits. 1. Motivation So far our attempts at limiting the impact of introduction flooding DoS attacks on onion services has been focused on horizontal scaling with Onionbalance, optimizing the CPU usage of Tor and applying congestion control using rate limiting. While these measures move the goalpost forward, a core problem with onion service DoS is that building rendezvous circuits is a costly procedure both for the service and for the network. If we ever hope to have truly reachable global onion services, we need to make it harder for attackers to overload the service with introduction requests. This proposal achieves this by allowing onion services to specify an optional dynamic proof-of-work scheme that its clients need to participate in if they want to get served. With the right parameters, this proof-of-work scheme acts as a gatekeeper to block amplification attacks by attackers while letting legitimate clients through. 1.1. Threat model [THREAT_MODEL] 1.1.1. Attacker profiles [ATTACKER_MODEL] This proposal is written to thwart specific attackers. A simple PoW proposal cannot defend against all and every DoS attack on the Internet, but there are adverary models we can defend against. Let's start with some adversary profiles: "The script-kiddie" The script-kiddie has a single computer and pushes it to its limits. Perhaps it also has a VPS and a pwned server. We are talking about an attacker with total access to 10 Ghz of CPU and 10 GBs of RAM. We consider the total cost for this attacker to be zero $. "The small botnet" The small botnet is a bunch of computers lined up to do an introduction flooding attack. Assuming 500 medium-range computers, we are talking about an attacker with total access to 10 Thz of CPU and 10 TB of RAM. We consider the upfront cost for this attacker to be about $400. "The large botnet" The large botnet is a serious operation with many thousands of computers organized to do this attack. Assuming 100k medium-range computers, we are talking about an attacker with total access to 200 Thz of CPU and 200 TB of RAM. The upfront cost for this attacker is about $36k. We hope that this proposal can help us defend against the script-kiddie attacker and small botnets. To defend against a large botnet we would need more tools in our disposal (see [FUTURE_WORK]). {XXX: Do the above make sense? What other attackers do we care about? What other metrics do we care about? Network speed? I got the botnet costs from here [REF_BOTNET] Back up our claims of defence.} 1.1.2. User profiles [USER_MODEL] We have attackers and we have users. Here are a few user profiles: "The standard web user" This is a standard laptop/desktop user who is trying to browse the web. They don't know how these defences work and they don't care to configure or tweak them. They are gonna use the default values and if the site doesn't load, they are gonna close their browser and be sad at Tor. They run a 2Ghz computer with 4GB of RAM. "The motivated user" This is a user that really wants to reach their destination. They don't care about the journey; they just want to get there. They know what's going on; they are willing to tweak the default values and make their computer do expensive multi-minute PoW computations to get where they want to be. "The mobile user" This is a motivated user on a mobile phone. Even tho they want to read the news article, they don't have much leeway on stressing their machine to do more computation. We hope that this proposal will allow the motivated user to always connect where they want to connect to, and also give more chances to the other user groups to reach the destination. 1.1.3. The DoS Catch-22 [CATCH22] This proposal is not perfect and it does not cover all the use cases. Still, we think that by covering some use cases and giving reachability to the people who really need it, we will severely demotivate the attackers from continuing the DoS attacks and hence stop the DoS threat all together. Furthermore, by increasing the cost to launch a DoS attack, a big class of DoS attackers will disappear from the map, since the expected ROI will decrease. 2. System Overview 2.1. Tor protocol overview +----------------------------------+ | | +-------+ INTRO1 +-----------+ INTRO2 +--------+ | |Client |-------->|Intro Point|------->| PoW |-----------+ | +-------+ +-----------+ |Verifier| | | +--------+ | | | | | | | | | +----------v---------+ | | |Intro Priority Queue| | +---------+--------------------+---+ | | | Rendezvous | | | circuits | | | v v v The proof-of-work scheme specified in this proposal takes place during the introduction phase of the onion service protocol. It's an optional mechanism that only occurs if the service requires it. It can be enabled and disabled either through its torrc or through the control port. In summary, the following steps are taken for the protocol to complete: 1) Service encodes PoW parameters in descriptor [DESC_POW] 2) Client fetches descriptor and computes PoW [CLIENT_POW] 3) Client completes PoW and sends results in INTRO1 cell [INTRO1_POW] 4) Service verifies PoW and queues introduction based on PoW effort [SERVICE_VERIFY] 2.2. Proof-of-work overview 2.2.1. Primitives For our proof-of-work scheme we want to minimize the spread of resources between a motivated attacker and legitimate clients. This means that we are looking to minimize any benefits that GPUs or ACICs can offer to an attacker. For this reason we chose argon2 [REF_ARGON2] as the hash function for our proof-of-work scheme since it's well audited and GPU-resistant and to some extend ASIC-resistant as well. As a password hash function, argon2 by default outputs 32 bytes of hash, and takes as primary input a message and a nonce/salt. For the purposes of this specification we will define an argon2() function as: uint8_t hash_output[32] = argon2(uint8_t *message, uint8_t *nonce)'. See section [ARGON_PARAMS] for more information on the secondary inputs of argon2. 2.2.2. Dynamic PoW DoS is a dynamic problem where the attacker's capabilities constantly change, and hence we want our proof-of-work system to be dynamic and not stuck with a static difficulty setting. Hence, instead of forcing clients to go below a static target like in Bitcoin to be successful, we ask clients to "bid" using their PoW effort. Effectively, a client gets higher priority the higher effort they put into their proof-of-work. This is similar to how proof-of-stake works but instead of staking coins, you stake work. The benefit here is that legitimate clients who really care about getting access can spend a big amount of effort into their PoW computation, which should guarantee access to the service given reasonable adversary models. See [POW_SECURITY] for more details about these guarantees and tradeoffs. 3. Protocol specification 3.1. Service encodes PoW parameters in descriptor [DESC_POW] This whole protocol starts with the service encoding the PoW parameters in the 'encrypted' (inner) part of the v3 descriptor. As follows: "pow-params" SP type SP seed-b64 SP expiration-time NL [At most once] type: The type of PoW system used. We call the one specified here "v1" seed-b64: A random seed that should be used as the input to the PoW hash function. Should be 32 random bytes encoded in base64 without trailing padding. expiration-time: A timestamp after which the above seed expires and is no longer valid as the input for PoW. It's needed so that the size of our replay cache does not grow infinitely. It should be set to an hour in the future (+- some randomness). {TODO: PARAM_TUNING} {XXX: Expiration time makes us even more susceptible to clock skews, but it's needed so that our replay cache refreshes. How to fix this? See [CLIENT_BEHAVIOR] for more details.} 3.2. Client fetches descriptor and computes PoW [CLIENT_POW] If a client receives a descriptor with "pow-params", it should assume that the service is expecting a PoW input as part of the introduction protocol. In such cases, the client should have been configured with a specific PoW 'target' (which is a 32-byte integer similar to the 'target' of Bitcoin [REF_TARGET]). See [POW_SECURITY] for more information of how such a target should be set. For the purposes of this section, we will assume that the target has been set automatically by Tor, or the user configured it manually. Now the client parses the descriptor and extracts the PoW parameters. It makes sure that the expiration-time has not expired and if it has, it needs to fetch a new descriptor. To complete the PoW the client follows the following logic: a) Client generates 'nonce' as 32 random bytes. b) Client derives 'seed' by decoding 'seed-b64'. c) Client computes hash_output = argon2(seed, nonce) d) Client interprets hash_output as a 32-byte big-endian integer. e) Client checks if int(hash_output) <= target. e1) If yes, success! The client uses 'hash_output' as the hash and 'nonce' and 'seed' as its inputs. e2) If no, fail! The client interprets 'nonce' as a big-endian integer, increments it by one, and goes back to step (c). At the end of the above procedure, the client should have a triplet (hash_output, seed, nonce) that can be used as the answer to the PoW puzzle. How quickly this happens depends solely on the 'target' parameter. 3.3. Client sends PoW in INTRO1 cell [INTRO1_POW] Now that the client has an answer to the puzzle it's time to encode it into an INTRODUCE1 cell. To do so the client adds an extension to the encrypted portion of the INTRODUCE1 cell by using the EXTENSIONS field (see [PROCESS_INTRO2] section in rend-spec-v3.txt). The encrypted portion of the INTRODUCE1 cell only gets read by the onion service and is ignored by the introduction point. We propose a new EXT_FIELD_TYPE value: [01] -- PROOF_OF_WORK The EXT_FIELD content format is: POW_VERSION [1 byte] POW_SEED [32 bytes] POW_NONCE [32 bytes] POW_OUTPUT [32 bytes] where: POW_VERSION is 1 for the protocol specified in this proposal POW_SEED is 'seed' from the section above POW_NONCE is 'nonce' from the section above POW_OUTPUT is 'hash_output' from the section above {XXX: do we need POW_VERSION? Perhaps we can use EXT_FIELD_TYPE as version} {XXX: do we need to encode the SEED? Perhaps we can ommit it since the service already knows it. But what happens in cases of desynch, if client has diff seed from service?} {XXX: Do we need to include the output? Probably not. The service has to compute it anyway during verification. What's the use?} This will increase the INTRODUCE1 payload size by 99 bytes since the extension type and length is 2 extra bytes, the N_EXTENSIONS field is always present and currently set to 0 and the EXT_FIELD is 97 bytes. According to ticket #33650, INTRODUCE1 cells currently have more than 200 bytes available. 3.4. Service verifies PoW and handles the introduction [SERVICE_VERIFY] When a service receives an INTRODUCE1 with the PROOF_OF_WORK extension, it should check its configuration on whether proof-of-work is required to complete the introduction. If it's not required, the extension SHOULD BE ignored. If it is required, the service follows the procedure detailed in this section. 3.4.1. PoW verification To verify the client's proof-of-work the service extracts (hash_output, seed, nonce) from the INTRODUCE1 cell and MUST do the following steps: 1) Make sure that the client's seed is identical to the active seed. 2) Check the client's nonce for replays (see [REPLAY_PROTECTION] section). 3) Verify that 'hash_output =?= argon2(seed, nonce) If any of these steps fail the service MUST ignore this introduction request and abort the protocol. If all the steps passed, then the circuit is added to the introduction queue as detailed in section [INTRO_QUEUE]. 3.4.1.1. Replay protection [REPLAY_PROTECTION] The service MUST NOT accept introduction requests with the same (seed, nonce) tuple. For this reason a replay protection mechanism must be employed. The simplest way is to use a simple hash table to check whether a (seed, nonce) tuple has been used before for the actiev duration of a seed. Depending on how long a seed stays active this might be a viable solution with reasonable memory/time overhead. If there is a worry that we might get too many introductions during the lifetime of a seed, we can use a Bloom filter as our replay cache mechanism. The probabilistic nature of Bloom filters means that sometimes we will flag some connections as replays even if they are not; with this false positive probability increasing as the number of entries increase. However, with the right parameter tuning this probability should be negligible and well handled by clients. {TODO: PARAM_TUNING} 3.4.2. The Introduction Queue [INTRO_QUEUE] 3.4.2.1. Adding introductions to the introduction queue When PoW is enabled and a verified introduction comes through, the service instead of jumping straight into rendezvous, queues it and prioritizes it based on how much effort was devoted by the client to PoW. This means that introduction requests with high effort should be prioritized over those with low effort. To do so, the service maintains an "introduction priority queue" data structure. Each element in that priority queue is an introduction request, and its priority is the effort put into its PoW: When a verified introduction comes through, the service interprets the PoW hash as a 32-byte big-endian integer 'hash_int' and based on that integer it inserts it into the right position of the priority_queue: The smallest 'hash_int' goes forward in the queue. If two elements have the same value, the older one has priority over the newer one. {XXX: Is this operation with 32-bytes integers expensive? How to make cheaper?} {TODO: PARAM_TUNING: If the priority queue is only ordered based on the effort what attacks can happen in various scenarios? Do we want to order on time+effort? Which scenarios and attackers should we examine here?} {TODO: PARAM_TUNING: What's the max size of the queue? How do we trim it? Can we use WRED usefully?} 3.4.2.2. Handling introductions from the introduction queue [HANDLE_QUEUE] The service should handle introductions by pulling from the introduction queue. Similar to how our cell scheduler works, the onion service subsystem will poll the priority queue every 100ms tick and process the first 20 cells from the priority queue (if they exist). The service will perform the rendezvous and the rest of the onion service protocol as normal. With this tempo, we can process 200 introduction cells per second. {XXX: Is this good?} {TODO: PARAM_TUNING: STRAWMAN: This needs hella tuning. Processing 20 cells per 100ms is probably unmaintainable, since each cell is quite expensive: doing so involving path selection, crypto and making circuits. We will need to profile this procedure and see how we can do this scheduling better.} {XXX: This might be a nice place to promote multithreading. Queues and pools are nice objects to do multithreading since you can have multiple threads pull from the queue, or leave stuff on the queue. Not sure if this should be in the proposal tho.} 4. Attacker strategies [ATTACK_META] Now that we defined our protocol we need to start tweaking the various knobs. But before we can do that, we first need to understand a few high-level attacker strategies to see what we are fighting against. 4.1.1. Total overwhelm strat Given the way the introduction queue works (see [HANDLE_QUEUE]), a very effective strategy for the attacker is to totally overwhelm the queue processing by sending more high-effort introductions than the onion service can handle at any given tick. To do so, the attacker would have to send at least 20 high-effort introduction cells every 100ms, where high-effort is a PoW which is above the estimated level of "the motivated user" (see [USER_MODEL]). An easier attack for the adversary, is the same strategy but with introduction cells that are all above the comfortable level of "the standard user" (see [USER_MODEL]). This would block out all standard users and only allow motivated users to pass. {XXX: What other attack strategies we should care about?} 5. Parameter tuning [POW_SECURITY] There are various parameters in this system that need to be tuned. We will first start by tuning the default difficulty of our PoW system. That's gonna define an expected time for attackers and clients to succeed. We are then gonna tune the parameters of the argon2 hash function. That will define the resources that an attacker needs to spend to overwhelm the onion service, the resources that the service needs to spend to verify introduction requests, and the resources that legitimate clients need to spend to get to the onon service. 5.1. PoW Difficulty settings The difficulty setting of our PoW basically dictates how difficult it should be to get a success in our PoW system. In classic PoW systems, "success" is defined as getting a hash output below the "target". However, since our system is dynamic, we define "success" as an abstract high-effort computation. Even tho our system is dynamic, we still need default difficulty settings that will define the metagame. The client and attacker can still aim higher or lower, but for UX purposes and for analysis purposes we do need to define some difficulties. We hence created the table (see [REF_TABLE]) below which shows how much time a legitimate client with a single machine should expect to burn before they get a single success. The x-axis is how many successes we want the attacker to be able to do per second: the more successes we allow the adversary, the more they can overwhelm our introduction queue. The y-axis is how many machines the adversary has in her disposal, ranging from just 5 to 1000. =============================================================== | Expected Time (in seconds) Per Success For One Machine | =========================================================================== | | | Attacker Succeses 1 5 10 20 30 50 | | per second | | | | 5 5 1 0 0 0 0 | | 50 50 10 5 2 1 1 | | 100 100 20 10 5 3 2 | | Attacker 200 200 40 20 10 6 4 | | Boxes 300 300 60 30 15 10 6 | | 400 400 80 40 20 13 8 | | 500 500 100 50 25 16 10 | | 1000 1000 200 100 50 33 20 | | | ============================================================================ Here is how you can read the table above: - If an adversary has a botnet with 1000 boxes, and we want to limit her to 1 success per second, then a legitimate client with a single box should be expected to spend 1000 seconds getting a single success. - If an adversary has a botnet with 1000 boxes, and we want to limit her to 5 successes per second, then a legitimate client with a single box should be expected to spend 200 seconds getting a single success. - If an adversary has a botnet with 500 boxes, and we want to limit her to 5 successes per second, then a legitimate client with a single box should be expected to spend 100 seconds getting a single success. - If an adversary has access to 50 boxes, and we want to limit her to 5 successes per second, then a legitimate client with a single box should be expected to spend 10 seconds getting a single success. - If an adversary has access to 5 boxes, and we want to limit her to 5 successes per second, then a legitimate client with a single box should be expected to spend 1 seconds getting a single success. With the above table we can create some profiles for default values of our PoW difficulty. So for example, we can use the last case as the default parameter for Tor Browser, and then create three more profiles for more expensive cases, scaling up to the first case which could be hardest since the client is expected to spend 15 minutes for a single introduction. {TODO: PARAM_TUNING You can see that this section is completely CPU/memory agnostic, and it does not take into account potential optimizations that can come from GPU/ASICs. This is intentional so that we don't put more variables into this equation right now, but as this proposal moves forward we will need to put more concrete values here.} 5.2. Argon2 parameters [ARGON_PARAMS] We now need to define the secondary argon2 parameters as defined in [REF_ARGON2]. This includes the number of lanes 'h', the memory size 'm', the number of iterations 't'. Section 9 of [REF_ARGON2] recommends an approach of how to tune these parameters. To tune these parameters we are looking to *minimize* the verification speed of an onion service, while *maximizing* the sparse resources spent by an adversary trying to overwhelm the service using [ATTACK_META]. When it comes to verification speed, to verify a single introduction cell the service needs to do a single argon2 call: so the service will need to do hundreds of those per second as INTRODUCE2 cells arrive. The service will have to do this verification step even for very cheap zero-effort PoW received, so this has to be a cheap procedure so that it doesn't become a DoS vector of each own. Hence each individual argon2 call must be cheap enough to be able to be done comfortably and plentifuly by an onion service with a single host (or horizontally scaled with Onionbalance). At the same time, the adversary will have to do thousands of these calls if she wants to make high-effort PoW, so it's this assymetry that we are looking to exploit here. Right now, the most expensive resource for adversaries is the RAM size, and that's why we chose argon2 which is memory-hard. To minmax this game we will need {TODO: PARAM_TUNING: I've had a hard time minmaxing this game for argon2. Even argon2 invocations with a small memory parameter will take multiple milliseconds to run on my machine, and the parameters recommended in section 8 of the paper all take many hundreds of milliseconds. This is just not practical for our use case, since we want to process hundreds of such PoW per second... I also did not manage to find a benchmark of argon2 calls for different CPU/GPU/FPGA configurations.} 5. Client behavior [CLIENT_BEHAVIOR] This proposal introduces a bunch of new ways where a legitimate client can fail to reach the onion service. Furthermore, there is currently no end-to-end way for the onion service to inform the client that the introduction failed. The INTRO_ACK cell is not end-to-end (it's from the introduction point to the client) and hence it does not allow the service to inform the client that the rendezvous is never gonna occur. Let's examine a few such cases: 5.1. Timeout issues Alice can fail to reach the onion service if her introduction request falls off the priority queue, or if the priority queue is so big that the connection times out. Is building a new introduction circuit sufficient here? Or do we need to build an end-to-end mechanism over the introduction circuit to inform her? {XXX} How should timeout values change here since the priority queue will cause bigger delays than usual to rendezvous? Can there be some feedback mechanism to inform the client of its queue position or ETA? 5.2. Seed expiration issues As mentioned in [DESC_POW], the expiration timestamp on the PoW seed can cause issues with clock skewed clients. Furthermore, even not clock skewed clients can encounter TOCTOU-style race conditions here. How should this be handled? Should we have multiple active seeds at the same time similar to how we have overlapping descriptors and time periods in v3? This would solve the problem but it grows the complexity of the system substantially. {XXX} 5.3. Other descriptor issues Another race condition here is if the service enables PoW, while a client has a cached descriptor. How will the client notice that PoW is needed? Does it need to fetch a new descriptor? Should there be another feedback mechanism? {XXX} 5. Discussion 5.1. UX This proposal has user facing UX consequences. Here are a few UX approaches with increasing engineering difficulty: a) Tor Browser needs a "range field" which the user can use to specify how much effort they want to spend in PoW if this ever occurs while they are browsing. The ranges could be from "Easy" to "Difficult", or we could try to estimate time using an average computer. This setting is in the Tor Browser settings and users need to find it. b) We start with a default effort setting, and then we use the new onion errors (see #19251) to estimate when an onion service connection has failed because of DoS, and only then we present the user a "range field" which they can set dynamically. Detecting when an onion service connection has failed because of DoS can be hard because of the lack of feedback (see [CLIENT_BEHAVIOR]) c) We start with a default effort setting, and if things fail we automatically try to figure out an effort setting that will work for the user by doing some trial-and-error connections with different effort values. Until the connection succeeds we present a "Service is overwhelmed, please wait" message to the user. For this proposal to work initially we need at least (a), and then we can start thinking of how far we want to take it. 5.2. Future directions [FUTURE_WORK] This is just the beginning in DoS defences for Tor and there are various future avenues that we can investigate. Here is a brief summary of these: "More advanced PoW schemes" -- We could use more advanced memory-hard PoW schemes like MTP-argon2 or Itsuku to make it even harder for adversaries to create successful PoWs. Unfortunately these schemes have much bigger proof sizes, and they won't fit in INTRODUCE1 cells. See #31223 for more details. "Third-party anonymous credentials" -- We can use anonymous credentials and a third-party token issuance server on the clearnet to issue tokens based on PoW or CAPTCHA and then use those tokens to get access to the service. See [REF_CREDS] for more details. "PoW + Anonymous Credentials" -- We can make a hybrid of the above ideas where we present a hard puzzle to the user when connecting to the onion service, and if they solve it we then give the user a bunch of anonymous tokens that can be used in the future. This can all happen between the client and the service without a need for a third party. All of the above approaches are much more complicated than this proposal, and hence we want to start easy before we get into more serious projects. 5.3. Environment We love the environment! We are concerned of how PoW schemes can waste energy by doing useless hash iterations. Here is a few reasons we still decided to pursue a PoW approach here: "We are not making things worse" -- DoS attacks are already happening and attackers are already burning energy to carry them out both on the attacker side, on the service side and on the network side. We think that asking legitimate clients to carry out PoW computations is not gonna affect the equation too much, since an attacker right now can very quickly cause the same damage that hundreds of legitimate clients do a whole day. "We hope to make things better" -- The hope is that proposals like this will make the DoS actors go away and hence the PoW system will not be used. As long as DoS is happening there will be a waste of energy, but if we manage to demotivate them with technical means, the network as a whole will less wasteful. Also see [CATCH22] for a similar argument. 6. References [REF_ARGON2]: https://github.com/P-H-C/phc-winner-argon2/blob/master/argon2-specs.pdf https://password-hashing.net/#argon2 [REF_TABLE]: The table is based on the script below plus some manual editing for readability: https://gist.github.com/asn-d6/99a936b0467b0cef88a677baaf0bbd04 [REF_BOTNET]: https://media.kasperskycontenthub.com/wp-content/uploads/sites/43/2009/07/0… [REF_CREDS]: https://lists.torproject.org/pipermail/tor-dev/2020-March/014198.html [REF_TARGET]: https://en.bitcoin.it/wiki/Target

3 5

Walking onions status update: week 6 notes
by Nick Mathewson 10 Apr '20

10 Apr '20

Walking Onions: week 6 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html Week 5: Voting (part 2) https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html This week I did a pass through all of the proposal that's written so far, trying to revise it for consistency and to look for unexpected gaps. Then I wrote a first take on a section about client behavior, and outlined sections about how to keep onions services working. == Fixing inconsistencies There were, predictably, some places where I had wrote things in an introduction to a section, and then turn around and did them differently later in the section. I tried to make those consistent when I could. For one example, I think Microdescs should now be embedded in votes. They were separate before, for space reasons, but now that we have defined support for compression and diffs in voting, we don't need to keep votes as small as we did before. But when I went to explain voting, I didn't actually give a place to put in microdescriptors. This is fixed now. == Example rules, and changes to voting. Thanks to Teor's help and feedback on previous iterations of voting, the voting operations [VOTING] are now more consistent, and specified more consistently. I had to describe a "sorting" operation for use with voting, since CBOR's default "canonical" ordering is not what we actually want semantically. To try to make sure voting could do what we need, I wrote up an example VotingRules object in a new appendix [EXRULES]. This turned up a few impedance mismatches between the available voting rules, the SNIP format, and the ENDIVE format. I fixed some of these, and marked others with "XXX" strings to come back and fix on the next pass through voting. == More index and extension protocol work I've found a need for a new link specifier (LS) for use with onion services. Unlike a regular routing index LS, this kind includes a routing index and a count of how many relays to skip after that position. This lets a client say, for example, "Give me the third relay after the one whose HDirIndex is 0xbf9630". (In order to answer this query, the extending relay needs to return not only the SNIP for the third relay, but all SNIPs between the first relay and that relay, so that the client can trust the answer.) This kind of LS can be used with the new NIL handshake if the client _only_ wants to fetch the descriptors; the details are in the circuit-protocol section [EXTENDING]. == Client operations, part 1 In a new section 06, I've sketched out solutions for guards, bridges, bootstrapping, and several kinds of path restriction [CLIENTS]. In doing so I've turned up a possible need for new IP-based and key-based indices; I've added those to a new appendix where I'm listing the different indices I think we'll need [INDEXLIST]. == Outlines of work to come I've added outlines for a new section 7 about making onion services work, and a new section 8 about tracking some new ways that relays might have to misbehave. I've also added quick sketches for a pair of additional proposals we'll need: one for a better family notation, and one for finally migrating away from the last vestiges of TAP. But none of those are quite next: In the coming week, if time permits, I want to get exit policies figured out. They are the biggest missing piece of section 06 right now, and seem to me to be the biggest remaining gap in the specification. == Fun facts: At 109KiB, this is now the longest Tor proposal ever. I will need to come up with a different fun fact next week. == References: [CLIENTS] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/06-clien… [EXTENDING] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/05-exten… [EXRULES] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/AF-examp… [INDEXLIST] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/AG-index… [VOTING] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/03-votin…

1 0

Walking onions status update: week 5 notes
by Nick Mathewson 06 Apr '20

06 Apr '20

Walking Onions: week 5 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html This week found me still rather distracted by fallout from current events and preparations for more isolation at home. I didn't get a complete rewrite done. But I do, however, think I figured out voting. == The big idea for voting Our current system for voting on relay properties is fairly piecemeal: every property has its own independent specification in dir-spec.txt [DIRSPEC], which can get kind of cumbersome. Every time we have a new property to specify, if it can't be defined as a flag, we need to define a new consensus method that puts it in the consensus or the microdescriptors. Until authorities have upgraded to support the new method, they can't sign consensuses generated with it, since they don't know how those consensuses are generated. We can solve both problems by defining "vote tabulation" as a set of parameterized operations operating over a set of CBOR inputs. Individual operations can be simple, like "Take the median of all integer votes", or progressively more complex. What's more, we can specify these voting operations themselves as data, which gives us more advantages: First, that we no longer need independent English-language specs for each field in the consensus--and second, that we can introduce new voting rules or change the existing ones without adding new code. As long as a quorum of authorities agree about the rules for voting on some field, the other authorities can safely follow those rules, and include the new field in the consensus ENDIVE and SNIPs. We would only need to introduce new consensus methods when we wanted to introduce support for an entirely new operation. After voting is done, some transformations are needed to transform the objects it produces. SNIP signatures need to be signed, indexes built, and so on. I've left this unspecified for now, since I am likely to return to the SNIP and ENDIVE formats again in the coming weeks, and I'll want to tweak the transformation quite a lot. This is not something I would want to implement in C. Rust would be a better fit -- or some functional language. Fortunately, only authorities need to follow this algorithm. I have initial specifications written for these operations that I believe are sufficient (or nearly so?) for all the things that we do in voting now. These are in the draft voting spec as it stands [VOTING]. I expect that they are full of errors, but I wouldn't be too concerned right now; I'm going to need to do another pass over them, but I want to return to voting a bit later (see below) once I have more SNIP uses specified. == Backward compatibility in voting There are two separate backward compatibility issues in voting. The first issue is easier: how to operate when we aren't sure whether all the authorities have upgraded to support this new voting format. For this purpose, I'm including legacy votes as part of the new vote format, signatures and all. This will make the new votes bigger until all the authorities have definitely upgraded, but other changes (voting diffs and compression) should offset this risk. We can stop sending legacy votes when everybody has upgraded. (I thought about having an "implicit version" of each legacy vote derived from the new vote, but that seemed riskier, and like it would involve lots more new code of dubious benefit.) The second issue is a bit harder: for clients that haven't upgraded to walking onions, we'll need to generate legacy consensus documents for a some time. This doesn't need to follow exactly the same rules as consensus voting uses today: instead we should have legacy consensus documents derived from the CBOR consensus, so that we only need to maintain one voting algorithm long term. I've left this part unspecified for now too; I'll be comping back to it after a round of revisions and SNIP usage design. == Next steps I plan to spend the first part of week going over what I've written so far and trying to make it correct and consistent, and to fill in the baps that I have. After that, I'm planning to write the example sections for how circuit extension works. If time remains, this is when I finally move on to section 6 of the proposal, which will be pretty involved -- I've got to explain how to implement all (or most?) of our current client behavior using walking onions, and what that implies about the set of indexes and fields in our documents. == Other updates Our paper got into USENIX Security 2020! (This is the paper about Walking Onions that Chelsea Komlo and Ian Goldberg wrote with me. It has basically the same content as our existing tech report [TECHREPORT].) The conference is scheduled in mid-August, in Boston: we'll see whether people are gathering in groups by then. If not, I imagine we'll be putting together some kind of video. [DIRSPEC] https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt [TECHREPORT] https://crysp.uwaterloo.ca/software/walkingonions/ [VOTING] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/03-votin…

1 0

Re: [tor-dev] Chutney tests fail for tor/maint-0.3.5 (bug #33677)
by c 01 Apr '20

01 Apr '20

(Forgot to hit "reply all" in my mail client) On Mon, 30 Mar 2020 13:22:21 +1000 teor <teor(a)riseup.net> wrote: > This ticket is a cleanup ticket, after some other changes have been made. > I've edited the ticket description to make that clearer. Understood, thank you for clarifying. > Check for onion service descriptor uploads: > https://trac.torproject.org/projects/tor/ticket/33609 > > Someone else is working on the microdescriptor changes at the moment. > > Would you like to start working on the onion service descriptor changes? Sure, it would give me an opportunity to learn about onion descriptors in more detail. I will get started on it. But first, looking at it more, I think my struggle with understanding what to do stems from my unfamiliarity with the Chutney codebase (first I have heard of the tool was with this project, even though commits date back to 2011). I need to make sure I understand #33609 sufficiently: - Is the requested functionality only for Chutney or will Tor potentially need any changes to allow for HS verification? - So I know where to begin looking in the codebase, the ticket wants us to "check each onion service log" -- is this referring to Tor log output (such as the instances chutney spawns), chutney-specific logs, or something else entirely? - For "check v2 and v3 onion services" -- check if they've propagated the network? - For "call it an extra 200% 'bootstrap' stage" -- again is this chutney-specific? I only know bootstrapping percentage from Tor notice-level logging and obviously it only goes up to 100%, so I'm wondering if "200%" is a magic number here or something arbitrary. From this and the parent #33050 it doesn't seem to me like the request is very clear. I am reading proposals 311-313 after sending this message so maybe I can come across some answers to my questions/confusions via the proposals themselves. The proposals will probably give me a better idea of the work I am in for overall, too, and perhaps I should have come across them sooner. I figure it is wise regardless to ask for clarification here and read while I wait for feedback. Efficiency and all :) Caitlin

2 1