April 2020 - tor-dev - lists.torproject.org

Walking onions status update: week 6 notes
by Nick Mathewson 10 Apr '20

10 Apr '20

Walking Onions: week 6 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html Week 5: Voting (part 2) https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html This week I did a pass through all of the proposal that's written so far, trying to revise it for consistency and to look for unexpected gaps. Then I wrote a first take on a section about client behavior, and outlined sections about how to keep onions services working. == Fixing inconsistencies There were, predictably, some places where I had wrote things in an introduction to a section, and then turn around and did them differently later in the section. I tried to make those consistent when I could. For one example, I think Microdescs should now be embedded in votes. They were separate before, for space reasons, but now that we have defined support for compression and diffs in voting, we don't need to keep votes as small as we did before. But when I went to explain voting, I didn't actually give a place to put in microdescriptors. This is fixed now. == Example rules, and changes to voting. Thanks to Teor's help and feedback on previous iterations of voting, the voting operations [VOTING] are now more consistent, and specified more consistently. I had to describe a "sorting" operation for use with voting, since CBOR's default "canonical" ordering is not what we actually want semantically. To try to make sure voting could do what we need, I wrote up an example VotingRules object in a new appendix [EXRULES]. This turned up a few impedance mismatches between the available voting rules, the SNIP format, and the ENDIVE format. I fixed some of these, and marked others with "XXX" strings to come back and fix on the next pass through voting. == More index and extension protocol work I've found a need for a new link specifier (LS) for use with onion services. Unlike a regular routing index LS, this kind includes a routing index and a count of how many relays to skip after that position. This lets a client say, for example, "Give me the third relay after the one whose HDirIndex is 0xbf9630". (In order to answer this query, the extending relay needs to return not only the SNIP for the third relay, but all SNIPs between the first relay and that relay, so that the client can trust the answer.) This kind of LS can be used with the new NIL handshake if the client _only_ wants to fetch the descriptors; the details are in the circuit-protocol section [EXTENDING]. == Client operations, part 1 In a new section 06, I've sketched out solutions for guards, bridges, bootstrapping, and several kinds of path restriction [CLIENTS]. In doing so I've turned up a possible need for new IP-based and key-based indices; I've added those to a new appendix where I'm listing the different indices I think we'll need [INDEXLIST]. == Outlines of work to come I've added outlines for a new section 7 about making onion services work, and a new section 8 about tracking some new ways that relays might have to misbehave. I've also added quick sketches for a pair of additional proposals we'll need: one for a better family notation, and one for finally migrating away from the last vestiges of TAP. But none of those are quite next: In the coming week, if time permits, I want to get exit policies figured out. They are the biggest missing piece of section 06 right now, and seem to me to be the biggest remaining gap in the specification. == Fun facts: At 109KiB, this is now the longest Tor proposal ever. I will need to come up with a different fun fact next week. == References: [CLIENTS] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/06-clien… [EXTENDING] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/05-exten… [EXRULES] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/AF-examp… [INDEXLIST] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/AG-index… [VOTING] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/03-votin…

1 0

Walking onions status update: week 5 notes
by Nick Mathewson 06 Apr '20

06 Apr '20

Walking Onions: week 5 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html This week found me still rather distracted by fallout from current events and preparations for more isolation at home. I didn't get a complete rewrite done. But I do, however, think I figured out voting. == The big idea for voting Our current system for voting on relay properties is fairly piecemeal: every property has its own independent specification in dir-spec.txt [DIRSPEC], which can get kind of cumbersome. Every time we have a new property to specify, if it can't be defined as a flag, we need to define a new consensus method that puts it in the consensus or the microdescriptors. Until authorities have upgraded to support the new method, they can't sign consensuses generated with it, since they don't know how those consensuses are generated. We can solve both problems by defining "vote tabulation" as a set of parameterized operations operating over a set of CBOR inputs. Individual operations can be simple, like "Take the median of all integer votes", or progressively more complex. What's more, we can specify these voting operations themselves as data, which gives us more advantages: First, that we no longer need independent English-language specs for each field in the consensus--and second, that we can introduce new voting rules or change the existing ones without adding new code. As long as a quorum of authorities agree about the rules for voting on some field, the other authorities can safely follow those rules, and include the new field in the consensus ENDIVE and SNIPs. We would only need to introduce new consensus methods when we wanted to introduce support for an entirely new operation. After voting is done, some transformations are needed to transform the objects it produces. SNIP signatures need to be signed, indexes built, and so on. I've left this unspecified for now, since I am likely to return to the SNIP and ENDIVE formats again in the coming weeks, and I'll want to tweak the transformation quite a lot. This is not something I would want to implement in C. Rust would be a better fit -- or some functional language. Fortunately, only authorities need to follow this algorithm. I have initial specifications written for these operations that I believe are sufficient (or nearly so?) for all the things that we do in voting now. These are in the draft voting spec as it stands [VOTING]. I expect that they are full of errors, but I wouldn't be too concerned right now; I'm going to need to do another pass over them, but I want to return to voting a bit later (see below) once I have more SNIP uses specified. == Backward compatibility in voting There are two separate backward compatibility issues in voting. The first issue is easier: how to operate when we aren't sure whether all the authorities have upgraded to support this new voting format. For this purpose, I'm including legacy votes as part of the new vote format, signatures and all. This will make the new votes bigger until all the authorities have definitely upgraded, but other changes (voting diffs and compression) should offset this risk. We can stop sending legacy votes when everybody has upgraded. (I thought about having an "implicit version" of each legacy vote derived from the new vote, but that seemed riskier, and like it would involve lots more new code of dubious benefit.) The second issue is a bit harder: for clients that haven't upgraded to walking onions, we'll need to generate legacy consensus documents for a some time. This doesn't need to follow exactly the same rules as consensus voting uses today: instead we should have legacy consensus documents derived from the CBOR consensus, so that we only need to maintain one voting algorithm long term. I've left this part unspecified for now too; I'll be comping back to it after a round of revisions and SNIP usage design. == Next steps I plan to spend the first part of week going over what I've written so far and trying to make it correct and consistent, and to fill in the baps that I have. After that, I'm planning to write the example sections for how circuit extension works. If time remains, this is when I finally move on to section 6 of the proposal, which will be pretty involved -- I've got to explain how to implement all (or most?) of our current client behavior using walking onions, and what that implies about the set of indexes and fields in our documents. == Other updates Our paper got into USENIX Security 2020! (This is the paper about Walking Onions that Chelsea Komlo and Ian Goldberg wrote with me. It has basically the same content as our existing tech report [TECHREPORT].) The conference is scheduled in mid-August, in Boston: we'll see whether people are gathering in groups by then. If not, I imagine we'll be putting together some kind of video. [DIRSPEC] https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt [TECHREPORT] https://crysp.uwaterloo.ca/software/walkingonions/ [VOTING] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/03-votin…

1 0

Re: [tor-dev] Chutney tests fail for tor/maint-0.3.5 (bug #33677)
by c 01 Apr '20

01 Apr '20

(Forgot to hit "reply all" in my mail client) On Mon, 30 Mar 2020 13:22:21 +1000 teor <teor(a)riseup.net> wrote: > This ticket is a cleanup ticket, after some other changes have been made. > I've edited the ticket description to make that clearer. Understood, thank you for clarifying. > Check for onion service descriptor uploads: > https://trac.torproject.org/projects/tor/ticket/33609 > > Someone else is working on the microdescriptor changes at the moment. > > Would you like to start working on the onion service descriptor changes? Sure, it would give me an opportunity to learn about onion descriptors in more detail. I will get started on it. But first, looking at it more, I think my struggle with understanding what to do stems from my unfamiliarity with the Chutney codebase (first I have heard of the tool was with this project, even though commits date back to 2011). I need to make sure I understand #33609 sufficiently: - Is the requested functionality only for Chutney or will Tor potentially need any changes to allow for HS verification? - So I know where to begin looking in the codebase, the ticket wants us to "check each onion service log" -- is this referring to Tor log output (such as the instances chutney spawns), chutney-specific logs, or something else entirely? - For "check v2 and v3 onion services" -- check if they've propagated the network? - For "call it an extra 200% 'bootstrap' stage" -- again is this chutney-specific? I only know bootstrapping percentage from Tor notice-level logging and obviously it only goes up to 100%, so I'm wondering if "200%" is a magic number here or something arbitrary. From this and the parent #33050 it doesn't seem to me like the request is very clear. I am reading proposals 311-313 after sending this message so maybe I can come across some answers to my questions/confusions via the proposals themselves. The proposals will probably give me a better idea of the work I am in for overall, too, and perhaps I should have come across them sooner. I figure it is wise regardless to ask for clarification here and read while I wait for feedback. Efficiency and all :) Caitlin

2 1

txtorcon 20.0.0
by meejah 31 Mar '20

31 Mar '20

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 I'm pleased to announce txtorcon 20.0.0. This fixes a few bugs and officially deprecates Python 2 support. * Use real GeoIP database or nothing (https://github.com/meejah/txtorcon/issues/250) * Change abstract base classes import in preperation for Python 3.8 (thanks @glowatsk) * Python 3.4 is no longer supported * Python 2 is deprecated; all new code should be Python 3. Support for Python 2 will be removed in a future release. You can download the release from PyPI or GitHub (or of course "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/20.0.0 https://github.com/meejah/txtorcon/releases/tag/v20.0.0 Releases are also available from the hidden service: http://timaq4ygg2iegci7.onion/txtorcon-20.0.0.tar.gz http://timaq4ygg2iegci7.onion/txtorcon-20.0.0.tar.gz.asc Or via a "version 3" service: http://fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion/txtor… http://fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion/txtor… You can verify the sha256sum of both by running the following 4 lines in a shell wherever you have the files downloaded: cat <<EOF | sha256sum --check 122cd081786396f57718adda1c1a5eb01b8e9a4832fcd140918b45c10359377a txtorcon-20.0.0.tar.gz 5c1145566a8733105da851b794e7b15cb56cea48693635b55c4bddfb79fe629d txtorcon-20.0.0-py2.py3-none-any.whl EOF thanks, meejah -----BEGIN PGP SIGNATURE----- iQFFBAEBCgAvFiEEnVor1WiOy4id680/wmAoAxKAaacFAl6EDnURHG1lZWphaEBt ZWVqYWguY2EACgkQwmAoAxKAaadv4AgAkIny/Pt55kEppkxlqu4/B4ddfU7oQNLD 8lwoYyUZbYtS8XwiBaqY6SR/UU9MhhswdoPU50WBq3h4Im+3jrmm7MbVD4uR6Uvt BnZIMzCJlQNcGpfbaGmtyrtbBNazQLTDOAxxQ7WM/MD8rip1zlP8MeL7yBgE3VEC JInw9DWN+fbQhbsTQgdiLLLhBXOFqZuXgz7auPnGzu+5LsIeJd6ZnWzuQ6TlzL3R rLgrMAasxTMzKoJB51jmDpCOeaj4b68XK+MY130/xSdENb2ZyGbuSF4LpZUjs4RU Sqiwtevu/bT/th0XBqcg79RHnwGLK/g1pJRB27RaOVbuh6f+lENc4Q== =4q9v -----END PGP SIGNATURE-----

1 0