- tor-dev - lists.torproject.org

Proposal 319: RELAY_FRAGMENT cells
by Nick Mathewson 19 May '20

19 May '20

``` Filename: 319-wide-everything.md Title: RELAY_FRAGMENT cells Author: Nick Mathewson Created: 11 May 2020 Status: Open ``` (This proposal is part of the Walking Onions spec project.) # Introduction Proposal 249 described a system for `CREATE` cells to become wider, in order to accommodate hybrid crypto. And in order to send those cell bodies across circuits, it described a way to split `CREATE` cells into multiple `EXTEND` cells. But there are other cell types that can need to be wider too. For example, `INTRODUCE` and `RENDEZVOUS` cells also contain key material used for a handshake: if handshakes need to grow larger, then so do these cells. This proposal describes an encoding for arbitrary "wide" relay cells, that can be used to send a wide variant of anything. To be clear, although this proposal describes a way that all relay cells can become "wide", I do not propose that wide cells should actually be _allowed_ for all relay cell types. # Proposal We add a new relay cell type: `RELAY_FRAGMENT`. This cell type contains part of another relay cell. A `RELAY_FRAGEMENT` cell can either introduce a new fragmented cell, or can continue one that is already in progress. The format of a RELAY_FRAGMENT body is one of the following: // First body in a series struct fragment_begin { // What relay_command is in use for the underlying cell? u8 relay_command; // What will the total length of the cell be once it is reassembled? u16 total_len; // Bytes for the cell body u8 body[]; } // all other cells. struct fragment_continued { // More bytes for the cell body. u8 body[]; } To send a fragmented cell, first a party sends a RELAY_FRAGMENT cell containing a "fragment_begin" payload. This payload describes the total length of the cell, the relay command Fragmented cells other than the last one in sequence MUST be sent full of as much data as possible. Parties SHOULD close a circuit if they receive a non-full fragmented cell that is not the last fragment in a sequence. Fragmented cells MUST NOT be interleaved with other relay cells on a circuit, other than cells used for flow control. (Currently, this is only SENDME cells.) If any party receives any cell on a circuit, other than a flow control cell or a RELAY_FRAGEMENT cell, before the fragmented cell is complete, than it SHOULD close the circuit. Parties MUST NOT send extra data in fragmented cells beyond the amount given in the first 'total_len' field. Not every relay command may be sent in a fragmented cell. In this proposal, we allow the following cell types to be fragmented: EXTEND2, EXTENDED2, INTRODUCE1, INTRODUCE2, RENDEZVOUS. Any party receiving a command that they believe should not be fragmented should close the circuit. Not all lengths up to 65535 are valid lengths for a fragmented cell. Any length under 499 bytes SHOULD cause the circuit to close, since that could fit into a non-fragmented RELAY cell. Parties SHOULD enforce maximum lengths for cell types that they understand. All `RELAY_FRAGMENT` cells for the fragmented cell must have the same Stream ID. (For those cells allowed above, the Stream ID is always zero.) Implementations SHOULD close a circuit if they receive fragments with mismatched Stream ID. # Onion service concerns. We allocate a new extension for use in the ESTABLISH_INTRO by onion services, to indicate that they can receive a wide INTRODUCE2 cell. This extension contains: struct wide_intro2_ok { u16 max_len; } We allocate a new extension for use in the `ESTABLISH_RENDEZVOUS` cell, to indicate acceptance of wide `RENDEZVOUS2` cells. This extension contains: struct wide_rend2_ok { u16 max_len; } (Note that `ESTABLISH_RENDEZVOUS` cells do not currently have a an extension mechanism. They should be extended to use the same extension format as `ESTABLISH_INTRO` cells, with extensions placed after the rendezvous cookie.) # Handling RELAY_EARLY The first fragment of each EXTEND cell should be tagged with `RELAY_EARLY`. The remaining fragments should not. Relays should accept `EXTEND` cells if and only if their _first_ fragment is tagged with `RELAY_EARLY`. > Rationale: We could allow any fragment to be tagged, but that would give > hostile guards an opportunity to move RELAY_EARLY tags around and build a > covert channel. But if we later move to a relay encryption method that > lets us authenticate RELAY_EARLY, we could then require only that _any_ > fragment has RELAY_EARLY set. # Compatibility This proposal will require the allocation of a new 'Relay' protocol version, to indicate understanding of the RELAY_FRAGMENTED command.

4 3

Re: [tor-dev] Chutney code refactoring
by c 17 May '20

17 May '20

On Fri, 15 May 2020 11:20:54 +1000 teor <teor(a)riseup.net> wrote: > You could try deleting functions, and then running tor's > "make test-network-all". If that test still passes, then the code is probably > unused (in practice). > > Chutney's CI does a few more tests for different tor versions, but I'm pretty > sure they all use the same code. This is what I will end up doing. I planned on it, but humans tend to know the purpose of the code better, and computers just run what is given to them :) so I figured it was best to check here first, with people more experienced with the codebase. Thanks, Caitlin

1 0

Re: [tor-dev] Chutney code refactoring
by c 17 May '20

17 May '20

On Thu, 14 May 2020 19:58:45 -0400 Nick Mathewson <nickm(a)freehaven.net> wrote: > isInExpectedDirInfoDocs looks like it might once have done something useful. My impression is to remove it unless we anticipate needing it in the near future. > configure and restart and wait_for_bootstrap are all in use; they are > invoked by name, from the command line, at the end of runConfigFile, > where it says `return getattr(network, verb)()`. Ah, I see now, thank you. I think it would be wise to comment the functions as such to indicate that they are used on the command line. > I think the rest are likely to be unused. I did grep most of these and found no signs of use, so this is my impression as well. For now I will make a mental note of all this while I work on more pertinent improvements to the code, but I do believe it would be nice to deal with truly unused code sooner rather than later. Caitlin

1 0

Deprecating Tor Protocol Versions
by teor 15 May '20

15 May '20

Hi all, Nick and I were talking about how we remove legacy features in tor, and their corresponding subprotocol versions. Here is a list of the current subprotocol versions: https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n2049 Here's a recent protocol version proposal, which deals with recommending and requiring new features: https://gitweb.torproject.org/torspec.git/tree/proposals/303-protover-remov… But we don't have a similar proposal for removing support for older protocol versions from the tor codebase. For an example of what that proposal could look like, see our proposal for deprecating consensus methods: https://gitweb.torproject.org/torspec.git/tree/proposals/290-deprecate-cons… Here's the original conversation Nick and I had: https://github.com/torproject/tor/pull/1874#discussion_r423713579 But after reading our consensus methods deprecation proposal, I've changed my mind. I think that we should check for "protocol N, and any later version" by default. That's what we do for consensus methods, and it seems to work well. We can drop the earliest consensus methods, and recent tor versions just keep working. If we need an incompatible change, we can make it another protocol version, and recommend then require support for it. So here's an edited version of my notes on that ticket: There are a few instances of ">=" and "=" confusion in protocol versions. We should try to fix them all. It only matters when we remove protocol versions. We haven't really specified, tested, or exercised this functionality in practice. And so our reviewers lack experience. (And when we did discover a need for it with NSS and LinkAuth, it was more complicated than we expected.) I'd like to see a proposal that tells us how to check future protocol versions as they are added. Along with a migration plan for disabling protocol versions. So let's also open a ticket to check for "any future version". We should replace all "=" checks with ">=". Let's make sure we check all the places where we use protocol versions, even if they don't have a summary flag. Overall, I think it would be helpful if future protocol versions were orthogonal. Or if they depend on earlier features, that dependency should be clearly documented. (For example, Relay=3 IPv6 extends depends on Relay=2 EXTEND2 cells. So if we were checking EXTEND2 cell support, it would be Relay=2 or Relay=3.) T -- teor ----------------------------------------------------------------------

2 2

Re: [tor-dev] Chutney code refactoring
by c 15 May '20

15 May '20

On Fri, 17 Apr 2020 18:01:42 -0400 Nick Mathewson <nickm(a)freehaven.net> wrote: > If you want to work on this it would be helpful to maybe start by > listing (here or elsewhere) some places where you *don't* feel like > you could (or would want to) write documentation: those would be a > good target for devs who _have_ worked on Chutney before. I came across Node.specialize() which does not seem to be called elsewhere, and I cannot guess at its purpose. So, if anyone knows off the top of their head what this is for (not what it does, because it's quite obvious from the code itself) then I'd greatly appreciate and I will document its purpose accordingly. It is one of the functions that is listed as "to be called by the user". Other than that I added docstrings where they were missing in TorNet (useful both for Python's on-line help() and for checking whether code agrees with intent) and made mental note of some areas I would like to improve in the code (changing function names to better align with what they actually do, figure out if some longer functions can be split into more atomical and easier-to-digest parts, ...). Caitlin

3 3

woswos - GSoC 2020
by Barkin Simsek 12 May '20

12 May '20

Hi everyone, I'm Barkin (woswos in IRC), and I study Computer Engineering at New York University Abu Dhabi. This summer, I will be working on the "Cloudflare CAPTCHA Monitoring" project as a part of the GSoC program with my mentors Georg and Roger. I’m very excited about working on this project! The Cloudflare CAPTCHA Monitoring project aims to track how often Cloudflare fronted webpages return CAPTCHAs to Tor clients. The project aims to achieve this by fetching webpages via both Tor and other mainstream web browsers and comparing the results. The tests are repeated periodically to find the patterns over time. Collected metadata, metrics, and results are analyzed and displayed on a dashboard to understand how Cloudflare manipulates internet traffic and affects people's access to the internet. I will be creating a bunch of trac tickets to record my progress and to get feedback from the community. Until then, you can leave your comments and suggestions under the ticket #33010 You can find more detailed information about the project on this wiki page: https://trac.torproject.org/projects/tor/wiki/doc/CAPTCHAMonitor This wiki page will also contain the links to the trac tickets I mentioned. I try to place all sorts of information on the wiki page to make it the "go-to place" to get information about my project. Best, Barkin

1 0

Proposal 318: Limit protover values to 0-63
by Nick Mathewson 12 May '20

12 May '20

``` Filename: 318-limit-protovers.md Title: Limit protover values to 0-63. Author: Nick Mathewson Created: 11 May 2020 Status: Open ``` # Limit protover values to 0-63. I propose that we no longer accept protover values higher than 63, so that they can all fit nicely into 64-bit fields. (This proposal is part of the Walking Onions spec project.) ## Motivation Doing this will simplify our implementations and our protocols. Right now, an efficient protover implementation needs to use ranges to represent possible protocol versions, and needs workarounds to prevent an attacker from constructing a protover line that would consume too much memory. With Walking Onions, we need lists of protocol versions to be represented in an extremely compact format, which also would benefit from a limited set of possible versions. I believe that we will lose nothing by making this change. Currently, after nearly two decades of Tor development and 3.5 years of experiences with protovers in production, we have no protocol version high than 5. Even if we did someday need to implement higher protocol versions, we could simply add a new subprotocol name instead. For example, instead of "HSIntro=64", we could say "HSIntro2=1". ## Migration Immediately, authorities should begin rejecting relays with protocol versions above 63. (There are no such relays in the consensus right now.) Once this change is deployed to a majority of authorities, we can remove support in other Tor environments for protocol versions above 63.

2 1

Walking onions: week 10 update
by Nick Mathewson 12 May '20

12 May '20

Walking Onions: week 10 update On our current grant from the zcash foundation, I'm working on a full specification for the Walking Onions design. I'm going to try to send out these updates once a week. My previous updates are linked below: Week 1: formats, preliminaries, git repositories, binary diffs, metaformat decisions, and Merkle Tree trickery. https://lists.torproject.org/pipermail/tor-dev/2020-March/014178.html Week 2: Specifying details of SNIP and ENDIVE formats, in CDDL. https://lists.torproject.org/pipermail/tor-dev/2020-March/014181.html Week 3: Expanding ENDIVEs into SNIPs. https://lists.torproject.org/pipermail/tor-dev/2020-March/014194.html Week 4: Voting (part 1) and extending circuits. https://lists.torproject.org/pipermail/tor-dev/2020-March/014207.html Week 5: Voting (part 2) https://lists.torproject.org/pipermail/tor-dev/2020-April/014218.html Week 6: Editing, voting rules, and client behavior (part 1) https://lists.torproject.org/pipermail/tor-dev/2020-April/014222.html Week 7: Exit policies: How do they work? https://lists.torproject.org/pipermail/tor-dev/2020-April/014232.html Week 8: Onion services, relay honesty, migration and families https://lists.torproject.org/pipermail/tor-dev/2020-April/014255.html Week 9: (There was no week 9) == Now with an introduction I left the Introduction section of the proposal till late in the game; now I have it in place as a new section 1: == Proposals into the pipeline I've also revised all of the "other proposals" that were logically separate from the rest of Walking Onions. Three of them were ready to go to tor-dev, so I've added them as: 318-limit-protovers.md Limit protover values to 0-63. 319-wide-everything.md RELAY_FRAGMENT cells 320-tap-out-again.md Removing TAP usage from v2 onion services == In the home stretch I've got till the end of the month to wrap up this proposal, so let's see where we are. We have a minimal viable proposal, I believe: if we were to sit down and start implementing, we would probably get most of the way based on what we have now. There are a few additional things I'd like to wrap up before I call the proposal done, however: * I've got all the necessary machinery written up for generating and calculating indices, but before I actually move ahead, I should fill in the details in sections 2 through 4. * One of the original goals of Walking Onions was to permit us to move to alternative topologies where not every relay connects to every other relay. That's not something that we can include in this proposal, since the research here is quite preliminary, but I still want to prepare for it by making sure that clients can handle getting SNIPs that say "this SNIP is only valid when given by a relay of type X". * I should take another pass through the CDDL format to see whether there are any opportunities to save a bunch of bytes. I should also fill in a couple of example SNIPs to see how big they are with this proposal. * There are places all over the proposal where I have marked uncertain points with XXXX. I should go through them and resolve them all before I call the proposal done. [INTRO] https://github.com/nmathewson/walking-onions-wip/blob/master/specs/01-intro…

1 0

Proposal 315: Updating the list of fields required in directory documents
by Nick Mathewson 11 May '20

11 May '20

Filename: 315-update-dir-required-fields.txt Title: Updating the list of fields required in directory documents Author: Nick Mathewson Created: 23 April 2020 Status: Open 1. Introduction When we add a new field to a directory document, we must at first describe it as "optional", since older Tor implementations will not generate it. When those implementations are obsolete and unsupported, however, we can safely describe those fields as "required", since they are always included in practice. Making fields required is not just a matter of bookkeeping: it helps prevent bugs in two ways. First, it simplifies our code. Second, it makes our code's requirements match our assumptions about the network. Here I'll describe a general policy for making fields required when LTS versions become unsupported, and include a list of fields that should become required today. This document does not require to us to make all optional fields required -- only those which we intend that all Tor instances should always generate and expect. When we speak of making a field "required", we are talking about describing it as "required" in dir-spec.txt, so that any document missing that field is no longer considered well-formed. 2. When fields should become required We have three relevant kinds of directory documents: those generated by relays, those generated by authorities, and those generated by onion services. Relays generate extrainfo documents and routerdesc documents. For these, we can safely make a field required when it is always generated by all relay versions that the authorities allow to join the network. To avoid partitioning, authorities should start requiring the field before any relays or clients do. (If a relay field indicates the presence of a now-required feature, then instead of making the field mandatory, we may change the semantics so that the field is assumed to be present. Later we can remove the option.) Authorities generate authority certificates, votes, consensus documents, and microdescriptors. For these, we can safely make a field required once all authorities are generating it, and we are confident that we do not plan to downgrade those authorities. Onion services generate service descriptors. Because of the risk of partitioning attacks, we should not make features in service descriptors required without a phased process, described in the following section. 2.1. Phased addition of onion service descriptor changes Phase one: we add client and service support for the new field, but have this support disabled by default. By default, services should not generate the new field, and clients should not parse it when it is present. This behavior is controlled by a pair of network parameters. (If the feature is at all complex, the network parameters should describe a _minimum version_ that should enable the feature, so that we can later enable it only in the versions where the feature is not buggy.) During this phase, we can manually override the defaults on particular clients and services to test the new field. Phase two: authorities use the network parameters to enable the client support and the service support. They should only do this once enough clients and services have upgraded to a version that supports the feature. Phase three: once all versions that support the feature are obsolete and unsupported, the feature may be marked as required in the specifications, and the network parameters ignored. Phase four: once all versions that used the network parameters are obsolete and unsupported, authorities may stop including those parameters in their votes. 3. Directory fields that should become required. These fields in router descriptors should become required: * identity-ed25519 * master-key-ed25519 * onion-key-crosscert * ntor-onion-key * ntor-onion-key-crosscert * router-sig-ed25519 * proto These fields in router descriptors should become "assumed present": * hidden-service-dir These fields in extra-info documents should become required: * identity-ed25519 * router-sig-ed25519 The following fields in microdescriptors should become required: * ntor-onion-key The following fields in votes and consensus documents should become required: * pr

3 2

Is anybody using OnionPerf's measurement results in the .tpf format?
by Karsten Loesing 10 May '20

10 May '20

Hello list, if the subject line doesn't make any sense to you, this email is very likely irrelevant for you. But if it does make sense to you and you did use OnionPerf's [0] measurement in the .tpf format [1] in the past and/or plan to use this data in the future, you should comment on ticket #34141 [2] and tell us about your use case, ideally in the next couple of days. For what it's worth, the same measurements data is also available in OnionPerf's JSON format [3], and existing data from past measurements will continue to be available in the .tpf format. It's just that we want to get rid of the .tpf format for new measurements. All the best, Karsten [0] https://metrics.torproject.org/torperf.html [1] https://metrics.torproject.org/collector.html#type-torperf [2] https://trac.torproject.org/projects/tor/ticket/34141 [3] https://metrics.torproject.org/collector.html#type-onionperf

1 0