October 2023 - tor-dev - lists.torproject.org

When RFC 7686 and transparent proxies collide
by Matt Jolly 28 Sep '24

28 Sep '24

Hello Everyone, I recently inadvertently opened a much larger can of worms than I'd intended when fixing a bug reported downstream where cURL would, when configured with certain DNS backends, fail to resolve .onion addresses. https://bugs.gentoo.org/887287 After doing some digging I discovered that the c-ares library was updated in 2018 to intentionally fail to resolve .onion addresses in line with RFC 7686, and another reported 'Bug' in cURL for leaking .onion DNS requests: https://github.com/c-ares/c-ares/issues/196 https://github.com/curl/curl/issues/543 I took the obviously sane and uncontroversial approach of making sure that cURL would always behave the same way regardless of the DNS backend in use, and that it would output a useful error message when it failed to resolve a .onion address. Unfortunately, this has made a lot of people very angry and been ~~widely regarded as a bad move~~panned by a small subset of downstream cURL users: https://github.com/curl/curl/discussions/11125 https://infosec.exchange/@harrysintonen/110977751446379372 https://gitlab.torproject.org/tpo/core/torspec/-/issues/202 I accept that, in particular, transproxy users are being inconvenienced, but I also don't want to go back to 'cURL leaks .onion DNS requests _sometimes_'. As a career sysadmin and downstream bug triager: this is the stuff that keeps me up late at night. Quite literally, far too often. I have found, however that the downstreams that I expected to be inconvenienced most (Whonix and Tails) simply use socks: https://github.com/Kicksecure/sdwdate/commit/5724d83b258a469b7a9a7bbc651539… https://github.com/Kicksecure/tb-updater/commit/d040c12085a527f4d39cb1751f2… https://github.com/Kicksecure/usability-misc/blob/8f722bbbc7b7f2f3a35619a5a… https://gitlab.tails.boum.org/tails/tails/-/issues/19488 https://gitlab.tails.boum.org/tails/tails/-/merge_requests/1123 I've asked in the Tor Specifications issue (inspired by Silvio's suggestions), and in the cURL issue, but I seem to be getting nowhere and the impacted users are clamouring for a quick band-aid solution, which I feel will work out worse for everyone in the long run: >How can client applications (safely): > >1.discover that they're in a Tor-enabled environment >2.resolve onion services only via Tor in that circumstance >3.not leak .onion resolution attempts at all > >Right now, not making these requests in the first place is the >safest (and correct) thing to do, however inconvenient it may be. >Rather than immediately trying to come up with a band-aid approach >to this problem, a sane mechanism needs to be implemented to: > >1.prevent each application from coming up with their own solution >2.prevent inconsistency in .onion resolution (i.e. no "oh it only >leaks if DO_ONION_RESOLUTION is set") >3.provide a standardised mechanism for applications that want to be Tor >aware to discover that they're in a Tor-enabled environment. I'm not particularly attached to that last point, but it's worth discussing. On a related note: -is the use of a transparent proxy recommended? -is there a sane alternative that involves as minimal configuration as possible for these users? I'm not sure what the best way forward is here, but I'm hoping that actual Tor developers might have a useful opinion on the matter, or at least be able to point me in the right direction. Thanks for your time, Cheers, Matt

7 9

Fact-checking a claim about relay/bridge fingerprint authentication
by David Fifield 25 Feb '24

25 Feb '24

Cecylia, Arlo, Serene, Shelikhoo, and I are writing a research paper about Snowflake. Here is a draft: https://www.bamsoftware.com/papers/snowflake/snowflake.20231003.e6e1c30d.pdf We're writing to check a factual claim in the section about having multiple backend bridges. Basically, we wanted it to be possible for there to be multiple Snowflake bridge sites run by different groups of people, and we did not want to share the same relay identity keys across all bridge sites, because of the increased risk of the keys being exposed. Therefore every bridge site has its own relay identity, which requires the client to know the relay fingerprints in advance and that it be the client (and not, e.g., the broker) that decides which bridge to use. 1. Is our general description (quoted below) of the design constraints as they bear on Tor correct? 2. Is §4.2 "CERTS cells" the right part of tor-spec to cite to make our point? https://gitlab.torproject.org/tpo/core/torspec/-/blob/b345ca044131b2eb18e6a… https://github.com/turfed/snowflake-paper/blob/e6e1c30dde6716dc5e54a32f2134… A Tor bridge is identified by a long-term identity public key. If, on connecting to a bridge, the client finds that the bridge's identity is not the expected one, the client will terminate the connection \cite[\S 4.2]{tor-spec}. The Tor client can configure at most one identity per bridge; there is no way to indicate (with a certificate, for example) that multiple identities should be considered equivalent. This constraint leaves two options: either all Snowflake bridges must share the same cryptographic identity, or else it must be the client that makes the choice of what bridge to use. While the former option is possible to do (by synchronizing identity keys across servers), every added bridge would increase the risk of compromising the all-important identity keys. Our vision was that different bridge sites would run in different locations with their own management teams, and that any compromise of a bridge site should affect that site only. In my own experiments, providing an incorrect relay fingerprint leads to errors in connection_or_client_learned_peer_id: https://gitlab.torproject.org/tpo/core/tor/-/blob/tor-0.4.7.13/src/core/or/… [warn] Tried connecting to router at 192.0.2.3:80 ID=<none> RSA_ID=2B280B23E1107BB62ABFC40DDCC8824814F80A71, but RSA + ed25519 identity keys were not as expected: wanted 1111111111111111111111111111111111111111 + no ed25519 key but got 2B280B23E1107BB62ABFC40DDCC8824814F80A72 + 1zOHpg+FxqQfi/6jDLtCpHHqBTH8gjYmCKXkus1D5Ko. [warn] Problem bootstrapping. Stuck at 14% (handshake): Handshaking with a relay. (Unexpected identity in router certificate; IDENTITY; count 1; recommendation warn; host 1111111111111111111111111111111111111111 at 192.0.2.3:80)

2 3

Proposal 347: Domain separation for certificate signing keys
by Nick Mathewson 19 Oct '23

19 Oct '23

To see this rendered, go to https://spec.torproject.org/proposals/347-domain-separation.html ``` Filename: 347-domain-separation.md Title: Domain separation for certificate signing keys Author: Nick Mathewson Created: 19 Oct 2023 Status: Open ``` ## Our goal We'd like to be able to use the "family key" from proposal 321 as a general purpose signing key, to authenticate other things than the membership of a family. For example, we might want to have a challenge/response mechanism where the challenger says, "If you want to log in as the owner of the account corresponding to this family, sign the following challenge with your key. Or we might want to have a message authentication scheme where an operator can sign a message in a way that proves key ownership. We _might_ also like to use relay identity keys or onion service identitiy keys for the same purpose. ## The problem When we're using a signing key for two purposes, it's important to perform some kind of domain separation so that documents signed for one purpose can't be mistaken for documents signed for the other. For example, in the challenge/response example, it would be bad if the challenger could provide a challenge string that would cause the signer to inadvertently authenticate an incorrect family. These keys are currently used in some places with no personalization. Their signature format is as described in `cert-spec.txt`, which says: > The signature is created by signing all the fields in > the certificate up until "SIGNATURE" (that is, signing > sizeof(ed25519_cert) - 64 bytes). ## One solution This one is pretty easy: we would extend `cert-spec` as follows. > ### Using signing keys for other purposes. > > Other protocols may introduce other uses for the signing keys > in these certificates other than those specified here. When > they do, they MUST ensure that the documents being signed > cannot be confused with the certificate bodies of this document. > > In some existing cases in the Tor protocols, we achieve this by > specifying an ASCII prefix string that must be prepended to the > other protocol's signed object before it is signed. > > For future protocols, we recommend that this be done by > specifying that the signing key is to be used to sign a > cSHAKE digest (or other secure customizable digest) > of the other protocol's signed object, using > a customization string unique to the other protocol. We would also make this amendment: > ### Future versions of this specification > > In order to maintain the domain separation that currently exists > between the signatures on these certificates and other documents > signed with the same keys, it suffices (for now!) that these > certificates begin with the version byte `[01]`, whereas > the other documents are in printable ASCII, which never > includes `[01]`. > > Future versions of this specification SHOULD move to using > an ed25519-prehash construction, using a customizable hash > with built-in domain separation.

2 1

Proposal 346: Clarifying and extending the use of protocol versioning
by Nick Mathewson 19 Oct '23

19 Oct '23

Note: we've updated our specs website! You can read a rendered version of this proposal at https://spec.torproject.org/proposals/346-protovers-again.html . ``` Filename: 346-protovers-again.md Title: Clarifying and extending the use of protocol versioning Author: Nick Mathewson Created: 19 Oct 2023 Status: Open ``` # Introduction In proposal 264, we introduced "subprotocol versions" as a way to independently version different pieces of the Tor protocols, and communicate which parts of the Tor protocols are supported, recommended, and required. Here we clarify the semantics of individual subprotocol versions, and describe more ways to use and negotiate them. # Semantics: Protocol versions are feature flags One issue we left unclarified previously is the relationship between two different versions of the same subprotocol. That is, if we know the semantics of (say) `Handshake=7`, can we infer anything about a relay that supports `Handshake=8`? In particular, can we infer that it supports all of the same features implied by `Handshake=7`? If we want to know "does this relay support some feature supported by `Handshake=7`", must we check whether it supports `Handshake=7`, or should we check `Handshake=x for any x≥7`? In this proposal, we settle the question as follows: subprotocol versions are flags. They do not have any necessary semantic relationship between them. We reject the `≥` interpretation for several reasons: * It's tricky to implement. * It prevents us from ever removing a feature. * It requires us to implement features in the same order across all Tor versions. ## ...but sometimes a flag _is_ a version! There _are_ some places in our protocol (notably: directory authority consensus methods, and channel protocol versions) where there _is_ a semantic relationship between version numbers. Specifically: "higher numbers are already better". When parties need to pick a one of _these_ versions, they always pick the highest version number supported by enough of them. When this kind of _real_ version intersects with the "subprotocol versions" system, we use the same numbers: * `Link` subprotocols correspond one-to-one with the version numbers sent in a VERSIONS cell. * `Microdesc` and `Cons` subprotocols correspond to a _subset_ of the version numbers of consensus methods. ## How to document subprotocol versions When describing a subprotocol, we should be clear what relationship, if any, exists between its versions and any versions negotiated elsewhere in the specifications. Unless otherwise documented, all versions can be in use _at the same time_: if only one can exist at once (on a single circuit, a single document, etc), this must be documented. > Implication: This means that we must say (for example) that you > can't use Link=4 and Link=5 on the same channel. # Negotiating protocol versions in circuit handshakes. Here we describe a way for a client to opt into features as part of its circuit handshake, in order to avoid proliferating negotiating extensions. ## Binary-encoding protocol versions. We assign a one-byte encoding for each protocol version number, ordered in the same way as in tor-spec. | Protocol | Id | | --- | --- | | Link | 0 | | LinkAuth | 1 | | Relay | 2 | | DirCache | 3 | | HSDir | 4 | | HSIntro | 5 | | HSRend | 6 | | Desc | 7 | | MicroDesc| 8 | | Cons | 9 | | Padding | 10 | | FlowCtrl | 11 | | Conflux | 12 | | Datagram | 13 | > Note: This is the same encoding used in the [walking onions > proposal][prop323]. It takes its order from the ordering of > protocol versions in > [tor-spec][subprotocol-versioning] > and matches up with the values defined in for `protocol_type_t` in C > tor's `protover.h`. ## Requesting an opt-in circuit feature When a client wants to request a given set of features, it sends an `ntor_v3` extension containing: ``` struct subproto_request { struct req[..]; // up to end of extension } struct req { u8 protocol_id; u8 protovol_version; } ``` > Note 1: The above format does not include any parameters for each > req. Thus, if we're negotiating an extension that requires a client- > supplied parameter, it may not be appropriate to use this > request format. > > Note 2: This proposal does not include any relay extension > acknowledging support. In the case of individual subprotocols, we could > later say "If this subprotocol is in use, the relay MUST also send > extension foo". > > Note 3: The existence of this extension does not preclude the later > addition of other extensions to negotiate featuress differently, or > to do anything else. Each `req` entry corresponds to a single subprotocol version. A client MUST NOT send any `req` entry unless: * That subprotocol version is advertised by the relay, * OR that subprotocol version is listed as required for relays in the current consensus, using `required-relay-protocols`. > Note: We say above that a client may request a _required_ subprotocol > even if the relay does not advertise it. This is what allows > clients to send a `req` extension to introduction points and > rendezvous points, even when we do not recognize the relay from the > consensus. > > Note 2: If the intro/rend point does not support a _required_ protocol, > it should not be on the network, and the client/service should not have > selected it. If a relay receives a `subproto_request` extension for any subprotocol version that it does not support, it MUST reject the circuit with a DESTROY cell. > Alternatives: we _could_ give the relay the option to > decline to support an extension, and we _could_ require the > relay to acknowledge which extensions it is providing. > We aren't doing that, in the name of simplicity. Only certain subprotocol versions need to be negotiated in this way; they will be explicitly listed as such in our specifications, with language like "This extension is negotiated as part of the circuit extension handshake". Other subprotocol versions MUST NOT be listed in this extension; if they are, the relay SHOULD reject the circuit. > Alternative: We _could_ allow the client to list other subprotocols > that the relay supports which are nonetheless irrelevant to > the circuit protocol, like `Microdesc`, or ones that don't currently need > to be negotiated, like `HsRend`. > > This is not something we plan to do. Currently specified subprotocol versions which can be negotiated using this extension are: * FlowCtrl=2 (congestion control) * Packed-and-fragmented support (proposal 340) The availability of the `subproto_request` extension itself will be indicated by a new Relay=X flag. When used, it will supplant several older `ntor_v3` extensions, including: * (TODO: list these here, if we find any. I think FlowCtrl has an extension?) That is, if using `subproto_request`, there is no need to send the (TODO) extensions. ## Making features that can be disabled. Sometimes, we will want the ability to make features that can be enabled or disabled from the consensus. But if we were to make a single flag that can turn the feature on and off, we'd run into trouble: after the feature was turned off, every relay would stop providing it right away, but there would be a delay before clients realized that the relays had stopped advertising the feature. During this interval, clients would try to enable the feature, and the relays would reject their circuits. To solve this problem, we need to make features like these controlled by a _pair_ of consensus parameters: one to disable advertising the feature, and one to disable the feature itself. To disable a feature, first the authorities would tell relays to stop advertising it, and only later tell the relays to stop supporting it. (If we want to _enable_ a previously disabled feature, we can turn on advertisement and support at the same time.) These parameters would be specified something like this (for a hypthetical `Relay=33` feature). * `support-relay-33`: if set to 1, relays that can provide `Relay=33` should do so. * `advertise-relay-33`: if set to 1, relays that are providing `Relay=33` should include it in their advertised protocol versions. Note: as a safety measure, relays MUST NOT advertise any feature that they do not support. This is reflected in the descriptions of the parameters above. When we add a new feature of this kind, we should have the `advertise-*` flag parameter be `1` by default, and probably we should have `support-*` be `1` by default oo. # Subprotocol versions in onion services Here we describe how to expand the onion service protocols in order to better accomodate subprotocol versions. ## Advertising an onion service's subprotocols In its encrypted descriptor (the innermost layer), the onion service adds a new entry: * `"protocols"` - A list of supported subprotocol versions, in the same format as those listed in a microdescriptor or descriptor. Note that this is NOT a complete list of all the subprotocol versions actually supported by the onion service. Instead, onion services only advertise a subprotocol version if they support it, _and_ it is documented in the specs as being supported by onion services. > Alternative: I had considered having a mask that would be put in the > consensus document, telling the onion services which subprotocols > to advertise. I don't think that's a great idea, however. Right now, the following protocols should be advertised: - FlowCtrl - Conflux (?? Doesn't this take parameters? TODO) - Pow (??? Doesn't this take parameters? If we do it, we need to allocate a subprotocol for it. TODO) ## Negotiating subprotocols with an onion service. In the `hs_ntor` handshake sent by the client, we add an encrypted `subproto_request` extension of the same format, with the same semantics, as used in the `ntor-v3` handshake. This supplants the following: - (Congestion Control; Pow? TODO) ## Advertising other relays' subprotocols? > Alternative: I had previously considered a design where the introduction points > in the onion service descriptor would be listed along with their > subprotocols, and the `hs_ntor` handshake would contain the > subprotocols of the rendezvous point. > > I'm rejecting this design for now because it gives the onion service > and the client too much freedom to lie about relays. In the future, > the walking onions design would solve this, since the contact > information for intro and rend points would be authenticated. # Appendix New numbers to reserve: * An extension ID for the `ntor_v3` handshake `subproto_request` extension. * An extension ID for the `hs_ntor` handshake `subproto_request` extension. * A Relay= subprotocol indicating support for the ntor-v3 and hs_ntor extensions. * The numeric encoding of each existing subprotocol, in the table above. [prop323]: https://spec.torproject.org/proposals/323-walking-onions-full.html [subprotocol-versioning]: https://spec.torproject.org/tor-spec/subprotocol-versioning.html

1 0

PT bridge reporting significant numbers of <OR> IP addresses after upgrade to 0.4.8.6; Conflux related?
by David Fifield 10 Oct '23

10 Oct '23

The Snowflake bridges do not expose their plain ORPort (they have `ORPort 127.0.0.1:auto` in torrc), and consequently they have always reported ≈0 <OR> IP addresses in the bridge-ip-transports line of bridge-extra-info descriptors, with virtually all connecting IP addresses being instead attributed to the snowflake transport. (The <OR> count is low but nonzero, which I have rationalized as tor making some small number of internal connections, or something.) I upgraded one of the two Snowflake bridges from 0.4.7.13 to 0.4.8.6 on 2023-09-24. Since then, the number of <OR> IP addresses has been roughly equal to the number of snowflake IP addresses. The ORPort is still not exposed; these are not external vanilla bridge users. Did something change between these versions that might cause PT connections to be double-counted, once for the transport and once for <OR>? Here are excerpted bridge-extra-info descriptors from before and after the version upgrade. Note the bridge-ip-transports lines. ``` @type bridge-extra-info 1.3 extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0 master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks published 2023-09-19 10:46:08 transport snowflake bridge-ip-versions v4=13528,v6=1384 bridge-ip-transports <OR>=8,snowflake=14912 ``` ``` @type bridge-extra-info 1.3 extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0 master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks published 2023-09-29 17:33:20 transport snowflake bridge-ip-versions v4=2880,v6=336 bridge-ip-transports <OR>=1632,snowflake=1592 ``` Below is a time series for a whole month. The thing to notice is not the overall decrease in users on 2023-09-20; that's due to https://forum.torproject.org/t/problems-with-snowflake-since-2023-09-20-bro…. Rather, the thing to notice is that before 2023-09-24, <OR> ≈ 0; and after, <OR> ≈ snowflake. date <OR> snowflake 2023-08-30 4 45240 ← running 0.4.7.13 here 2023-08-31 21 193412 2023-09-01 30 195647 2023-09-02 40 200405 2023-09-03 40 201730 2023-09-04 42 195097 2023-09-05 35 190842 2023-09-06 30 189694 2023-09-07 26 191524 2023-09-08 35 195833 2023-09-09 32 196244 2023-09-10 27 146972 2023-09-11 6 55491 2023-09-12 18 182519 2023-09-13 17 173536 2023-09-14 21 161396 2023-09-15 21 166379 2023-09-16 16 174350 2023-09-17 17 165063 2023-09-18 23 164734 2023-09-19 32 164075 2023-09-20 23 121623 2023-09-21 2 25268 2023-09-22 0 4314 2023-09-23 0 746 2023-09-24 239 606 ← upgrade to 0.4.8.6 on this day 2023-09-25 1919 1900 2023-09-26 3533 3470 2023-09-27 11919 11687 2023-09-28 20242 19890 2023-09-29 26583 26100 2023-09-30 25929 25400

2 2

Proposal 345: Migrating the tor specifications to mdbook
by Nick Mathewson 03 Oct '23

03 Oct '23

``` Filename: 345-specs-in-mdbook.md Title: Migrating the tor specifications to mdbook Author: Nick Mathewson Created: 2023-10-03 Status: Open ``` # Introduction I'm going to propose that we migrate our specifications to a set of markdown files, specifically using the [`mdbook`] tool. This proposal does _not_ propose a bulk rewrite of our specs; it is meant to be a low-cost step forward that will produce better output, and make it easier to continue working on our specs going forward. That said, I think that this change will enable rewrites in the future. I'll explain more below. # What is mdbook? Mdbook is a tool developed by members of the Rust community to create books with [Markdown]. Each chapter is a single markdown file; the files are organized into a book using a [`SUMMARY.md`] file. Have a look at the [`mdbook` documentation][`mdbook`]; this is what the output looks like. Have a look at this [source tree]: that's the input that produces the output above. Markdown is extensible: it can use numerous [plugins] to enhance the semantics of the the markdown input, add diagrams, output in more formats, and so on. # What would using mdbook get us _immediately_? There are a bunch of changes that we could get immediately via even the simplest migration to mdbook. These immediate benefits aren't colossal, but they are things we've wanted for quite a while. * We'll have a document that's easier to navigate (via the sidebars). * We'll finally have good HTML output. * We'll have all our specifications organized into a single "document", able to link to one another and cross reference one another. * We'll have perma-links to sections. * We'll have a built-in text search function. (Go to the [`mdbook`] documentation and hit "`s`" to try it out.) ## How will mdbook help us _later on_ as we reorganize? Many of the benefits of mdbook will come later down the line as we improve our documentation. * Reorganizing will become much easier. * Our links will no longer be based on section number, so we won't have to worry about renumbering when we add new sections. * We'll be able to create redirects from old section filenames to new ones if we need to rename a file completely. * It will be far easier to break up our files into smaller files when we find that we need to reorganize material. * We will be able make our documents even _easier_ to navigate. * As we improve our documentation, we'll be able to use links to cross-reference our sections. * We'll be able to include real diagrams and tables. * We're already using the [`mermaid`] tool in Arti in generate [protocol diagrams] and [architecture diagrams]; we can use this in our specs too, instead of hand-drawn ASCII art. * We'll be able to integrate proposals more easily. * New proposals can become new chapters in our specification simply by copying them into a new 'md' file or files; we won't have to decide between integrating them into existing files or creating a new spec. * Implemented but unmerged proposals can become additional chapters in an appendix to the spec. We can refer to them with permalinks that will still work when they move to another place in the specs. # How should we do this? ## Strategy My priorities here are: * no loss of information, * decent-looking output, * a quick automated conversion process that won't lose a bunch of time. * a process that we can run experimentally until we are satisfied with the results With that in mind, I'm writing a simple set of [`torspec-converter`] scripts to convert our old torspec.git repository into its new format. We can tweak the scripts until we like the that they produce. After running a recent `torspec-converter` on a fairly recent torspec.git, here is how the branch looks: https://gitlab.torproject.org/nickm/torspec/-/tree/spec_conversion?ref_type… And here's the example output when running mdbook on that branch: https://people.torproject.org/~nickm/volatile/mdbook-specs/index.html > Note: these is not a permanent URL; we won't keep the example output > forever. When we actually merge the changes, they will move into > whatever final location we provide. The conversion script isn't perfect. It only recognizes three kinds of content: headings, text, and "other". Content marked "other" is marked with \`\`\` to reneder it verbatim. The choice of which sections to split up and which to keep as a single page is up to us; I made some initial decisions in the file above, but we can change it around as we please. See the [configuration section] at the end of the `grinder.py` script for details on how it's set up. ## Additional work that will be needed Assuming that we make this change, we'll want to build an automated CI process to build it as a website, and update the website whenever there is a commit to the specifications. (This automated CI process might be as simple as `git clone && mdbook build && rsync -avz book/ $TARGET`.) We'll want to go through our other documentation and update links, especially the permalinks in spec.torproject.org. It might be a good idea to use spec.torproject.org as the new location of this book, assuming weasel (who maintains spec.tpo) also thinks it's reasonable. If we do that, we need to decide on what we want the landing page to look like, and we need _very much_ to get our permalink story correct. Right now I'm generating a .htaccess file as part of the conversion. ## Stuff we shouldn't do. I think we should continue to use the existing torspec.git repository for the new material, and just move the old text specs into a new archival location in torspec. (We *could* make a new repository entirely, but I don't think that's the best idea. In either case, we shouldn't change the text specifications after the initial conversion.) We'll want to figure out our practices for keeping links working as we reorganize these documents. Mdbook has decent redirect support, but it's up to us to actually create the redicrets as necessary. # The transition, in detail * Before the transition: - Work on the script until it produces output we like. - Finalize this proposal and determine where we are hosting everything. - Develop the CI process as needed to keep the site up to date. - Get approval and comment from necessary stakeholders. - Write documentation as needed to support the new way of doing things. - Decide on the new layout we want for torspec.git. * Staging the transition: - Make a branch to try out the transition; explicitly allow force-pushing that branch. (Possibly nickm/torspec.git in a branch called mdbook-demo, or torspec.git in a branch called mdbook-demo assuming it is not protected.) - Make a temporary URL to target with the transition (possibly spec-demo.tpo) - Once we want to do the transition, shift the scripts to tpo/torspec.git:main and spec.tpo, possibly? * The transition: - Move existing specs to a new subdirectory in torspec.git. - Run the script to produce an mdbook instance in torspec.git with the right layout. - Install the CI process to keep the site up to date. * Post-transition - Update links elsewhere. - Continue to improve the specs. # Integrating proposals We could make all of our proposals into a separate book, like rust does at https://rust-lang.github.io/rfcs/ . We could also leave them as they are for now. (I don't currently think we should make all proposals part of the spec automatically.) # Timing I think the right time to do this, if we decide to move ahead, is before November. That way we have this issue as something people can work on during the docs hackathon. # Alternatives I've tried experimenting with Docusaurus here, which is even more full-featured and generates pretty react sites [like this](https://docusaurus.io/). (We're likely to use it for managing the Arti documentation and website.) For the purposes we have here, it seems slightly overkill, but I do think a migration is feasible down the road if we decide we _do_ want to move to docusaurus. The important thing is the ability to keep our URLs working, and I'm confident we could do that The main differences for our purposes here seem to be: * The markdown implementation in Docusaurus is extremely picky about stuff that looks like HTML but isn't; it rejects it, rather than passing it on as text. Thus, using it would require a more painstaking conversion process before we could include text like `"<state:on>"` or `"A <-> B"` as our specs do in a few places. * Instead of organizing our documents in a `SUMMARY.md` with an MD outline format, we'd have to organize them in a `sidebar.js` with a javascript syntax. * Docusaurus seems to be far more flexible and have a lot more features, but also seems trickier to configure. <-- References --> [`mdbook`]: https://rust-lang.github.io/mdBook/ [source tree]: https://github.com/rust-lang/mdBook/tree/master/guide/src/ [Markdown]: https://en.wikipedia.org/wiki/Markdown [`SUMMARY.md`]: https://rust-lang.github.io/mdBook/format/summary.html [plugins]: https://github.com/rust-lang/mdBook/wiki/Third-party-plugins [`mermaid`]: https://mermaid.js.org/ [architecture diagrams]: https://gitlab.torproject.org/tpo/core/arti/-/blob/main/doc/dev/Architectur… [protocol diagrams]: https://gitlab.torproject.org/tpo/core/arti/-/blob/main/doc/dev/hs-overview… [`torspec-converter`]: https://gitlab.torproject.org/nickm/torspec-converter [configuration section]: https://gitlab.torproject.org/nickm/torspec-converter/-/blob/main/grinder.p…

1 0

[Update] Network Status APIs
by Mattia Righetti 03 Oct '23

03 Oct '23

Greetings Tor devs, I wanted to give a quick update on the progress of the Network Status APIs[0]. First of all, a couple of weeks ago we have deployed an initial version of the service internally so that we can benchmark and test it and see how it holds. Plus, we now have developed three more APIs: `/details`, `/clients`, `/weights`. In total, we now have covered almost every endpoint that is currently provided by onionoo. Just to give you a little bit more insights, `/summary` and `/details` fetch data from the Network Health team database and return a response identical to the one that onionoo clients expect right now. On the other hand, `/clients`, `/weights` and `bandwidth` will proxy the request to VictoriaMetrics. In the upcoming weeks we plan to stabilize and test these endpoints further and do some perf improvements. Meanwhile, we are also working on documenting the APIs so that potential clients know how to use them, when they'll be available. Those can be found in the project's repo Wiki[1]. Please do reach out to me, hiro or GeKo if you have any ideas or feedback you'd like to share with us. Cheers, Matt [0]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/ [1]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/-…

1 0