- tor-dev - lists.torproject.org

Announcing Onionspray 1.6.0 with a SECURITY fix for Onion Services rewriting proxies
by rhatto 09 Feb '24

09 Feb '24

Hello everyone, I'd like to announce Onionspray, a tool for setting up Onion Services for existing public websites, working as a HTTPS rewriting proxy: https://tpo.pages.torproject.net/onion-services/onionspray/ It's a fork of Alec Muffett's EOTK (https://github.com/alecmuffett/eotk) with many enhancements but retaining compatibility, and relying on C Tor until an alternative in Arti is available. The first Onionspray version is 1.6.0, following the pre-existing version sequence from EOTK. Security fixes: * This release fixes a CRITICAL security vulnerability related to upstream HTTPS certificate verification, which is detailed at https://tpo.pages.torproject.net/onion-services/onionspray/security/advisor… A related fix is also available for EOTK: https://github.com/alecmuffett/eotk/pull/116 We urge Onionspray users that were testing the software while it was being on it's early stages to upgrade ASAP to 1.6.0 and update their configurations, and we recommend that EOTK to the same with the corresponding patch. This issue might also affect other similar rewriting proxy setups, and we urge operators to review and fix their Onion Service configurations. Main improvements over EOTK: * MetricsPort support (for gathering metrics data from the tor instances). * Denial of Service (DoS) protections. * Circuit ID exporting to NGINX logs and optionally to the upstream proxy (through the X-Onion-CircuitID HTTP header). * Onionbalance v3 support ("softmaps" are working again). * Revamped documentation. * Installation procedures added for recent Debian and Ubuntu releases. * Tor and OpenResty upgraded to the latest versions. * Option to keep Onionspray running in the foreground (`--no-daemonize`). * Local healthcheck action (`--health-local`), useful for containerized execution. The full ChangeLog is available at https://tpo.pages.torproject.net/onion-services/onionspray/changelog/ For those wishing to switch from EOTK to Onionspray, there's a migration guide at https://tpo.pages.torproject.net/onion-services/onionspray/migrating/ We also welcome people to report issues, send merge requests etc: https://tpo.pages.torproject.net/onion-services/onionspray/contact/ And we have a bunch of issues waiting for contributions: https://gitlab.torproject.org/tpo/onion-services/onionspray/-/issues Finally, I'd like to thank Alec Muffett for his important work with EOTK and for promoting Onion Services all these years :) Thanks! -- Silvio Rhatto pronouns he/him

1 0

docker bridge suddenly fails startup with "Directory /var/lib/tor cannot be read: Permission denied"
by gentle.luck4713＠doublequote.org 01 Feb '24

01 Feb '24

Hi tor-dev@, I have been running a bridge following the instructions at <https://community.torproject.org/relay/setup/bridge/docker/> on a Debian 12.4 system but when I just tried to make sure everything is up to date I got the following error message from docker-compose logs: obfs4-bridge_1 | Using NICKNAME=DockerObfs4Bridge, OR_PORT=3845, PT_PORT=443, and EMAIL=redacted(a)example.com. obfs4-bridge_1 | Additional properties from 'OBFS4V_' environment variables processing enabled obfs4-bridge_1 | Overriding 'AddressDisableIPv6' with value '1' obfs4-bridge_1 | Starting tor. obfs4-bridge_1 | Jan 31 22:49:20.501 [notice] Tor 0.4.8.10 running on Linux with Libevent 2.1.12-stable, OpenSSL 3.0.11, Zlib 1.2.13, Liblzma 5.4.1, Libzstd 1.5.4 and Glibc 2.36 as libc. obfs4-bridge_1 | Jan 31 22:49:20.501 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://support.torproject.org/faq/staying-anonymous/ obfs4-bridge_1 | Jan 31 22:49:20.501 [notice] Read configuration file "/etc/tor/torrc". obfs4-bridge_1 | Jan 31 22:49:20.502 [notice] Based on detected system memory, MaxMemInQueues is set to 6264 MB. You can override this by setting MaxMemInQueues by hand. obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Opening OR listener on 0.0.0.0:3845 obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Opened OR listener connection (ready) on 0.0.0.0:3845 obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Opening OR listener on [::]:3845 obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Opened OR listener connection (ready) on [::]:3845 obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Opening Extended OR listener on 127.0.0.1:0 obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Extended OR listener listening on port 44595. obfs4-bridge_1 | Jan 31 22:49:20.503 [notice] Opened Extended OR listener connection (ready) on 127.0.0.1:44595 obfs4-bridge_1 | Jan 31 22:49:20.504 [warn] Directory /var/lib/tor cannot be read: Permission denied obfs4-bridge_1 | Jan 31 22:49:20.504 [notice] Closing partially-constructed OR listener connection (ready) on 0.0.0.0:3845 obfs4-bridge_1 | Jan 31 22:49:20.504 [notice] Closing partially-constructed OR listener connection (ready) on [::]:3845 obfs4-bridge_1 | Jan 31 22:49:20.504 [notice] Closing partially-constructed Extended OR listener connection (ready) on 127.0.0.1:44595 obfs4-bridge_1 | Jan 31 22:49:20.504 [warn] Failed to parse/validate config: Couldn't create private data directory "/var/lib/tor" obfs4-bridge_1 | Jan 31 22:49:20.504 [err] Reading config failed--see warnings above. I confirmed that my docker-compose.yaml is unmodified and my .env is innocuous. I know that I can blow away the docker volumes but that'd remove the private keys etc. and that'd doesn't seem desirable. Any advice? Thanks, Hein

1 0

New Proposal - UDP Application Support in Tor
by Micah Elizabeth Scott 25 Jan '24

25 Jan '24

Hello tor-dev folks! Late last year I started taking a close look at what it would take to support applications on Tor which rely on UDP networking. This was originally to be based on Nick's proposal, 339-udp-over-tor. The scope of this work so far has been specifically focused on end-user application compatibility, and excludes fundamental changes to Tor's network structure or protocols for now. This combination of approach and scope left me with more questions than answers, so I started looking deeper into the available solutions along with their expected benefits and risks. This proposal is the result of that investigation. Please find the text attached, or in the torspec repo as proposal #348: https://gitlab.torproject.org/tpo/core/torspec/-/blob/main/proposals/348-ud… Unlike a typical proposal, this does not recommend any specific change to the Tor implementation. Several possible changes are presented, but ultimately the recommended approach is to use application-specific UDP relays to achieve compatibility. Integrated approaches are also presented, where Tor does involve itself in the transit of individual datagrams. These approaches offer advantages only when they are also part of a long-term plan to offer transport features beyond those offered by TCP. Until such a plan is in-scope, specific UDP extensions cannot be offered with confidence. I would appreciate any feedback on this proposal, whether it's about this particular shorter-term context or about longer-term plans to achieve some kind of optional unreliable transport. Thanks for your time! --beth

1 0

Timers in Arti?
by Michael Rogers 15 Jan '24

15 Jan '24

Hi, If I understand right, the C implementation of Tor has various state machines that are driven by local libevent timers as well as events from the network. For example, when building a circuit, if there hasn't been any progress for a certain amount of time then a timer fires to handle the timeout. When running Tor on Android, we need to prevent the OS from suspending so that these timers fire when they're supposed to. This uses a lot of battery, because normally the OS would spend most of its time suspended. Unlike a laptop, an Android device with its screen turned off is constantly dipping in an out of suspension, and a lot of the platform's power optimisations are aimed at batching whatever work needs to be done so that the periods of suspension can be longer. If we allowed the OS to suspend then the timers would fire with arbitrary delays, and I don't really know what impact that would have: I'd speculate that there might be connectivity problems (eg dead circuits not being detected and replaced) and/or network-visible changes in the behaviour of circuits that could affect anonymity. So I've had a longstanding but not-very-hopeful request that maybe Tor's reliance on timers could be reduced, or at least clarified, so that we could either allow the OS to suspend without breaking anything, or at least know what was likely to break. And basically I'd just like to renew that request for Arti. Could anyone give me an overview of how these local timers are handled in Arti, or any information about what's likely to happen if the timers are arbitrarily delayed? Thanks, Michael

4 8

txtorcon 23.11.0
by meejah 20 Nov '23

20 Nov '23

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 I'm energized to announce txtorcon 23.11.0 with the following changes: * Fix test-failures on Python 3.12 * Particular GETINFO hanging (https://github.com/meejah/txtorcon/issues/389) (ultra-long lines over 16KiB caused problems in the protocol) * Use built-in `mock` only (https://github.com/jelly) * Remove `incremental` (https://github.com/gdrosos) txtorcon is an implementation of the "control-spec" for Tor using the "Twisted" networking library. This is the library to use if you want to write event-based software (including asyncio; interop is possible) in Python that uses the Tor network as a client or a service (or to integrate Tor support into existing Twisted-using applications). You can download the release from PyPI or GitHub (or of course "pip install txtorcon"): https://pypi.python.org/pypi/txtorcon/23.11.0 https://github.com/meejah/txtorcon/releases/tag/v23.11.0 Releases are also available from the hidden service: http://fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion/txtor… http://fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion/txtor… You can verify the sha256sum of both by running the following 4 lines in a shell wherever you have the files downloaded: cat <<EOF | sha256sum --check c6c4f5ca5e94ed58be70de0f06d06153b7025cf4c6956b3724194a7cd8736904 dist/txtorcon-23.11.0.tar.gz 59ec801a38d6ea3cedc544cfcdb8e6f180b55d51bf9a01224b09f4bf48262f6b dist/txtorcon-23.11.0-py3-none-any.whl EOF thanks, meejah -----BEGIN PGP SIGNATURE----- iQFFBAEBCgAvFiEEnVor1WiOy4id680/wmAoAxKAaacFAmVbDTcRHG1lZWphaEBt ZWVqYWguY2EACgkQwmAoAxKAaafM/wgA4VdkdG6UgY7XiWQSfrjZZby5sszLJWUa 6T2x4PlHMnI6H2GvspU76n8rHwIfoi5MUIpZvqA17YnN2roPvBZYYwThczh9rhs1 2iTh4geeoBoLJyv8yokfy8gkv0AZ4tdfWa4b3AKA7yqoQ5YSK0hCji+m1jfqyY0q P1FCTDZ2cMl8nQ0/KMxtY1Ifg2tXpQi0OXRujPIHCdqQkA5u94WS1pZZBK3qSNJE 1/20vBI38938wDk/oNapHIeyXYVoVL4DAGq0ADMwYm0syoMZx7NES0a6T3UThKMt szmNj8YR+gwifu3/7UAEZbnAyosSgByqHPaFUu40+Z5LzlXaEnwiaA== =bHsG -----END PGP SIGNATURE-----

1 0

Proposal 347: Domain separation for certificate signing keys
by Nick Mathewson 19 Oct '23

19 Oct '23

To see this rendered, go to https://spec.torproject.org/proposals/347-domain-separation.html ``` Filename: 347-domain-separation.md Title: Domain separation for certificate signing keys Author: Nick Mathewson Created: 19 Oct 2023 Status: Open ``` ## Our goal We'd like to be able to use the "family key" from proposal 321 as a general purpose signing key, to authenticate other things than the membership of a family. For example, we might want to have a challenge/response mechanism where the challenger says, "If you want to log in as the owner of the account corresponding to this family, sign the following challenge with your key. Or we might want to have a message authentication scheme where an operator can sign a message in a way that proves key ownership. We _might_ also like to use relay identity keys or onion service identitiy keys for the same purpose. ## The problem When we're using a signing key for two purposes, it's important to perform some kind of domain separation so that documents signed for one purpose can't be mistaken for documents signed for the other. For example, in the challenge/response example, it would be bad if the challenger could provide a challenge string that would cause the signer to inadvertently authenticate an incorrect family. These keys are currently used in some places with no personalization. Their signature format is as described in `cert-spec.txt`, which says: > The signature is created by signing all the fields in > the certificate up until "SIGNATURE" (that is, signing > sizeof(ed25519_cert) - 64 bytes). ## One solution This one is pretty easy: we would extend `cert-spec` as follows. > ### Using signing keys for other purposes. > > Other protocols may introduce other uses for the signing keys > in these certificates other than those specified here. When > they do, they MUST ensure that the documents being signed > cannot be confused with the certificate bodies of this document. > > In some existing cases in the Tor protocols, we achieve this by > specifying an ASCII prefix string that must be prepended to the > other protocol's signed object before it is signed. > > For future protocols, we recommend that this be done by > specifying that the signing key is to be used to sign a > cSHAKE digest (or other secure customizable digest) > of the other protocol's signed object, using > a customization string unique to the other protocol. We would also make this amendment: > ### Future versions of this specification > > In order to maintain the domain separation that currently exists > between the signatures on these certificates and other documents > signed with the same keys, it suffices (for now!) that these > certificates begin with the version byte `[01]`, whereas > the other documents are in printable ASCII, which never > includes `[01]`. > > Future versions of this specification SHOULD move to using > an ed25519-prehash construction, using a customizable hash > with built-in domain separation.

2 1

Proposal 346: Clarifying and extending the use of protocol versioning
by Nick Mathewson 19 Oct '23

19 Oct '23

Note: we've updated our specs website! You can read a rendered version of this proposal at https://spec.torproject.org/proposals/346-protovers-again.html . ``` Filename: 346-protovers-again.md Title: Clarifying and extending the use of protocol versioning Author: Nick Mathewson Created: 19 Oct 2023 Status: Open ``` # Introduction In proposal 264, we introduced "subprotocol versions" as a way to independently version different pieces of the Tor protocols, and communicate which parts of the Tor protocols are supported, recommended, and required. Here we clarify the semantics of individual subprotocol versions, and describe more ways to use and negotiate them. # Semantics: Protocol versions are feature flags One issue we left unclarified previously is the relationship between two different versions of the same subprotocol. That is, if we know the semantics of (say) `Handshake=7`, can we infer anything about a relay that supports `Handshake=8`? In particular, can we infer that it supports all of the same features implied by `Handshake=7`? If we want to know "does this relay support some feature supported by `Handshake=7`", must we check whether it supports `Handshake=7`, or should we check `Handshake=x for any x≥7`? In this proposal, we settle the question as follows: subprotocol versions are flags. They do not have any necessary semantic relationship between them. We reject the `≥` interpretation for several reasons: * It's tricky to implement. * It prevents us from ever removing a feature. * It requires us to implement features in the same order across all Tor versions. ## ...but sometimes a flag _is_ a version! There _are_ some places in our protocol (notably: directory authority consensus methods, and channel protocol versions) where there _is_ a semantic relationship between version numbers. Specifically: "higher numbers are already better". When parties need to pick a one of _these_ versions, they always pick the highest version number supported by enough of them. When this kind of _real_ version intersects with the "subprotocol versions" system, we use the same numbers: * `Link` subprotocols correspond one-to-one with the version numbers sent in a VERSIONS cell. * `Microdesc` and `Cons` subprotocols correspond to a _subset_ of the version numbers of consensus methods. ## How to document subprotocol versions When describing a subprotocol, we should be clear what relationship, if any, exists between its versions and any versions negotiated elsewhere in the specifications. Unless otherwise documented, all versions can be in use _at the same time_: if only one can exist at once (on a single circuit, a single document, etc), this must be documented. > Implication: This means that we must say (for example) that you > can't use Link=4 and Link=5 on the same channel. # Negotiating protocol versions in circuit handshakes. Here we describe a way for a client to opt into features as part of its circuit handshake, in order to avoid proliferating negotiating extensions. ## Binary-encoding protocol versions. We assign a one-byte encoding for each protocol version number, ordered in the same way as in tor-spec. | Protocol | Id | | --- | --- | | Link | 0 | | LinkAuth | 1 | | Relay | 2 | | DirCache | 3 | | HSDir | 4 | | HSIntro | 5 | | HSRend | 6 | | Desc | 7 | | MicroDesc| 8 | | Cons | 9 | | Padding | 10 | | FlowCtrl | 11 | | Conflux | 12 | | Datagram | 13 | > Note: This is the same encoding used in the [walking onions > proposal][prop323]. It takes its order from the ordering of > protocol versions in > [tor-spec][subprotocol-versioning] > and matches up with the values defined in for `protocol_type_t` in C > tor's `protover.h`. ## Requesting an opt-in circuit feature When a client wants to request a given set of features, it sends an `ntor_v3` extension containing: ``` struct subproto_request { struct req[..]; // up to end of extension } struct req { u8 protocol_id; u8 protovol_version; } ``` > Note 1: The above format does not include any parameters for each > req. Thus, if we're negotiating an extension that requires a client- > supplied parameter, it may not be appropriate to use this > request format. > > Note 2: This proposal does not include any relay extension > acknowledging support. In the case of individual subprotocols, we could > later say "If this subprotocol is in use, the relay MUST also send > extension foo". > > Note 3: The existence of this extension does not preclude the later > addition of other extensions to negotiate featuress differently, or > to do anything else. Each `req` entry corresponds to a single subprotocol version. A client MUST NOT send any `req` entry unless: * That subprotocol version is advertised by the relay, * OR that subprotocol version is listed as required for relays in the current consensus, using `required-relay-protocols`. > Note: We say above that a client may request a _required_ subprotocol > even if the relay does not advertise it. This is what allows > clients to send a `req` extension to introduction points and > rendezvous points, even when we do not recognize the relay from the > consensus. > > Note 2: If the intro/rend point does not support a _required_ protocol, > it should not be on the network, and the client/service should not have > selected it. If a relay receives a `subproto_request` extension for any subprotocol version that it does not support, it MUST reject the circuit with a DESTROY cell. > Alternatives: we _could_ give the relay the option to > decline to support an extension, and we _could_ require the > relay to acknowledge which extensions it is providing. > We aren't doing that, in the name of simplicity. Only certain subprotocol versions need to be negotiated in this way; they will be explicitly listed as such in our specifications, with language like "This extension is negotiated as part of the circuit extension handshake". Other subprotocol versions MUST NOT be listed in this extension; if they are, the relay SHOULD reject the circuit. > Alternative: We _could_ allow the client to list other subprotocols > that the relay supports which are nonetheless irrelevant to > the circuit protocol, like `Microdesc`, or ones that don't currently need > to be negotiated, like `HsRend`. > > This is not something we plan to do. Currently specified subprotocol versions which can be negotiated using this extension are: * FlowCtrl=2 (congestion control) * Packed-and-fragmented support (proposal 340) The availability of the `subproto_request` extension itself will be indicated by a new Relay=X flag. When used, it will supplant several older `ntor_v3` extensions, including: * (TODO: list these here, if we find any. I think FlowCtrl has an extension?) That is, if using `subproto_request`, there is no need to send the (TODO) extensions. ## Making features that can be disabled. Sometimes, we will want the ability to make features that can be enabled or disabled from the consensus. But if we were to make a single flag that can turn the feature on and off, we'd run into trouble: after the feature was turned off, every relay would stop providing it right away, but there would be a delay before clients realized that the relays had stopped advertising the feature. During this interval, clients would try to enable the feature, and the relays would reject their circuits. To solve this problem, we need to make features like these controlled by a _pair_ of consensus parameters: one to disable advertising the feature, and one to disable the feature itself. To disable a feature, first the authorities would tell relays to stop advertising it, and only later tell the relays to stop supporting it. (If we want to _enable_ a previously disabled feature, we can turn on advertisement and support at the same time.) These parameters would be specified something like this (for a hypthetical `Relay=33` feature). * `support-relay-33`: if set to 1, relays that can provide `Relay=33` should do so. * `advertise-relay-33`: if set to 1, relays that are providing `Relay=33` should include it in their advertised protocol versions. Note: as a safety measure, relays MUST NOT advertise any feature that they do not support. This is reflected in the descriptions of the parameters above. When we add a new feature of this kind, we should have the `advertise-*` flag parameter be `1` by default, and probably we should have `support-*` be `1` by default oo. # Subprotocol versions in onion services Here we describe how to expand the onion service protocols in order to better accomodate subprotocol versions. ## Advertising an onion service's subprotocols In its encrypted descriptor (the innermost layer), the onion service adds a new entry: * `"protocols"` - A list of supported subprotocol versions, in the same format as those listed in a microdescriptor or descriptor. Note that this is NOT a complete list of all the subprotocol versions actually supported by the onion service. Instead, onion services only advertise a subprotocol version if they support it, _and_ it is documented in the specs as being supported by onion services. > Alternative: I had considered having a mask that would be put in the > consensus document, telling the onion services which subprotocols > to advertise. I don't think that's a great idea, however. Right now, the following protocols should be advertised: - FlowCtrl - Conflux (?? Doesn't this take parameters? TODO) - Pow (??? Doesn't this take parameters? If we do it, we need to allocate a subprotocol for it. TODO) ## Negotiating subprotocols with an onion service. In the `hs_ntor` handshake sent by the client, we add an encrypted `subproto_request` extension of the same format, with the same semantics, as used in the `ntor-v3` handshake. This supplants the following: - (Congestion Control; Pow? TODO) ## Advertising other relays' subprotocols? > Alternative: I had previously considered a design where the introduction points > in the onion service descriptor would be listed along with their > subprotocols, and the `hs_ntor` handshake would contain the > subprotocols of the rendezvous point. > > I'm rejecting this design for now because it gives the onion service > and the client too much freedom to lie about relays. In the future, > the walking onions design would solve this, since the contact > information for intro and rend points would be authenticated. # Appendix New numbers to reserve: * An extension ID for the `ntor_v3` handshake `subproto_request` extension. * An extension ID for the `hs_ntor` handshake `subproto_request` extension. * A Relay= subprotocol indicating support for the ntor-v3 and hs_ntor extensions. * The numeric encoding of each existing subprotocol, in the table above. [prop323]: https://spec.torproject.org/proposals/323-walking-onions-full.html [subprotocol-versioning]: https://spec.torproject.org/tor-spec/subprotocol-versioning.html

1 0

PT bridge reporting significant numbers of <OR> IP addresses after upgrade to 0.4.8.6; Conflux related?
by David Fifield 10 Oct '23

10 Oct '23

The Snowflake bridges do not expose their plain ORPort (they have `ORPort 127.0.0.1:auto` in torrc), and consequently they have always reported ≈0 <OR> IP addresses in the bridge-ip-transports line of bridge-extra-info descriptors, with virtually all connecting IP addresses being instead attributed to the snowflake transport. (The <OR> count is low but nonzero, which I have rationalized as tor making some small number of internal connections, or something.) I upgraded one of the two Snowflake bridges from 0.4.7.13 to 0.4.8.6 on 2023-09-24. Since then, the number of <OR> IP addresses has been roughly equal to the number of snowflake IP addresses. The ORPort is still not exposed; these are not external vanilla bridge users. Did something change between these versions that might cause PT connections to be double-counted, once for the transport and once for <OR>? Here are excerpted bridge-extra-info descriptors from before and after the version upgrade. Note the bridge-ip-transports lines. ``` @type bridge-extra-info 1.3 extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0 master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks published 2023-09-19 10:46:08 transport snowflake bridge-ip-versions v4=13528,v6=1384 bridge-ip-transports <OR>=8,snowflake=14912 ``` ``` @type bridge-extra-info 1.3 extra-info crusty5 91DA221A149007D0FD9E5515F5786C3DD07E4BB0 master-key-ed25519 1y9CAtinlbrhDuYBSOBNiCU9Ck1lcY7LErxnzhtxVks published 2023-09-29 17:33:20 transport snowflake bridge-ip-versions v4=2880,v6=336 bridge-ip-transports <OR>=1632,snowflake=1592 ``` Below is a time series for a whole month. The thing to notice is not the overall decrease in users on 2023-09-20; that's due to https://forum.torproject.org/t/problems-with-snowflake-since-2023-09-20-bro…. Rather, the thing to notice is that before 2023-09-24, <OR> ≈ 0; and after, <OR> ≈ snowflake. date <OR> snowflake 2023-08-30 4 45240 ← running 0.4.7.13 here 2023-08-31 21 193412 2023-09-01 30 195647 2023-09-02 40 200405 2023-09-03 40 201730 2023-09-04 42 195097 2023-09-05 35 190842 2023-09-06 30 189694 2023-09-07 26 191524 2023-09-08 35 195833 2023-09-09 32 196244 2023-09-10 27 146972 2023-09-11 6 55491 2023-09-12 18 182519 2023-09-13 17 173536 2023-09-14 21 161396 2023-09-15 21 166379 2023-09-16 16 174350 2023-09-17 17 165063 2023-09-18 23 164734 2023-09-19 32 164075 2023-09-20 23 121623 2023-09-21 2 25268 2023-09-22 0 4314 2023-09-23 0 746 2023-09-24 239 606 ← upgrade to 0.4.8.6 on this day 2023-09-25 1919 1900 2023-09-26 3533 3470 2023-09-27 11919 11687 2023-09-28 20242 19890 2023-09-29 26583 26100 2023-09-30 25929 25400

2 2

Proposal 345: Migrating the tor specifications to mdbook
by Nick Mathewson 03 Oct '23

03 Oct '23

``` Filename: 345-specs-in-mdbook.md Title: Migrating the tor specifications to mdbook Author: Nick Mathewson Created: 2023-10-03 Status: Open ``` # Introduction I'm going to propose that we migrate our specifications to a set of markdown files, specifically using the [`mdbook`] tool. This proposal does _not_ propose a bulk rewrite of our specs; it is meant to be a low-cost step forward that will produce better output, and make it easier to continue working on our specs going forward. That said, I think that this change will enable rewrites in the future. I'll explain more below. # What is mdbook? Mdbook is a tool developed by members of the Rust community to create books with [Markdown]. Each chapter is a single markdown file; the files are organized into a book using a [`SUMMARY.md`] file. Have a look at the [`mdbook` documentation][`mdbook`]; this is what the output looks like. Have a look at this [source tree]: that's the input that produces the output above. Markdown is extensible: it can use numerous [plugins] to enhance the semantics of the the markdown input, add diagrams, output in more formats, and so on. # What would using mdbook get us _immediately_? There are a bunch of changes that we could get immediately via even the simplest migration to mdbook. These immediate benefits aren't colossal, but they are things we've wanted for quite a while. * We'll have a document that's easier to navigate (via the sidebars). * We'll finally have good HTML output. * We'll have all our specifications organized into a single "document", able to link to one another and cross reference one another. * We'll have perma-links to sections. * We'll have a built-in text search function. (Go to the [`mdbook`] documentation and hit "`s`" to try it out.) ## How will mdbook help us _later on_ as we reorganize? Many of the benefits of mdbook will come later down the line as we improve our documentation. * Reorganizing will become much easier. * Our links will no longer be based on section number, so we won't have to worry about renumbering when we add new sections. * We'll be able to create redirects from old section filenames to new ones if we need to rename a file completely. * It will be far easier to break up our files into smaller files when we find that we need to reorganize material. * We will be able make our documents even _easier_ to navigate. * As we improve our documentation, we'll be able to use links to cross-reference our sections. * We'll be able to include real diagrams and tables. * We're already using the [`mermaid`] tool in Arti in generate [protocol diagrams] and [architecture diagrams]; we can use this in our specs too, instead of hand-drawn ASCII art. * We'll be able to integrate proposals more easily. * New proposals can become new chapters in our specification simply by copying them into a new 'md' file or files; we won't have to decide between integrating them into existing files or creating a new spec. * Implemented but unmerged proposals can become additional chapters in an appendix to the spec. We can refer to them with permalinks that will still work when they move to another place in the specs. # How should we do this? ## Strategy My priorities here are: * no loss of information, * decent-looking output, * a quick automated conversion process that won't lose a bunch of time. * a process that we can run experimentally until we are satisfied with the results With that in mind, I'm writing a simple set of [`torspec-converter`] scripts to convert our old torspec.git repository into its new format. We can tweak the scripts until we like the that they produce. After running a recent `torspec-converter` on a fairly recent torspec.git, here is how the branch looks: https://gitlab.torproject.org/nickm/torspec/-/tree/spec_conversion?ref_type… And here's the example output when running mdbook on that branch: https://people.torproject.org/~nickm/volatile/mdbook-specs/index.html > Note: these is not a permanent URL; we won't keep the example output > forever. When we actually merge the changes, they will move into > whatever final location we provide. The conversion script isn't perfect. It only recognizes three kinds of content: headings, text, and "other". Content marked "other" is marked with \`\`\` to reneder it verbatim. The choice of which sections to split up and which to keep as a single page is up to us; I made some initial decisions in the file above, but we can change it around as we please. See the [configuration section] at the end of the `grinder.py` script for details on how it's set up. ## Additional work that will be needed Assuming that we make this change, we'll want to build an automated CI process to build it as a website, and update the website whenever there is a commit to the specifications. (This automated CI process might be as simple as `git clone && mdbook build && rsync -avz book/ $TARGET`.) We'll want to go through our other documentation and update links, especially the permalinks in spec.torproject.org. It might be a good idea to use spec.torproject.org as the new location of this book, assuming weasel (who maintains spec.tpo) also thinks it's reasonable. If we do that, we need to decide on what we want the landing page to look like, and we need _very much_ to get our permalink story correct. Right now I'm generating a .htaccess file as part of the conversion. ## Stuff we shouldn't do. I think we should continue to use the existing torspec.git repository for the new material, and just move the old text specs into a new archival location in torspec. (We *could* make a new repository entirely, but I don't think that's the best idea. In either case, we shouldn't change the text specifications after the initial conversion.) We'll want to figure out our practices for keeping links working as we reorganize these documents. Mdbook has decent redirect support, but it's up to us to actually create the redicrets as necessary. # The transition, in detail * Before the transition: - Work on the script until it produces output we like. - Finalize this proposal and determine where we are hosting everything. - Develop the CI process as needed to keep the site up to date. - Get approval and comment from necessary stakeholders. - Write documentation as needed to support the new way of doing things. - Decide on the new layout we want for torspec.git. * Staging the transition: - Make a branch to try out the transition; explicitly allow force-pushing that branch. (Possibly nickm/torspec.git in a branch called mdbook-demo, or torspec.git in a branch called mdbook-demo assuming it is not protected.) - Make a temporary URL to target with the transition (possibly spec-demo.tpo) - Once we want to do the transition, shift the scripts to tpo/torspec.git:main and spec.tpo, possibly? * The transition: - Move existing specs to a new subdirectory in torspec.git. - Run the script to produce an mdbook instance in torspec.git with the right layout. - Install the CI process to keep the site up to date. * Post-transition - Update links elsewhere. - Continue to improve the specs. # Integrating proposals We could make all of our proposals into a separate book, like rust does at https://rust-lang.github.io/rfcs/ . We could also leave them as they are for now. (I don't currently think we should make all proposals part of the spec automatically.) # Timing I think the right time to do this, if we decide to move ahead, is before November. That way we have this issue as something people can work on during the docs hackathon. # Alternatives I've tried experimenting with Docusaurus here, which is even more full-featured and generates pretty react sites [like this](https://docusaurus.io/). (We're likely to use it for managing the Arti documentation and website.) For the purposes we have here, it seems slightly overkill, but I do think a migration is feasible down the road if we decide we _do_ want to move to docusaurus. The important thing is the ability to keep our URLs working, and I'm confident we could do that The main differences for our purposes here seem to be: * The markdown implementation in Docusaurus is extremely picky about stuff that looks like HTML but isn't; it rejects it, rather than passing it on as text. Thus, using it would require a more painstaking conversion process before we could include text like `"<state:on>"` or `"A <-> B"` as our specs do in a few places. * Instead of organizing our documents in a `SUMMARY.md` with an MD outline format, we'd have to organize them in a `sidebar.js` with a javascript syntax. * Docusaurus seems to be far more flexible and have a lot more features, but also seems trickier to configure. <-- References --> [`mdbook`]: https://rust-lang.github.io/mdBook/ [source tree]: https://github.com/rust-lang/mdBook/tree/master/guide/src/ [Markdown]: https://en.wikipedia.org/wiki/Markdown [`SUMMARY.md`]: https://rust-lang.github.io/mdBook/format/summary.html [plugins]: https://github.com/rust-lang/mdBook/wiki/Third-party-plugins [`mermaid`]: https://mermaid.js.org/ [architecture diagrams]: https://gitlab.torproject.org/tpo/core/arti/-/blob/main/doc/dev/Architectur… [protocol diagrams]: https://gitlab.torproject.org/tpo/core/arti/-/blob/main/doc/dev/hs-overview… [`torspec-converter`]: https://gitlab.torproject.org/nickm/torspec-converter [configuration section]: https://gitlab.torproject.org/nickm/torspec-converter/-/blob/main/grinder.p…

1 0

[Update] Network Status APIs
by Mattia Righetti 03 Oct '23

03 Oct '23

Greetings Tor devs, I wanted to give a quick update on the progress of the Network Status APIs[0]. First of all, a couple of weeks ago we have deployed an initial version of the service internally so that we can benchmark and test it and see how it holds. Plus, we now have developed three more APIs: `/details`, `/clients`, `/weights`. In total, we now have covered almost every endpoint that is currently provided by onionoo. Just to give you a little bit more insights, `/summary` and `/details` fetch data from the Network Health team database and return a response identical to the one that onionoo clients expect right now. On the other hand, `/clients`, `/weights` and `bandwidth` will proxy the request to VictoriaMetrics. In the upcoming weeks we plan to stabilize and test these endpoints further and do some perf improvements. Meanwhile, we are also working on documenting the APIs so that potential clients know how to use them, when they'll be available. Those can be found in the project's repo Wiki[1]. Please do reach out to me, hiro or GeKo if you have any ideas or feedback you'd like to share with us. Cheers, Matt [0]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/ [1]: https://gitlab.torproject.org/tpo/network-health/metrics/networkstatusapi/-…

1 0