On 12 Dec 2017, at 09:41, Tom Ritter tom@ritter.vg wrote:
I'm not sure, but I think https://trac.torproject.org/projects/tor/ticket/21377 needed a proposal so I tried to write one up.
Thanks Tom!
Proposals are great for new directory authority features, particularly when we're changing the directory specification.
I've quoted the proposal below with my comments:
Filename: xxx-expose-bwauth_votes.txt Title: Have Directory Authorities expose raw bwauth vote documents Author: Tom Ritter Created: 11-December-2017 Status: Open
- Introduction
Bandwidth Authorities (bwauths) perform scanning of the Tor Network and calculate observed speeds for each relay. They produce a 'bwauth vote file' that is given to a Directory Authority. The Directory Authority uses the speed value from this file in its vote file denoting its view of the speed of the relay.
After collecting all of the votes from other Authorities, a consensus is calculated, and the consensus's view of a relay's speed is determined by choosing the low-median value [or is it high-median?]
low-median
of all the authorities' values for each relay.
Only a single metric from the bwauth vote file is exposed by a Directory Authority's vote, however the original file contains considerably more diagnostic information about how the bwauth arrives at that measurement for that relay.
- Motivation
The bwauth vote file contains more information that is exposed in the overall vote file. This information is useful to debug anomalies in relays' utilization and suspected bugs in the (decrepit) bwauth code.
Currently, all bwauths expose the raw vote file through various (non- standard) means, and that file is downloaded (hourly) by a single person (as long as his home internet connection and home server is working) and archived (with a small amount of robustness.)
It would be preferable to have this exposed in a standard manner. Doing so would no longer require bwauths to run HTTP servers to expose the file, no longer require them to take additional manual steps to provide it, and would enable public consumption by any interested parties. We hope that Collector will begin archiving the files.
- Specification
An authority SHOULD publish the bwauth vote used to calculate its current vote. It should make the bwauth vote file available at the
SHOULD
same time as its normal vote file.
Since the file is created by an external process, and we never store a copy of any old files, I suggest that we make it available at the URL at all times, as "the file that the authority has right now".
(And return a 404 when the file does not exist, is not readable, etc.)
It should make the file available at http://<hostname>/tor/status-vote/next/bwauth.z
We shouldn't use next/ unless we're willing to cache a copy of the file we actually used to vote. If we do, we should serve it from next/ as soon as we vote using it, then serve it from current/ as soon as the consensus is created.
If we don't store a copy of the file, we should use a different URL, like status-vote/now/bwauth, and recommend that the file is fetched at hh:50, when the votes are created. This would allow us to implement current/bwauth and next/bwauth in a future version.
Have you thought about versioning the URL if we have multiple flavours of bwauth file? We could use bwauth-<FLAVOR> like consensuses.
Also, Tor has new compression options for zstd and lzma.
Given that this is an externally-controlled file, we could stream it from disk and compress it on the fly with something cheap like gzip or zstd.
It MUST NOT attempt to send its bwauth vote file in a HTTP POST to other authorities and it SHOULD NOT make bwauth vote files from other authorities available.
- Security Implications
The raw bwauth vote file does not [really: is not believed to] expose any sensitive information. All authorities currently make this document public already, an example is at https://bwauth.ritter.vg/bwauth/bwscan.V3BandwidthsFile
How large is the file? Maybe we should pre-compress it, to avoid CPU exhaustion attacks. If we did this, we could say that it's safe, as long as it is smaller than the full consensus or full set of descriptors.
- Compatibility
Exposing the document presents no compatibility concerns.
The compatibility concern is with applications that want to consume the document. The bwauth vote file has no specification, and has been extended in ad-hoc ways. Applications that merely wish to archive the document (e.g. Collector) won't have a problems. Applications that want to parse it may encounter errors if a new (unexpected) field is added, or assumptions are made about the text encoding or formatting of the document.
Parsers may also encounter issues if we transition to another bwauth implementation. We could remove fields if this happens.
T