Hi,
tldr:
- more outdated relays
(that is a claim I'm making and you could
easily proof me wrong by recreating the 0.3.3.x alpha
repos and ship 0.3.3.7 in them and see how things evolve
after a week or so)
- more work for the tpo website maintainer
- less happy relay operators [3][4]
- more work for repo maintainers? (since a new repo needs to be created)
When the tor 0.3.4 alpha repos (deb.torproject.org) first appeared on 2018-05-23
I was about to submit a PR for the website to include …
[View More]it in the sources.list
generator [1] on tpo but didn't do it because I wanted to wait for a previous PR to be merged first.
The outstanding PR got merged eventually (2018-06-28) but I still did not submit a PR to
update the repo generator for 0.3.4.x nonetheless and here is why.
Recently I was wondering why are there so many relays running tor version 0.3.3.5-rc? (see OrNetStats or Relay Search)
(> 3.2% CW fraction)
Then I realized that this was the last version the tor-experimental-0.3.3.x-*
repos were shipping before they got abandoned due to the new 0.3.4.x-* repos
(I can no longer verify it since they got removed by now).
Peter made it clear in the past that the current way to
have per-major-version debian alpha repos (i.e. tor-experimental-0.3.4.x-jessie)
will not change [2]:
> If you can't be bothered to change your sources.list once or twice a
> year, then you probably should be running stable.
but maybe someone else would be willing to invoke a
"ln" commands everytime a new new alpha repo is born.
tor-alpha-jessie -> tor-experimental-0.3.4.x-jessie
once 0.3.5.x repos are created the link would point to
tor-alpha-jessie -> tor-experimental-0.3.5.x-jessie
It is my opinion that this will help reduce the amount of relays running
outdated versions of tor.
It will certainly avoid having to update the tpo website, which isn't a big task
and could probably be automated but it isn't done currently.
"..but that would cause relay operators to jump from i.e. 0.3.3.x to 0.3.4.x alphas
(and break setups)!"
Yes, and I think that is better than relays stuck on an older version because
the former repo no longer exists and operators still can choose the old repos
which will not jump to newer major versions.
[1] https://www.torproject.org/docs/debian.html.en#ubuntu
[2] https://trac.torproject.org/projects/tor/ticket/14997#comment:3
[3] https://lists.torproject.org/pipermail/tor-relays/2018-June/015549.html
[4] https://trac.torproject.org/projects/tor/ticket/26474
--
https://twitter.com/nusenu_https://mastodon.social/@nusenu
[View Less]
Hi,
every now and then I'm in contact with relay operators
about the "health" of their relays.
Following these 1:1 discussions and the discussion on tor-relays@
I'd like to rise two issues with you (the developers) with the goal
to help improve relay operations and end user experience in the long term:
1) DNS (exits only)
2) tor relay health data
1) DNS
------
Current situation:
Arthur Edelstein provides public measurements to tor exit relay operators via
his page at: https://…
[View More]arthuredelstein.net/exits/
This page is updated once daily.
the process to use that data looks like this:
- first they watch Arthur's measurement results
- if their failure rate is non-zero they try to tweak/improve/change their setup
- wait for another 24 hours (next measurement)
This is a somewhat suboptimal and slow feedback loop and is probably also
less accurate and less valuable data when compared to the data the tor
process can provide.
Suggestion for improvement:
Exposes the following DNS status information
via tor's controlport to help debug and detect DNS issues on exit relays:
(total numbers since startup)
- amount of DNS queries send to the resolver
- amount of DNS queries send to the resolver due to a RESOLVE request
- DNS queries send to resolver due to a reverse RESOLVE request
- amount of queries that did not result in any answer from the resolver
- breakdown of number of responses by response code (RCODE)
https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-pa…
- max amount of DNS queries send per curcuit
If this causes a significant performance impact this feature should be disabled
by default.
2) general relay health metrics
--------------------------------
Compared to other server daemons (webserver, DNS server, ..)
tor provides little data for operators to detect operational issues
and anomalies.
I'd suggest to provide the following stats via the control port:
(most of them are already written to logfiles by default but not accessible
via the controlport as far as I've seen)
- total amount of memory used by the tor process
- amount of currently open circuits
- circuit handshake stats (TAP / NTor)
DoS mitigation stats
- amount of circuits killed with too many cells
- amount of circuits rejected
- marked addresses
- amount of connections closed
- amount of single hop clients refused
- amount of closed/failed circuits broken down by their reason value
https://gitweb.torproject.org/torspec.git/tree/tor-spec.txt#n1402https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n1994
- amount of closed/failed OR connections broken down by their reason value
https://gitweb.torproject.org/torspec.git/tree/control-spec.txt#n2205
If this causes a significant performance impact this feature should be disabled
by default.
cell stats
- extra info cell stats
as defined in:
https://gitweb.torproject.org/torspec.git/tree/dir-spec.txt#n1072
This data should be useful to answer the following questions:
- High level questions: Is the tor relay healthy?
- is it hitting any resource limits?
- is the tor process under unusual load?
- why is tor using more memory?
- is it slower than usual at handling circuits?
- can the DNS resolver handle the amount of DNS queries tor is sending it?
This data could help prevent errors from occurring or provide
additional data when trying to narrow down issues.
When it comes to the question:
**Is it "safe" to make this data accessible via the controlport?**
I assume it is safe for all information that current versions of
tor writes to logfiles or even publishes as part of its extra info descriptor.
Should tor provide this or similar data
I'm planing to write scripts for operators to make use
of that data (for example a munin plugin that connects to tor's controlport).
I'm happy to help write updates for control-spec should these features
seem reasonable to you.
Looking forward to hearing your feedback.
nusenu
--
https://twitter.com/nusenu_https://mastodon.social/@nusenu
[View Less]
Filename: xxx-flashflow.txt
Title: FlashFlow: A Secure Speed Test for Tor (Parent Proposal)
Author: Matthew Traudt, Aaron Johnson, Rob Jansen, Mike Perry
Created: 23 April 2020
Status: Draft
1. Introduction
FlashFlow is a new distributed bandwidth measurement system for Tor that
consists of a single authority node ("coordinator") instructing one or
more measurement nodes ("measurers") when and how to measure Tor relays.
A measurement consists of the following steps:
1. The measurement …
[View More]nodes demonstrate to the target relay permission to
perform measurements.
2. The measurement nodes open many TCP connections to the target relay
and create a one-hop circuit to the target relay on each one.
3. For 30 seconds the measurement nodes send measurement cells to the
target relay and verify that the cells echoed back match the ones
sent. During this time the relay caps the amount of background
traffic it transfers. Background and measurement traffic are
handled separately at the relay. Measurement traffic counts towards
all the standard existing relay statistics.
4. For every second during the measurement, the measurement nodes
report to the authority node how much traffic was echoed back. The
target relay also reports the amount of per-second background
(non-measurement) traffic.
5. The authority node sums the per-second reported throughputs into 30
sums (one for each second) and calculates the median. This is the
estimated capacity of the relay.
FlashFlow performs a measurement of every relay according to a schedule
described later in this document. Periodically it produces relay
capacity estimates in the form of a v3bw file, which is suitable for
direct consumption by a Tor directory authority. Alternatively an
existing load balancing system such as Simple Bandwidth Scanner could be
modified to use FlashFlow's v3bw file as input.
It is envisioned that each directory authority that wants to use
FlashFlow will run their own FlashFlow deployment consisting of a
coordinator that they run and one or more measurers that they trust
(e.g. because they run them themselves), similar to how each runs their
own Torflow/sbws. Section 5.2 of this proposal describes long term plans
involving multiple FlashFlow deployments.
FlashFlow is more performant than Torflow: FlashFlow takes 5 hours to
measure the entire existing Tor network from scratch (with 3 Gbit/s
measurer capacity) while Torflow takes 2 days; FlashFlow measures relays
it hasn't seen recently as soon as it learns about them (i.e. every new
consensus) while Torflow can take a day or more; and FlashFlow
accurately measures new high-capacity relays the first time and every
time while Torflow takes days/weeks to assign them their full fair share
of bandwidth (especially for non-exits). FlashFlow is more secure than
Torflow: FlashFlow allows a relay to inflate its measured capacity by up
to 1.33x (configured by a parameter) while Torflow allows weight
inflation by a factor of 89x [0] or even 177x [1].
After an overview in section 2 of the planned deployment stages, section
3, 4, and 5 discuss the short, medium, and long term deployment plans in
more detail.
2. Deployment Stages
FlashFlow's deployment shall be broken up into three stages.
In the short term we will implement a working FlashFlow measurement
system. This requires code changes in little-t tor and an external
FlashFlow codebase. The majority of the implementation work will be
done in the short term, and the product is a complete FlashFlow
measurement system. Remaining pieces (e.g. better authentication) are
added later for enhanced security and network performance.
In the medium term we will begin collecting data with a FlashFlow
deployment. The intermediate results and v3bw files produced will be
made available (semi?) publicly for study.
In the long term experiments will be performed to study ways of using FF
v3bw files to improve load balancing. Two examples: (1) using FF v3bw
files instead of sbws's (and eventually phasing out torflow/sbws), and
(2) continuing to run sbws but use FF's results as a better estimate of
relay capacity than observed bandwidth. Authentication and other
FlashFlow features necessary to make it completely ready for full
production deployment will be worked on during this long term phase.
3. FlashFlow measurement system: Short term
The core measurement mechanics will be implemented in little-t tor, but
a separate codebase for the FlashFlow side of the measurement system
will also be created. This section is divided into three parts: first a
discussion of changes/additions that logically reside entirely within
tor (essentially: relay-side modifications), second a discussion of the
separate FlashFlow code that also requires some amount of tor changes
(essentially: measurer-side and coordinator-side modifications), and
third a security discussion.
3.1 Little-T Tor Components
The primary additions/changes that entirely reside within tor on the
relay side:
- New torrc options/consensus parameters.
- New cell commands.
- Pre-measurement handshaking (with a simplified authentication
scheme).
- Measurement mode, during which the relay will echo traffic with
measurers, set a cap on the amount of background traffic it
transfers, and report the amount of transferred background traffic.
3.1.1 Parameters
FlashFlow will require some consensus parameters/torrc options. Each has
some default value if nothing is specified; the consensus parameter
overrides this default value; the torrc option overrides both.
FFMeasurementsAllowed: A global toggle on whether or not to allow
measurements. Even if all other settings would allow a measurement, if
this is turned off, then no measurement is allowed. Possible values: 0,
1. Default: 0 (disallowed).
FFAllowedCoordinators: The list of coordinator TLS certificate
fingerprints that are allowed to start measurements. Relays check their
torrc when they receive a connection from a FlashFlow coordinator to see
if it's on the list. If they have no list, they check the consensus
parameter. If nether exist, then no FlashFlow deployment is allowed to
measure this relay. Default: empty list.
FFMeasurementPeriod: A relay should expect on average, to be measured by
each FlashFlow deployment once each measurement period. A relay will not
allow itself to be measured more than twice by a FlashFlow deployment in
any time window of this length. Relays should not change this option
unless they really know what they're doing. Changing it at the relay
will not change how often FlashFlow will attempt to measure the relay.
Possible values are in the range [1 hour, 1 month] inclusive. Default: 1
day.
FFBackgroundTrafficPercent: The maximum amount of regular
non-measurement traffic a relay should handle while being measured, as a
percent of total traffic (measurement + non-measurement). This
parameter is a trade off between having to limit background traffic and
limiting how much a relay can inflate its result by handling no
background traffic but reporting that it has done so. Possible values
are in the range [0, 99] inclusive. Default: 25 (a maximum inflation
factor of 1.33).
FFMaxMeasurementDuration: The maximum amount of time, in seconds, that
is allowed to pass from the moment the relay is notified that a
measurement will begin soon and the end of the measurement. If this
amount of time passes, the relay shall close all measurement connections
and exit its measurement mode. Note this duration includes handshake
time, thus it necessarily is larger than the expected actual measurement
duration. Possible values are in the range [10, 120] inclusive.
Default: 45.
3.1.2 New Cell Types
FlashFlow will introduce a new cell command MEASURE.
The payload of each MEASURE cell consists of:
Measure command [1 byte]
Length [2 bytes]
Data [Length-3 bytes]
The measure commands are:
0 -- MSM_PARAMS [forward]
1 -- MSM_PARAMS_OK [backward]
2 -- MSM_ECHO [forward and backward]
3 -- MSM_BG [backward]
4 -- MSM_ERR [forward and backward]
Forward cells are sent from the measurer/coordinator to the relay.
Backward cells are sent from the relay to the measurer/coordinator.
MSM_PARAMS and MSM_PARAMS_OK are used during the pre-measurement stage
to tell the target what to expect and for the relay to positively
acknowledge the message. MSM_ECHO cells are the measurement traffic;
the measurer generates them, sends them to the target, and the target
echos them back. The target send a MSM_BG cell once per second to report
the amount of background traffic it is handling. MSM_ERR cells are used
to signal to the other party that there has been some sort of problem
and that the measurement should be aborted. These measure commands are
described in more detail in the next section.
The only cell that sometimes undergoes cell encryption is MSM_ECHO; no
other cell ever gets cell encrypted. (All cells are transmitted on a
regular TLS-wrapped OR connection; that encryption still exists.)
The relay "decrypts" MSM_ECHO cells before sending them back to the
measurer; this mirrors the way relays decrypt/encrypt RELAY_DATA cells
in order to induce realistic cryptographic CPU load. The measurer
usually skips encrypting MSM_ECHO cells to reduce its own CPU load;
however, to verify the relay is actually correctly decrypting all cells,
the measurer will choose random outgoing cells, encrypt them, remember
the ciphertext, and verify the corresponding incoming cell matches.
3.1.3 Pre-Measurement Handshaking/Starting a Measurement
The coordinator connects to the target relay and sends it a MSM_PARAMS
cell. If the target is unwilling to be measured at this time or if the
coordinator didn't use a TLS certificate that the target trusts, it
responds with an error cell and closes the connection. Otherwise it
checks that the parameters of the measurement are acceptable (e.g. the
version is acceptable, the duration isn't too long, etc.). If the
target is happy, it sends a MSM_PARAMS_OK, otherwise it sends a MSM_ERR
and closes the connection.
Upon learning the IP addresses of the measurers from the coordinator in
the MSM_PARAMS cell, the target whitelists their IPs in its DoS
detection subsystem until the measurement ends (successfully or
otherwise), at which point the whitelist is cleared.
Upon receiving a MSM_PARAMS_OK from the target, the coordinator will
instruct the measurers to open their TCP connections with the target. If
the coordinator or any measurer receives a MSM_ERR, it reports the error
to the coordinator and considers the measurement a failure. It is also a
failure if any measurer is unable to open at least half of its TCP
connections with the target.
The payload of MSM_PARAMS cells [XXX more may need to be added]:
- version [1 byte]
- msm_duration [1 byte]
- num_measurers [1 byte]
- measurer_info [num_measurers times]
- ipv4_addr [4 bytes]
- num_conns [2 bytes]
version dictates how this MSM_PARAMS cell shall be parsed. msm_duration
is the duration, in seconds, that the actual measurement will last.
num_measurers is how many measurer_info structs follow. For each
measurer, the ipv4_addr it will use when connecting to the target is
provided, as is num_conns, the number of TCP connections that measurer
will open with the target. Future versions of FlashFlow and MSM_PARAMS
will use TLS certificates instead of IP addresses.
MSM_PARAMS_OK has no payload: it's just padding bytes to make the cell
514 bytes long.
The payload of MSM_ECHO cells:
- arbitrary bytes [max to fill up 514 byte cell]
The payload of MSM_BG cells:
- second [1 byte]
- sent_bg_bytes [4 bytes]
- recv_bg_bytes [4 bytes]
second is the number of seconds since the measurement began. MSM_BG
cells are sent once per second from the relay to the FlashFlow
coordinator. The first cell will have this set to 1, and each
subsequent cell will increment it by one. sent_bg_bytes is the number of
background traffic bytes sent in the last second (since the last MSM_BG
cell). recv_bg_bytes is the same but for received bytes.
The payload of MSM_ERR cells:
- err_code [1 byte]
- err_str [possibly zero-len null-terminated string]
The error code is one of:
[... XXX TODO ...]
255 -- OTHER
The error string is optional in all cases. It isn't present if the first
byte of err_str is null, otherwise it is present. It ends at the first
null byte or the end of the cell, whichever comes first.
3.1.4 Measurement Mode
The relay considers the measurement to have started the moment it
receives the first MSM_ECHO cell from any measurer. At this point, the
relay
- Starts a repeating 1s timer on which it will report the amount of
background traffic to the coordinator over the coordinator's
connection.
- Enters "measurement mode" and limits the amount of background
traffic it handles according to the torrc option/consensus
parameter.
The relay decrypts and echos back all MSM_ECHO cells it receives on
measurement connections until it has reported its amount of background
traffic the same number of times as there are seconds in the measurement
(e.g. 30 per-second reports for a 30 second measurement). After sending
the last MSM_BG cell, the relay drops all buffered MSM_ECHO cells,
closes all measurement connections, and exits measurement mode.
During the measurement the relay targets a ratio of background traffic
to measurement traffic as specified by a consensus parameter/torrc
option. For a given ratio r, if the relay has handled x cells of
measurement traffic recently, Tor then limits itself to y = xr/(1-r)
cells of non-measurement traffic this scheduling round. The target will
enforce that a minimum of 10 Mbit/s of measurement traffic is recorded
since the last background traffic scheduling round to ensure it always
allows some minimum amount of background traffic.
3.2 FlashFlow Components
The FF coordinator and measurer code will reside in a FlashFlow
repository separate from little-t tor.
There are three notable parameters for which a FF deployment must choose
values. They are:
- The number of sockets, s, the measurers should open, in aggregate,
with the target relay. We suggest s=160 based on the FF paper.
- The bandwidth multiplier, m. Given an existing capacity estimate for
a relay, z, the coordinator will instruct the measurers to, in
aggregate, send m*z Mbit/s to the target relay. We recommend m=2.25.
- The measurement duration, d. Based on the FF paper, we recommend
d=30 seconds.
The rest of this section first discusses notable functions of the
FlashFlow coordinator, then goes on to discuss FF measurer code that
will require supporting tor code.
3.2.1 FlashFlow Coordinator
The coordinator is responsible for scheduling measurements, aggregating
results, and producing v3bw files. It needs continuous access to new
consensus files, which it can obtain by running an accompanying Tor
process in client mode.
The coordinator has the following functions, which will be described in
this section:
- result aggregation.
- schedule measurements.
- v3bw file generation.
3.2.1.1 Aggregating Results
Every second during a measurement, the measurers send the amount of
verified measurement traffic they have received back from the relay.
Additionally, the relay sends a MSM_BG cell each second to the
coordinator with amount of non-measurement background traffic it is
sending and receiving.
For each second's reports, the coordinator sums the measurer's reports.
The coordinator takes the minimum of the relay's reported sent and
received background traffic. If, when compared to the measurer's reports
for this second, the relay's claimed background traffic is more than
what's allowed by the background/measurement traffic ratio, then the
coordinator further clamps the relay's report down. The coordinator adds
this final adjusted amount of background traffic to the sum of the
measurer's reports.
Once the coordinator has done the above for each second in the
measurement (e.g. 30 times for a 30 second measurement), the coordinator
takes the median of the 30 per-second throughputs and records it as the
estimated capacity of the target relay.
3.2.1.2 Measurement Schedule
The short term implementation of measurement scheduling will be simpler
than the long term one due to (1) there only being one FlashFlow
deployment, and (2) there being very few relays that support being
measured by FlashFlow. In fact the FF coordinator will maintain a list
of the relays that have updated to support being measured and have opted
in to being measured, and it will only measure them.
The coordinator divides time into a series of 24 hour periods, commonly
referred to as days. Each period has measurement slots that are longer
than a measurement lasts (30s), say 60s, to account for pre- and
post-measurement work. Thus with 60s slots there's 1,440 slots in a
day.
At the start of each day the coordinator considers the list of relays
that have opted in to being measured. From this list of relays, it
repeatedly takes the relay with the largest existing capacity estimate.
It selects a random slot. If the slot has existing relays assigned to
it, the coordinator makes sure there is enough additional measurer
capacity to handle this relay. If so, it assigns this relay to this
slot. If not, it keeps picking new random slots until one has sufficient
additional measurer capacity.
Relays without existing capacity estimates are assumed to have the 75th
percentile capacity of the current network.
If a relay is not online when it's scheduled to be measured, it doesn't
get measured that day.
3.2.1.2.1 Example
Assume the FF deployment has 1 Gbit/s of measurer capacity. Assume the
chosen multiplier m=2. Assume there are only 5 slots in a measurement
period.
Consider a set of relays with the following existing capacity estimates
and that have opted in to being measured by FlashFlow.
- 500 Mbit/s
- 300 Mbit/s
- 250 Mbit/s
- 200 Mbit/s
- 100 Mbit/s
- 50 Mbit/s
The coordinator takes the largest relay, 500 Mbit/s, and picks a random
slot for it. It picks slot 3. The coordinator takes the next largest,
300, and randomly picks slot 2. The slots are now:
0 | 1 | 2 | 3 | 4
-------|-------|-------|-------|-------
| | 300 | 500 |
| | | |
The coordinator takes the next largest, 250, and randomly picks slot 2.
Slot 2 already has 600 Mbit/s of measurer capacity reserved (300*m);
given just 1000 Mbit/s of total measurer capacity, there is just 400
Mbit/s of spare capacity while this relay requires 500 Mbit/s. There is
not enough room in slot 2 for this relay. The coordinator picks a new
random slot, 0.
0 | 1 | 2 | 3 | 4
-------|-------|-------|-------|-------
250 | | 300 | 500 |
| | | |
The next largest is 200 and the coordinator randomly picks slot 2 again
(wow!). As there is just enough spare capacity, the coordinator assigns
this relay to slot 2.
0 | 1 | 2 | 3 | 4
-------|-------|-------|-------|-------
250 | | 300 | 500 |
| | 200 | |
The coordinator randomly picks slot 4 for the last remaining relays, in
that order.
0 | 1 | 2 | 3 | 4
-------|-------|-------|-------|-------
250 | | 300 | 500 | 100
| | 200 | | 50
3.2.1.3 Generating V3BW files
Every hour the FF coordinator produces a v3bw file in which it stores
the latest capacity estimate for every relay it has measured in the last
week. The coordinator will create this file on the host's local file
system. Previously-generated v3bw files will not be deleted by the
coordinator. A symbolic link at a static path will always point to the
latest v3bw file.
$ ls -l
v3bw -> v3bw.2020-03-01-05-00-00
v3bw.2020-03-01-00-00-00
v3bw.2020-03-01-01-00-00
v3bw.2020-03-01-02-00-00
v3bw.2020-03-01-03-00-00
v3bw.2020-03-01-04-00-00
v3bw.2020-03-01-05-00-00
3.2.2 FlashFlow Measurer
The measurers take commands from the coordinator, connect to target
relays with many sockets, send them traffic, and verify the received
traffic is the same as what was sent. Measurers need access to a lot of
internal tor functionality. One strategy is to house as much logic as
possible inside an compile-time-optional control port module that calls
into other parts of tor. Alternatively FlashFlow could link against tor
and call internal tor functions directly.
[XXX for now I'll assume that an optional little-t tor control port
module housing a lot of this code is the best idea.]
Notable new things that internal tor code will need to do on the
measurer (client) side:
1. Open many TLS+TCP connections to the same relay on purpose.
2. Verify echo cells.
3.2.2.1 Open many connections
FlashFlow prototypes needed to "hack in" a flag in the
open-a-connection-with-this-relay function call chain that indicated
whether or not we wanted to force a new connection to be created. Most
of Tor doesn't care if it reuses an existing connection, but FF does
want to create many different connections. The cleanest way to
accomplish this will be investigated.
On the relay side, these measurer connections do not count towards DoS
detection algorithms.
3.2.2.2 Verify echo cells
A parameter will exist to tell the measurers with what frequency they
shall verify that cells echoed back to them match what was sent. This
parameter does not need to exist outside of the FF deployment (e.g. it
doesn't need to be a consensus parameter).
The parameter instructs the measurers to check 1 out of every N cells.
The measurer keeps a count of how many measurement cells it has sent. It
also logically splits its output stream of cells into buckets of size N.
At the start of each bucket (when num_sent % N == 0), the measurer
chooses a random index in the bucket. Upon sending the cell at that
index (num_sent % N == chosen_index), the measurer records the cell.
The measurer also counts cells that it receives. When it receives a cell
at an index that was recorded, it verifies that the received cell
matches the recorded sent cell. If they match, no special action is
taken. If they don't match, the measurer indicates failure to the
coordinator and target relay and closes all connections, ending the
measurement.
3.2.2.2.1 Example
Consider bucket_size is 1000. For the moment ignore cell encryption.
We start at idx=0 and pick an idx in [0, 1000) to record, say 640. At
idx=640 we record the cell. At idx=1000 we choose a new idx in [1000,
2000) to record, say 1236. At idx=1236 we record the cell. At idx=2000
we choose a new idx in [2000, 3000). Etc.
There's 2000+ cells in flight and the measurer has recorded two items:
- (640, contents_of_cellA)
- (1236, contents_of_cellB)
Consider the receive side now. It counts the cells it receives. At
receive idx=640, it checks the received cell matches the saved cell from
before. At receive idx=1236, it again checks the received cell matches.
Etc.
3.2.2.2.2 Motivation
A malicious relay may want to skip decryption of measurement cells to
save CPU cycles and obtain a higher capacity estimate. More generally,
it could generate fake measurement cells locally, ignore the measurement
traffic it is receiving, and flood the measurer with more traffic that
it (the measurer) is even sending.
The security of echo cell verification is discussed in section 3.3.1.
3.3 Security
In this section we discuss the security of various aspects of FlashFlow
and the tor changes it requires.
3.3.1 Echo Cell Verification: Bucket Size
A smaller bucket size means more cells are checked and FF is more likely
to detect a malicious target. It also means more bookkeeping overhead
(CPU/RAM).
An adversary that knows bucket_size and cheats on one item out of every
bucket_size items will have a 1/bucket_size chance of getting caught in
the first bucket. This is the worst case adversary. While cheating on
just a single item per bucket yields very little advantage, cheating on
more items per bucket increases the likelihood the adversary gets
caught. Thus only the worst case is considered here.
In general, the odds the adversary can successfully cheat in a single
bucket are
(bucket_size-1)/bucket_size
Thus the odds the adversary can cheat in X consecutive buckets are
[(bucket_size-1)/bucket_size]^X
In our case, X will be highly varied: Slow relays won't see very many
buckets, but fast relays will. The damage to the network a very slow
relay can do by faking being only slightly faster is limited.
Nonetheless, for now we motivate the selection of bucket_size with a
slow relay:
- Assume a very slow relay of 1 Mbit/s capacity that will cheat 1 cell
in each bucket. Assume a 30 second measurement.
- The relay will handle 1*30 = 30 Mbit of traffic during the
measurement, or 3.75 MB, or 3.75 million bytes.
- Cells are 514 bytes. Approximately (e.g. ignoring TLS) 7300 cells
will be sent/recv over the course of the measurement.
- A bucket_size of 50 results in about 146 buckets over the course of
the 30s measurement.
- Therefore, the odds of the adversary cheating successfully as
(49/50)^(146), or about 5.2%.
This sounds high, but a relay capable of double the bandwidth (2 Mbit/s)
will have (49/50)^(2*146) or 0.2% odds of success, which is quite low.
Wanting a <1% chance that a 10 Mbit/s relay can successfully cheat
results in a bucket size of approximately 125:
- 10*30 = 300 Mbit of traffic during 30s measurement. 37.5 million
bytes.
- 37,500,000 bytes / 514 bytes/cell = ~73,000 cells
- bucket_size of 125 cells means 73,000 / 125 = 584 buckets
- (124/125)^(584) = 0.918% chance of successfully cheating
Slower relays can cheat more easily but the amount of extra weight they
can obtain is insignificant in absolute terms. Faster relays are
essentially unable to cheat.
3.3.2 Weight Inflation
Target relays are an active part of the measurement process; they know
they are getting measured. While a relay cannot fake the measurement
traffic, it can trivially stop transferring client background traffic
for the duration of the measurement yet claim it carried some. More
generally, there is no verification of the claimed amount of background
traffic during the measurement. The relay can claim whatever it wants,
but it will not be trusted above the ratio the FlashFlow deployment is
configured to know. This places an easy to understand, firm, and (if set
as we suggest) low cap on how much a relay can inflate its measured
capacity.
Consider a background/measurement ratio of 1/4, or 25%. Assume the relay
in question has a hard limit on capacity (e.g. from its NIC) of 100
Mbit/s. The relay is supposed to use up to 25% of its capacity for
background traffic and the remaining 75%+ capacity for measurement
traffic. Instead the relay ceases carrying background traffic, uses all
100 Mbit/s of capacity to handle measurement traffic, and reports ~33
Mbit/s of background traffic (33/133 = ~25%). FlashFlow would trust this
and consider the relay capable of 133 Mbit/s. (If the relay were to
report more than ~33 Mbit/s, FlashFlow limits it to just ~33 Mbit/s.)
With r=25%, FlashFlow only allows 1.33x weight inflation.
Prior work shows that Torflow allows weight inflation by a factor of 89x
[0] or even 177x [1].
The ratio chosen is a trade-off between impact on background traffic and
security: r=50% allows a relay to double its weight but won't impact
client traffic for relays with steady state throughput below 50%, while
r=10% allows a very low inflation factor but will cause throttling of
client traffic at far more relays. We suggest r=25% (and thus
1/(1-0.25)=1.33x inflation) for a reasonable trade-off between
performance and security.
It may be possible to catch relays performing this attack, especially if
they literally drop all background traffic during the measurement: have
the measurer (or some party on its behalf) create a regular stream
through the relay and measure the throughput on the stream
before/during/after the measurement. This can be explored longer term.
3.3.3 Incomplete Authentication
The short term FlashFlow implementation has the relay set two torrc
options if they would like to allow themselves to be measured: a flag
allowing measurement, and the list of coordinator TLS certificate that
are allowed to start a measurement.
The relay drops MSM_PARAMS cells from coordinators it does not trust,
and immediately closes the connection after that. A FF coordinator
cannot convince a relay to enter measurement mode unless the relay
trusts its TLS certificate.
A trusted coordinator specifies in the MSM_PARAMS cell the IP addresses
of the measurers the relay shall expect to connect to it shortly. The
target adds the measurer IP addresses to a whitelist in the DoS
connection limit system, exempting them from any configured connection
limit. If a measurer is behind a NAT, an adversary behind the same NAT
can DoS the relay's available sockets until the end of the measurement.
The adversary could also pretend to be the measurer. Such an adversary
could induce measurement failures and inaccuracies. (Note: the whitelist
is cleared after the measurement is over.)
4. FlashFlow measurement system: Medium term
The medium term deployment stage begins after FlashFlow has been
implemented and relays are starting to update to a version of Tor that
supports it.
We plan to host a FlashFlow deployment consisting of a FF coordinator
and a single FF measurer on a single 1 Gbit/s machine. Data produced by
this deployment will be made available (semi?) publicly, including both
v3bw files and intermediate results.
Any development changes needed during this time would go through
separate proposals.
5. FlashFlow measurement system: Long term
In the long term, finishing-touch development work will be done,
including adding better authentication and measurement scheduling, and
experiments will be run to determine the best way to integrate FlashFlow
into the Tor ecosystem.
Any development changes needed during this time would go through
separate proposals.
5.1 Authentication to Target Relay
Short term deployment already had FlashFlow coordinators using TLS
certificates when connecting to relays, but in the long term, directory
authorities will vote on the consensus parameter for which coordinators
should be allowed to perform measurements. The voting is done in the
same way they currently vote on recommended tor versions.
FlashFlow measurers will be updated to use TLS certificates when
connecting to relays too. FlashFlow coordinators will update the
contents of MSM_PARAMS cells to contain measurer TLS certificates
instead of IP addresses, and relays will update to expect this change.
5.2 Measurement Scheduling
Short term deployment only has one FF deployment running. Long term this
may no longer be the case because, for example, more than one directory
authority decides to adopt it and they each want to run their own
deployment. FF deployments will need to coordinate between themselves
to not measure the same relay at the same time, and to handle new relays
as they join during the middle of a measurement period (during the day).
The following is quoted from Section 4.3 of the FlashFlow paper.
To measure all relays in the network, the BWAuths periodically
determine the measurement schedule. The schedule determines when and
by whom a relay should be measured. We assume that the BWAuths have
sufficiently synchronized clocks to facilitate coordinating their
schedules. A measurement schedule is created for each measurement
period, the length p of which determines how often a relay is
measured. We use a measurement period of p = 24 hours.
To help avoid active denial-of-service attacks on targeted relays,
the measurement schedule is randomized and known only to the
BWAuths. Before the next measurement period starts, the BWAuths
collectively generate a random seed (e.g. using Tor’s
secure-randomness protocol). Each BWAuth can then locally determine
the shared schedule using pseudorandom bits extracted from that
seed. The algorithm to create the schedule considers each
measurement period to be divided into a sequence of t-second
measurement slots. For each old relay, slots for each BWAuth to
measure it are selected uniformly at random without replacement
from all slots in the period that have sufficient unallocated
measurement capacity to accommodate the measurement. When a new
relay appears, it is measured separately by each BWAuth in the first
slots with sufficient unallocated capacity. Note that this design
ensures that old relays will continue to be measured, with new
relays given secondary priority in the order they arrive.
5.3 Experiments
[XXX todo]
5.4 Other Changes/Investigations/Ideas
- How can FlashFlow data be used in a way that doesn't lead to poor load
balancing given the following items that lead to non-uniform client
behavior:
- Guards that high-traffic HSs choose (for 3 months at a time)
- Guard vs middle flag allocation issues
- New Guard nodes (Guardfraction)
- Exit policies other than default/all
- Directory activity
- Total onion service activity
- Super long-lived circuits
- Add a cell that the target relay sends to the coordinator indicating
its CPU and memory usage, whether it has a shortage of sockets, how
much bandwidth load it has been experiencing lately, etc. Use this
information to lower a relays weight, never increase.
- If FlashFlow and sbws work together (as opposed to FlashFlow replacing
sbws), consider logic for how much sbws can increase/decrease FF
results
- Coordination of multiple FlashFlow deployments: scheduling of
measurements, seeding schedule with shared random value.
- Other background/measurement traffic ratios. Dynamic? (known slow
relay => more allowed bg traffic?)
- Catching relays inflating their measured capacity by dropping
background traffic.
- What to do about co-located relays. Can they be detected reliably?
Should we just add a torrc option a la MyFamily for co-located relays?
- What is the explanation for dennis.jackson's scary graphs in this [2]
ticket? Was it because of the speed test? Why? Will FlashFlow produce
the same behavior?
6. Citations
[0] F. Thill. Hidden Service Tracking Detection and Bandwidth Cheating
in Tor Anonymity Network. Master’s thesis, Univ. Luxembourg, 2014.
[1] A. Johnson, R. Jansen, N. Hopper, A. Segal, and P. Syverson.
PeerFlow: Secure Load Balancing in Tor. Proceedings on Privacy
Enhancing Technologies (PoPETs), 2017(2), April 2017.
[2] Mike Perry: Graph onionperf and consensus information from Rob's
experiments https://trac.torproject.org/projects/tor/ticket/33076
[View Less]
On Fri, 17 Apr 2020 18:01:42 -0400
Nick Mathewson <nickm(a)freehaven.net> wrote:
> If you want to work on this it would be helpful to maybe start by
> listing (here or elsewhere) some places where you *don't* feel like
> you could (or would want to) write documentation: those would be a
> good target for devs who _have_ worked on Chutney before.
I came across Node.specialize() which does not seem to be called
elsewhere, and I cannot guess at its purpose. So, if anyone knows …
[View More]off
the top of their head what this is for (not what it does, because it's
quite obvious from the code itself) then I'd greatly appreciate and I
will document its purpose accordingly. It is one of the functions that
is listed as "to be called by the user".
Other than that I added docstrings where they were missing in TorNet
(useful both for Python's on-line help() and for checking whether code
agrees with intent) and made mental note of some areas I would like to
improve in the code (changing function names to better align with what
they actually do, figure out if some longer functions can be split into
more atomical and easier-to-digest parts, ...).
Caitlin
[View Less]
Filename: 315-update-dir-required-fields.txt
Title: Updating the list of fields required in directory documents
Author: Nick Mathewson
Created: 23 April 2020
Status: Open
1. Introduction
When we add a new field to a directory document, we must at first
describe it as "optional", since older Tor implementations will
not generate it. When those implementations are obsolete and
unsupported, however, we can safely describe those fields as
"required", since they are always included …
[View More]in practice.
Making fields required is not just a matter of bookkeeping: it
helps prevent bugs in two ways. First, it simplifies our code.
Second, it makes our code's requirements match our assumptions
about the network.
Here I'll describe a general policy for making fields required
when LTS versions become unsupported, and include a list of
fields that should become required today.
This document does not require to us to make all optional fields
required -- only those which we intend that all Tor instances
should always generate and expect.
When we speak of making a field "required", we are talking about
describing it as "required" in dir-spec.txt, so that any document
missing that field is no longer considered well-formed.
2. When fields should become required
We have three relevant kinds of directory documents: those
generated by relays, those generated by authorities, and those
generated by onion services.
Relays generate extrainfo documents and routerdesc documents.
For these, we can safely make a field required when it is always
generated by all relay versions that the authorities allow to
join the network. To avoid partitioning, authorities should
start requiring the field before any relays or clients do.
(If a relay field indicates the presence of a now-required
feature, then instead of making the field mandatory, we may
change the semantics so that the field is assumed to be
present. Later we can remove the option.)
Authorities generate authority certificates, votes, consensus
documents, and microdescriptors. For these, we can safely make a
field required once all authorities are generating it, and we are
confident that we do not plan to downgrade those authorities.
Onion services generate service descriptors. Because of the risk
of partitioning attacks, we should not make features in service
descriptors required without a phased process, described in the
following section.
2.1. Phased addition of onion service descriptor changes
Phase one: we add client and service support for the new field,
but have this support disabled by default. By default, services
should not generate the new field, and clients should not parse
it when it is present. This behavior is controlled by a pair of
network parameters. (If the feature is at all complex, the
network parameters should describe a _minimum version_ that
should enable the feature, so that we can later enable it only in
the versions where the feature is not buggy.)
During this phase, we can manually override the defaults on
particular clients and services to test the new field.
Phase two: authorities use the network parameters to enable the
client support and the service support. They should only do this
once enough clients and services have upgraded to a version that
supports the feature.
Phase three: once all versions that support the feature are
obsolete and unsupported, the feature may be marked as required
in the specifications, and the network parameters ignored.
Phase four: once all versions that used the network parameters
are obsolete and unsupported, authorities may stop including
those parameters in their votes.
3. Directory fields that should become required.
These fields in router descriptors should become required:
* identity-ed25519
* master-key-ed25519
* onion-key-crosscert
* ntor-onion-key
* ntor-onion-key-crosscert
* router-sig-ed25519
* proto
These fields in router descriptors should become "assumed present":
* hidden-service-dir
These fields in extra-info documents should become required:
* identity-ed25519
* router-sig-ed25519
The following fields in microdescriptors should become
required:
* ntor-onion-key
The following fields in votes and consensus documents should
become required:
* pr
[View Less]
Hi all,
We are investigating how Tor protects itself against Denial-of-Service
(DoS) attacks. So far, it has been difficult to find a comprehensive
top-level design document for the DoS subsystem (e.g., a torspec or
proposal) that reflects the decisions that lead to the subsystem in its
current form.
Specifically, we are looking at the DoS mitigation subsystem code for
entry guards at src/core/or/dos.{h,c} [1]. We are trying to understand
the chosen countermeasures and how the default and …
[View More]current consensus
values came to be, e.g., the decision to limit to 3 circuits per second
after the initial burst.
1) Could you kindly point us in the right direction if any such document
exists?
2) If it does not exist, would you mind briefly explaining how the DoS
threshold values (such as DoSCircuitCreationMinConnections,
DoSCircuitCreationRate, DoSCircuitCreationBurst, and
DoSConnectionMaxConcurrentCount) were chosen?
Thank you very much in advance.
Kind regards
Lennart Oldenburg
KU Leuven
[1] https://gitweb.torproject.org/tor.git/tree/src/core/or/dos.c
[View Less]
Hey guys!
So thanks to teor's insightful response yesterday I decided to try to run a
second tor relay (my middle node) on my private network.
Unfortunately, I can't do it with Chutney because my python client is
running on a windows machine. But I do have 3 machines at my disposal:
1. A windows machine (the python client)
2. A mac (the guard node)
3. Another mac (the middle node)
However, after connecting all three machines to my private LAN (meaning
they now all have local IP addresses), …
[View More]the EXTEND from the guard node to
the middle node fails.
When my guard node tries to connect to my middle node after receiving from
the client a RELAY_EXTEND cell, the guard node logs the following error:
Apr 28 17:00:31.000 [info] circuit_extend: Client asked me to extend to a
private address
Apr 28 17:00:31.000 [info] circuit_receive_relay_cell:
connection_edge_process_relay_cell (away from origin) failed.
Apr 28 17:00:31.000 [info] command_process_relay_cell:
circuit_receive_relay_cell (forward) failed. Closing.
So regarding this, I have two questions:
1. Is there a way for me to change something in my torrc file to override
this error and allow my relay to extend to private IP addresses?
My torrc is currently configured as such (Notice I put some place holders
for the drectories and for the ip address tha aren't actually whats written
there):
ContactInfo e <draftkingschaching(a)gmail.com>mail(a)example.com
ControlPort 9051
DataDirectory </path/to/data/dir>
ExitPolicy reject *:*
ExitRelay 0
GeoIPFile </path/to/geo/ip/file>
GeoIPv6File </path/to/geo/ipv6/file>
Log notice file <path/to/log/dirs/>/notice.log
Log debug file <path/to/log/dirs/>/debug.log
Log warn file <path/to/log/dirs/>/warn.log
Nickname vtoria
ORPort 443 NoAdvertise
ORPort Relay.Public.IP.Example:443 <http://79.183.54.194:443/> NoListen
SafeLogging 0
ExtendAllowPrivateAddresses 1
2. Would there maybe be a better way to run this private tor network
(without chutney)?
Thanks in advance for any answers!
Eli
[View Less]
Hello again everyone!
For anyone who doesn't know\remember, my name is Eli, I'm a high school
student from Israel and I'm currently trying to implement a TOR Client in
Python.
Currently, my project is configured so that my python client (OP) has its
guard node set as my local machine (which is running a downloaded version
of TOR). I do this for debugging purposes so that if I send a malformed
cell as the implemented client, I can read the debug log that the OR
generates and see what I did …
[View More]wrong.
Last time I sent a question to the dev-list I was stuck trying to get my
CREATE cells to work, and Nick Mathewson immediately answered my question
and helped me a lot! Thanks Nick!
I've made pretty good progress since then, and as of writing this, my
implemented OP can successfully send a CREATE cell and even an EXTEND
cell to my guard node.
However, I am running into a problem while trying to extend my circuit to
the third node (my exit node in this case).
After sending a RELAY_EXTEND cell to my middle node (which is encrypted
twice with my middle node and then guard node forward key using AES128 in
CTR mode), the Response I get back is a relay cell with the correct Circuit
Id and command (the command I get is the RELAY command which is represented
as 3), BUT the payload of the response cell is very weird. I am unable to
'recognize' it (as specified in section 6.1 of the tor-spec). and
furthermore, it does not seem to be any type of cell, it just seems like a
bunch of nonsense.
Seeing this I initially thought that the response cell I kept getting was a
RELAY_EXTENDED cell that I couldn't 'recognize' due to an error while
decrypting the cell payload.
But then I looked at the debug log of my guard node (remember that my guard
node is on my local machine) and it said that it had received a DESTROY
Cell back from the middle node and was passing on a RELAY_TRUNCATED cell to
me:
*Apr 26 16:11:03.014 [debug] command_process_destroy_cell: Received for
circID 2297363203.* <------this is the circuit ID between my gaurd node and
the middle node
*Apr 26 16:11:03.014 [debug] command_process_destroy_cell: Delivering
'truncated' back.Apr 26 16:11:03.014 [debug] relay_send_command_from_edge_:
delivering 9 cell backward.*
To my understanding what this log means is that some part of the EXTEND
cell I sent to the middle node was wrong or malformed and because of this
when the middle node tried to extend the circuit, an error occurred, and
the circuit needed to be torn down.
This is very weird because when I send an EXTEND cell that is meant for my
guard node (meaning I want to extend the circuit from one hop to two hops)
everything works fine, and I can even successfully derive the shared key
material for the middle node.
So I have several questions regarding this:
1. First of all, I didn't quite understand the exact format of a
RELAY_TRUNCATED cell. Does it contain a relay cell command +
recognized+field +digest and so on? or is it just a single octet that
immediately follows the cell command field? If some could show me an
example of the cell, it would be much appreciated...
2. What are some common errors that would make an OR drop a RELAY EXTEND
cell? I thought maybe it was a problem with my TAP handshake data, but
after extensive checking that doesn't seem to be the case.
3. If someone could describe the exact steps of extending a circuit to a
third node, it would greatly help me to make sure that I didn't miss a step
or do something wrong.
Thanks in advance for any answers, examples, or comments! I am having a lot
of fun doing this project so far and I hope to hear back from anyone who
has an answer :)
Regards,
Eli
[View Less]