[tor-commits] [torspec/master] Revise George's hidden-service statistics proposal.
nickm at torproject.org
nickm at torproject.org
Tue Jan 6 17:54:02 UTC 2015
commit 89282aba291d12a1a539606ebf02af3047bd61fa
Author: Karsten Loesing <karsten.loesing at gmx.net>
Date: Wed Nov 19 10:32:56 2014 +0100
Revise George's hidden-service statistics proposal.
---
proposals/238-hs-relay-stats.txt | 151 +++++++++++++++++++++++++++++++-------
1 file changed, 124 insertions(+), 27 deletions(-)
diff --git a/proposals/238-hs-relay-stats.txt b/proposals/238-hs-relay-stats.txt
index a081989..135048b 100644
--- a/proposals/238-hs-relay-stats.txt
+++ b/proposals/238-hs-relay-stats.txt
@@ -45,48 +45,145 @@ Status: Incomplete
2. Implementation
-2.1. Hidden service traffic statistics
-
- Tor HSDirs will add the following field to their extra-info
- descriptor:
-
- "hs-traffic" ... XXX
-
-2.2. HSDir hidden service counting
+2.0. Hidden service statistics interval
- Tor HSDirs will add the following field to their extra-info
- descriptor:
+ We want relays to report hidden-service statistics over a long-enough
+ time period to not put users at risk. Similar to other statistics, we
+ suggest a 24-hour statistics interval. All related statistics are
+ collected at the end of that interval and included in the next
+ extra-info descriptors published by the relay.
- "dirreq-v3-hsdir" key=val,... NL
- [At most once.]
+ Tor relays will add the following line to their extra-info descriptor:
- Statistics about HS directory activities.
- The current list of statistics is as follows:
+ "hidserv-stats-end" YYYY-MM-DD HH:MM:SS (NSEC s) NL
+ [At most once.]
- "hs-num": The approximate number of HSes that the HSDir is
- hosting descriptors for at the time the extra-info
- descriptor was created.
+ YYYY-MM-DD HH:MM:SS defines the end of the included measurement
+ interval of length NSEC seconds (86400 seconds by default).
+ A "hidserv-stats-end" line, as well as any other "hidserv-*" line,
+ is first added after the relay has been running for at least 24
+ hours.
- To derive this, HSDirs are expected to walk over their descriptor
- caches and count the number of HSes contained. The number is then
- obfuscated slightly by a small noise factor that introduces 10%
- inaccuracy.
-
- More specifically:
-
- hs-num = <number of HSes> * <random real \in [0.9, 1.1]>
+2.1. Hidden service traffic statistics
+ We want to learn how much of the total Tor network traffic is caused by
+ hidden service usage. There are three phases in the rendezvous
+ protocol where traffic is generated: (1) when hidden services make
+ themselves available in the network, (2) when clients open connections
+ to hidden services, and (3) when clients exchange application data with
+ hidden services. We expect (3) to consume most bytes here, so we're
+ focusing on this only. More precisely, we measure hidden service
+ traffic by counting RELAY cells seen on a rendezvous point after
+ receiving a RENDEZVOUS1 cell. These RELAY cells include commands to
+ open or close application streams, and they include application data.
+
+ Tor relays will add the following line to their extra-info descriptor:
+
+ "hidserv-rend-relayed-cells" SP num NL
+ [At most once.]
+
+ Approximate number of RELAY cells seen in either direction on a
+ circuit after receiving and successfully processing a RENDEZVOUS1
+ cell. The actual number observed by the directory is multiplied
+ with a random number in [0.9, 1.1] before being reported.
+
+ The keyword indicates that this line is part of hidden-service
+ statistics ("hidserv") and contains aggregate data from the relay
+ acting as rendezvous point ("rend").
+
+ We plan to extrapolate reported values to network totals by dividing
+ values by the probability of clients picking relays as rendezvous
+ point. This approach should become more precise on faster relays and
+ the more relays report these statistics.
+
+ We also plan to compare reported values with "cell-*" statistics to
+ learn what fraction of traffic can be attributed to hidden services.
+
+ Ideally, we'd be able to compare values to "write-history" and
+ "read-history" lines to compute similar fractions of traffic used for
+ hidden services. The goal would be to avoid enabling "cell-*"
+ statistics by default. In order for this to work we'll have to
+ multiply reported cell numbers with the default cell size of 512 bytes.
+2.2. HSDir hidden service counting
- time_t cutoff = now - REND_CACHE_MAX_AGE - REND_CACHE_MAX_SKEW;
+ We also want to learn how many hidden services exist in the network.
+ The best place to learn this is at hidden service directories where
+ hidden services publish their descriptors.
+
+ Tor relays will add the following line to their extra-info descriptor:
+
+ "hidserv-dir-published-ids" SP num NL
+ [At most once.]
+
+ Approximate number of unique hidden-service identities seen in
+ descriptors published to and accepted by this hidden-service
+ directory. The actual number observed by the directory is
+ multiplied with a random number in [0.9, 1.1] before being
+ reported.
+
+ This statistic requires keeping a separate data structure with unique
+ identities seen during the current statistics interval. We could, in
+ theory, have relays iterate over their descriptor caches when producing
+ the daily hidden-service statistics blurb. But it's unclear how
+ caching would affect results from such an approach, because descriptors
+ published at the start of the current statistics interval could already
+ have been removed, and descriptors published in the last statistics
+ interval could still be present. Keeping a separate data structure,
+ possibly even a probabilistic one, seems like the more accurate
+ approach.
+
+ We plan to extrapolate this value to network totals by calculating what
+ fraction of hidden-service identities this relay was supposed to see.
+ This extrapolation will be very rough, because each hidden-service
+ directory is only responsible for a tiny share of hidden-service
+ descriptors, and there is no way to increase that share significantly.
+
+ Here are some numbers: there are about 3000 directories, and each
+ descriptor is stored on three directories. So, each directory is
+ responsible for roughly 1/1000 of descriptor identifiers. There are
+ two replicas for each descriptor, and descriptor identifiers change
+ once per day. Hence, each descriptor is stored to four places in
+ identifier space throughout a 24-hour period. The probability of any
+ given directory to see a given hidden-service identity is
+ 1-(1-1/1000)^4 = 0.00399 = 1/250. This approximation constitutes an
+ upper threshold, because it assumes that services are running all day.
+ An extrapolation based on this formula will lead to undercounting the
+ total number of hidden services.
+
+ A possible inaccuracy in the estimation algorithm comes from the fact
+ that a relay may not be acting as hidden-service directory during the
+ full statistics interval. We suggest adding the following line to
+ handle this case better.
+
+ Tor relays also add the following line to their extra-info descriptor,
+ preceding any "hidserv-dir-*" lines:
+
+ "hidserv-dir-start" YYYY-MM-DD HH:00:00 NL
+ [At most once.]
+
+ YYYY-MM-DD HH:00:00 defines the first hour when this
+ hidden-service directory accepted either a publish or fetch
+ request for a hidden-service descriptor.
+
+ Finally, the intentionally added randomness leads to either under- or
+ overcounting hidden services by up to 10%.
3. Discussion
3.1. Count only RP cells? Or also IP cells?
+ As discussed on IRC, counting only RP cells should be fine for now.
+ Everything else is protocol overhead, which includes HSDir traffic,
+ IPo traffic, RPo traffic before the first RELAY cell, etc. We can
+ always be smarter later. -KL
3.2. Why obfuscation on HSDirs stats? And how much?
-
+ As discussed on IRC, maybe we should obfuscate small numbers more than
+ large numbers by adding a random number in [-20, 20]. Or we could
+ require a reporting threshold, if we can figure out how that cannot be
+ gamed by the adversary by making the required number of requests
+ themselves. Let's ask Aaron Johnson. -KL
[XXX]: guard discovery: https://lists.torproject.org/pipermail/tor-dev/2014-September/007474.html
More information about the tor-commits
mailing list