- network-health - lists.torproject.org

Queries about user statistic on Tor Metrics
by Nowshin, Nafisa 22 Sep '21

22 Sep '21

Hello, I am a graduate student at Oregon State University doing research on digital privacy and security. My advisor is Dr. Glencora Borradaile. We are trying to use tor metrics data in order to understand the reason why people are connecting to tor and if it is because of some external motivation (the topic of our research) or because they are regular users. Looking at the user statistics available in tor metrics website, we had a few questions we wanted to clarify. 1. How frequently tor browsers send out update pings? Is there a fixed frequency or do they vary over time? 2. Are update pings sent every time a user opens their browser or only when new updates are pushed to the tor browser? Thank you for your time. Regards Nafisa Nowshin Graduate Assistant School of EECS, Oregon State University

3 2

merging with metrics-team@ mailing list
by Gaba 10 Sep '21

10 Sep '21

Hi people! We just moved everyone that was subscribed to metrics-team@ mailing list to this network-health@ one. Welcome everybody! This new team is maintaining all the metrics software that we run and is responsible of the state of the network. You can read more about it at http://eweiibe6tdjsdprb4px6rqrzzcsi22m4koia44kc5pcjr7nec2rlxyad.onion/tpo/n… cheers, gaba -- pronouns she/her/they GPG Fingerprint EE3F DF5C AD91 643C 21BE 8370 180D B06C 59CA BD19

1 0

'margot like' feature request
by Roger Dingledine 02 Apr '21

02 Apr '21

Hi David! A feature request based on (what I understand of) your hackweek project: I often find myself in the position where I have a set of fingerprints, and I want to know if I've "got them all". So something that inputs a set of fingerprints, and outputs an ordered list of candidate "similar" relays, would be a great tool. What exactly we mean by similar, of course, will be a thing we need to iterate on. If we had a huge dataset -- with ground truth -- we could machine learning it and see how the black box gods answer. But while that thought leads to "good idea, let's collect that data set", we need something else in the mean time. Here's a start: * Component-wise similarity is computed differently for each component (version/contactinfo/nickname is a string comparison, IP address is looking for other relays on the same /x network, uptime is computing the real uptime based on the descriptor timestamp minus the descriptor uptime, etc), * But the *strength* of a match is based on how few other relays also matched. That is, we find a component match less compelling the more relays there are that match. For example, maybe the /16 address similarity is a weak match, because there are many relays on that same /16, but the /24 address similarity is a strong match because there's only one other relay closeby. * We can start out with simple comparisons, but eventually we'll want fuzzy comparisons, where e.g. we look at the Levenshtein distance for nickname ("Anonymous1, Anonymous2, Anonymous3"), or check if the two uptimes are within a threshold percentage of each other, etc. * And then the overall similarity is a function of the strength of each component-wise similarity. To start, we could just have it be a linear combination of each, with equal weighting, which means the more components have a strong match, the greater the overall similarity score. There's a lot of opportunity for improvement here, like, weighting certain components more, or non-linear weights where e.g. it's especially interesting when both bandwidthrate and bandwidthburst are strong matches. * So that's how we decide if *two* relays are similar to each other. But if I give you *ten* relays, we need a way to figure out what those relays have in common, to decide whether an eleventh relay is related to the group as a whole. The easy version here is to just do pairwise comparisons between each input relay and each consensus relay, and then sum the overall similarities into a score for each candidate. This approach should *implicitly* take into account similarity between input relays, on the theory that if A is similar to C and B is similar to C, then A is probably similar to B too. For extra credit, we could do some smarter combination than just summing them, but I don't have anything in mind yet. For comparison, check out phw's trnnr tool from the SybilHunter paper: https://github.com/NullHypothesis/trnnr/blob/master/trnnr.py which does a simplified version of the above (it only takes in one relay, it doesn't do the "strength of match is inversely proportional to how much of the population matches" part, and also it has a slick way of comparing all the components at once by just turning them all into a big unified string, which is simple but also has bizarre effects: every character in the IP address string is equally important, and the importance of each component is a function of how many characters it takes to represent it, which is...weird. :) But in general, this script is a great start to look at first. --Roger

1 0

Relays that can't extend
by Roger Dingledine 31 Mar '21

31 Mar '21

Hi folks, I've been probing which relays can't extend to my relay: https://metrics.torproject.org/rs.html#details/7B35DB92BA72BA0BBFD51B35B11A… I'm simply making two-hop paths from my client through every relay to my destination relay. Here's a snapshot of some relays that fail most/all of the extend attempts. The first data set is Feb 27 through Mar 1, and I've attached it as feb27-mar1.txt. The second data set is Mar 30, and it's attached as mar30.txt. The way to read each line of the ratio is: number of failed extend attempts / (number of failed + number of succeeded). (There are some edge cases where I failed to reach the relay as the first hop, for example because it has gone down, or it's in the new consensus but my Tor client doesn't have the descriptor for it yet. I've omitted those edge cases from the ratio calculation, since they're not failures and they're not successes so they just muddy the analysis.) Then I combined them: cat feb27-mar1.txt mar30.txt |sort > combined.txt to make it easy to see which relays had problems in both sets. I've started mailing some of the relay operators to ask them to investigate. (Many of them alas have no workable contactinfo.) The most likely explanation is that they're out of file descriptors (in which case their Tor logs should be complaining constantly). A backup explanation might be that their relay is censored from reaching my relay, perhaps by destination port or address. Eventually we'll want to use these results to compare against the self-reported "overload" metrics from Proposal 328 ("Make Relays Report When They Are Overloaded". I'm not sure what the step after that is, but maybe it will be reducing the consensus weights for relays that aren't performing well. --Roger

2 3

estimating the carrying capacity for Tor relays
by bentham＠danwin1210.me 19 Mar '21

19 Mar '21

Dear all, Over the past six years or so, the total population of Tor relays has capped out around 7,000, even as the total data carried by the Tor network has increased fourfold. This is not great, since it suggests an intrinsic limit to the number of Tor relays: a 'carrying capacity', in the sense of population dynamics. It is worth understanding the limiting factor, and there are several valid hypotheses. One is that the number of relays is determined by something extrinsic, for example, not enough people volunteering to run Tor relays to support continued growth. However, the increasing competitiveness for relays and de facto minimum requirements for entering the Tor consensus would seem to weaken this argument. Also, if this were true, it would seem sensible to expect that the number of Tor relays would be a random walk: sometimes new people choose to run relays, and sometimes people choose to stop running relays, and it just so happens that we've reached an equilibrium, with an OU process around that equilibrium point. An alternative hypothesis is that there is some intrinsic phenomenon that is limiting the number of relays: that for every relay that is added to the consensus, it is more difficult for the population of relays to increase further. We can test this hypothesis by estimating the Hurst exponent for the number of relays over time, using the hourly observations in the consensus records on collector.torproject.org. We can also drill into this question further: Is the increase in the number of relays limited by something that is making it harder for new relays to enter the consensus, is it limited by something that is making it more likely that existing relays would be removed from the consensus, or neither? We can explore this by examining the churn as a function of the total population. If churn as a proportion of the total population increases with the size of the population, then it is harder for new relays to enter the consensus as the population increases; if the churn decreases with the size of the population, then it is more likely for existing relays to fall out of the consensus as the population increases. We explore both questions below; see [code] and [output]. It turns out that the Hurst exponent is consistently about 0.3 for each of the past six years, suggesting that the reversion is statistically significant at the three-sigma level. So, it is not a random walk: with every relay that enters the consensus, it is harder for the next relay to enter the consensus. This observation supports the argument that the limit to the number of Tor relays is endogenous, not exogenous. We also examined the churn as a function of population size. It turns out that the relationship between churn and population size is much weaker. There is a small positive effect of population size on churn rate, although our analysis suggests that population size explains only about 6% of the churn rate and is not a particularly important factor. Also, the effect has not been consistent across years; for example, in 2018, the effect was reversed. We can conclude that the effect of population size on churn is small. So what is causing it to be difficult for new routers to be added to the consensus? Personally, I am partial to the conjecture that the cause is the number of pairwise TCP connections that the network must maintain to continue functioning. Every router must connect to every other router, so the n^2 complexity seems like an obvious limit to scalability. But this is not the only possible explanation. Perhaps there is some limit to the consensus process wherein the cadence of successive consensuses is simply too fast to accommodate a sufficiently large number of routers. Or maybe something about the way relays are evaluated or selected by clients tends toward approving only a certain number. Suggest that we explore this further. Suggest that if we want to see 10,000 Tor relays some day, we will need to identify the limiting factor first. Jeremy Bentham [code] #!/usr/bin/python3 import hurst import pandas import matplotlib from numpy import arange from scipy import stats matplotlib.use('agg') import matplotlib.pyplot as plt def desc(sdf, name): # Hurst exponent n = len(sdf) mu = sdf['n'].mean() try: H, c, val = hurst.compute_Hc(sdf['n']) print("Hurst exponent for %s (n=%d, mu=%f): %0.6f" % (name, n, mu, H)) except: pass # churn scatter plot x = sdf['n'] y = list(map(lambda x: float(x)/2, sdf['in']+sdf['out'])) x, y = zip(*filter( lambda i: i[0] > 5000 and i[1] < 1200, zip(x, y) )) y = list(map(lambda a, b: b/a, x, y)) slope, intercept, r_value, p_value, std_err = stats.linregress(x, y) print("slope=%s, intercept=%s, r_value=%s, p_value=%s, std_err=%s\n" % ( slope, intercept, r_value, p_value, std_err )) slope, intercept, r_value, p_value, std_err = stats.linregress(x, y) line_x = arange(min(x), max(x)) line_y = slope*line_x+intercept plt.figure(figsize=(10,10)) plt.scatter(x, y, c=range(len(x)), cmap='cool') plt.plot( line_x, line_y, label='$%.2fx + %.2f$, $R^2=%.2f$' % (slope, intercept, r_value**2) ) plt.legend(loc='best') plt.title("Tor Relay Churn %s" % name) plt.xlabel("Relay Population") plt.ylabel("Hourly Churn") plt.tight_layout() plt.savefig("scatter-%s.png" % name) plt.clf() plt.close() df = pandas.read_csv("delta.csv", names=['dt', 'n', 'in', 'out'], index_col=0) for y in range(2015, 2022): sdf = df["%d-01-01" % y:"%d-01-01" % (y+1)] desc(sdf, y) print("----") sdf = df["2015-01-01":] desc(sdf, "2015-2021") [output] Hurst exponent for 2015 (n=8751, mu=6688.496400): 0.292922 slope=6.828983625513537e-06, intercept=0.03150033656688232, r_value=0.24326729749329457, p_value=4.610820027076726e-118, std_err=2.911196095555647e-07 Hurst exponent for 2016 (n=8773, mu=7118.746153): 0.280950 slope=6.52665957746843e-06, intercept=0.026579771388178977, r_value=0.13035103343825263, p_value=1.5008838134149767e-34, std_err=5.300969996645152e-07 Hurst exponent for 2017 (n=8756, mu=6983.947579): 0.363264 slope=-5.6911267269861e-06, intercept=0.11128434282175681, r_value=-0.3197803122785664, p_value=2.4559989048776613e-207, std_err=1.802368096605417e-07 Hurst exponent for 2018 (n=8760, mu=6340.653425): 0.273098 slope=-1.8991031221395063e-05, intercept=0.19279021686921058, r_value=-0.4143375031745955, p_value=0.0, std_err=4.45750735830525e-07 Hurst exponent for 2019 (n=8754, mu=6422.183459): 0.281159 slope=2.4396305737493384e-06, intercept=0.05420159095635396, r_value=0.1028780872185714, p_value=4.925103966921193e-22, std_err=2.5213725374993213e-07 Hurst exponent for 2020 (n=8780, mu=6315.970046): 0.253770 slope=3.773262957205024e-06, intercept=0.04167914097261437, r_value=0.14185475596231883, p_value=5.692359499532742e-39, std_err=2.874775311046666e-07 Hurst exponent for 2021 (n=1574, mu=6921.173443): 0.361722 slope=-4.583624673579448e-07, intercept=0.07325673164319593, r_value=-0.008646008325663415, p_value=0.7324417790859385, std_err=1.3404752250927981e-06 ---- Hurst exponent for 2015-2021 (n=54148, mu=6653.012096): 0.301609 slope=1.2316718714554112e-06, intercept=0.06349668094844796, r_value=0.06418146747221241, p_value=3.5734133468741144e-50, std_err=8.260683313218318e-08

1 1

JARM fingerprinting of Tor nodes
by Corl3ss 05 Mar '21

05 Mar '21

Hi, On an original idea of jvoisin, we have been working on fingerprinting Tor nodes with JARM. Here is a short description of this experimental work : https://hackmd.io/TWiUy4knQ06SYk9RBxnXPQ?view We share it here after a short talk with GeKo. The aim is to : * share technical opinions on these results * evaluate the interest to go further, e.g. using JARM fingerprinting for network health issues If you have now read what is on the link, you could have questions as GeKo did so here some complementary thoughts: * how would you detect bad configuration/behavior? As the fingerprint only tell us what configuration is shared between Tor nodes, we made then a packet analysis to explain the differences and detect potential misconfiguration/misbehaviour. We haven't detected anything suspicious yet just: .some rare / odd configurations (see link) .the fact that some rare fingerprints have gone offline fast, so they were perhaps misconfigured/suspicious. It was too late to make a packet analysis on those. * should we have uniform fingerprints? The first 30 digits of the fingerprints depend on TLS version answer and used ciphers. We have only 13 such fingerprints on more than 7.000 tested relays. It seems finally pretty uniform. I think it could be used to watch if nodes have an odd fingerprint and give an alert in such case. If useful. * do we know what actually causes fingerprints to change? Yes, as said above (TLS version and ciphers). For a detailled comparison, full results of the packet analysis are available on the link above. Fingerprints are not OS-specific, nor Tor version-specific. I would assume specific of (open|libre)ssl mainly. Open questions: * fingerprint diversity seems normal to you in regard of the Tor TLS implementation ? * do you see any problem / dangerous behaviour in packet analysis ? * usefulness for a network health monitoring ? * ... Would read your feedback with interest ! Corl3ss

2 1

Guard fuzz detected
by trooned 04 Mar '21

04 Mar '21

Hi all, after restarting my bridge today, I got this warning message: 08:32:43 [WARN] Guard fuzz ($153DC150D6000C06E8FAB14B79BDD41B72ED2B21) is failing an extremely large amount of circuits. This could indicate a route manipulation attack, extreme network overload, or a bug. Success counts are 43/255. Use counts are 71/71. 249 circuits completed, 0 were unusable, 207 collapsed, and 2 timed out. For reference, your timeout cutoff is 60 seconds. Given the fact that this is the first time for me receiving a warning message like this, I don't really know what the correct procedure is or if this is useful at all for the network health team. Just thought, I'd let you know that something might be odd with this guard. I would be very happy if someone could clearify, whether I should keep an eye on those warnings or I can simply ignore them. Cheers! Trooned

1 0

New attack on the dir auths (Jan 27)
by Roger Dingledine 28 Jan '21

28 Jan '21

Hi folks, This afternoon a new traffic overload appeared, and this one looks definitely intentional. Somebody is trying to harm the Tor network. Maybe they're doing it to impact v3 onion services (because of the other bug, which is fixed but not everybody has the update) or maybe they're just trying to be jerks. We've missed several consensuses recently, and we should expect to miss more. I'm running a branch on moria1 that I think will work as a stopgap, but a majority of the dir auth operators are happily asleep right now, so on the theory that things won't burn down in the next hours, I'm going to let them finish sleeping and they can apply the patch on their Thursday. So expect things to remain uncomfortable for some hours yet. The fix as we currently imagine it will involve exit relays applying a patch too -- or if they can't patch easily, upgrading to a new upcoming version of Tor that has a fix in it. --Roger

1 1

v3 onion services unusable for a few hours today
by Roger Dingledine 27 Jan '21

27 Jan '21

Hi folks, The overload bugs on the directory authorities are continuing, but they also continue to be mostly manageable. But today, we spent several consensus intervals in a row without making a consensus -- which is supposed to be fine because Tor clients can do fine so long as they have a consensus from within the last 24 hours, but it exposed a bug in the v3 onion service implementation, where v3 onion clients and v3 onion services choose not to even attempt to fetch or publish v3 onion descriptors: Jan 10 11:32:24.364 [warn] No live consensus so we can't get the responsible hidden service directories. The fix imo is that we need to make v3 onion services work under the same time assumptions as the other parts of Tor -- if you have a usable consensus, then use it. I've opened this ticket for the network team: https://gitlab.torproject.org/tpo/core/tor/-/issues/40237 --Roger

1 1

Metrics Portal - REST API
by David Goulet 23 Nov '20

23 Nov '20

Hi Karsten! (Metrics and health team CCed) Network team has been working on a new "MetricsPort"[1] in tor which can expose counters of different metrics within "tor". It currently uses the Prometheus model [2] which then allows us to create proper monitoring graphs using tools like Grafana (see some example screenshots in #40063). The short term goal here also is to provide Grafana templates for monitoring a relay or onion service so people can just download them automatically from their marketplace and are ready to go. Then that made us think, what if we could have something similar on metrics.torproject.org. A page that we could query like "/prometheus" that would just give us a set of counters of the current state of the network. A bit like a REST API but less "API-issh". I do recall having seen at one point a REST API item on the metrics roadmap but I'm not entirely sure about my memory hence why I'm probing you about this. Likely at first, what such a page would expose is not different from what metrics has at the moment _but_ the difference is that it would allow anyone (most importantly us) to be able to aggregate visualization in one dashboard using latest visualization tech (Grafana for instance). This kind of page can usually handle thousands of requests a second without blinking so the load impact should be minimal since this is exposing an already existing state to the world rather than querying a state (like I assume Onionoo does?). Maybe the solution here could be to instead write an "exporter" that queries Onionoo and formats it nicely for a Prometheus server but I do fear the load that it could put on Onionoo if let say A LOT of metrics are queried every 5 seconds or so? The other thing is maybe the exporter idea is better, unclear, if we want to be more agile at integrating other types of metrics like let say monitoring the consensus like consensus-health does or extracting different data from extra-info. Thoughts? Cheers! David [1] https://gitlab.torproject.org/tpo/core/tor/-/issues/40063 [2] https://prometheus.io/docs/concepts/data_model/ -- 6/YYHbjbLDDZ9bG5utUI1Ts1W+SmKf0zyZc8rfr0FXY=

2 1