- network-health - lists.torproject.org

Guard fuzz detected
by trooned 04 Mar '21

04 Mar '21

Hi all, after restarting my bridge today, I got this warning message: 08:32:43 [WARN] Guard fuzz ($153DC150D6000C06E8FAB14B79BDD41B72ED2B21) is failing an extremely large amount of circuits. This could indicate a route manipulation attack, extreme network overload, or a bug. Success counts are 43/255. Use counts are 71/71. 249 circuits completed, 0 were unusable, 207 collapsed, and 2 timed out. For reference, your timeout cutoff is 60 seconds. Given the fact that this is the first time for me receiving a warning message like this, I don't really know what the correct procedure is or if this is useful at all for the network health team. Just thought, I'd let you know that something might be odd with this guard. I would be very happy if someone could clearify, whether I should keep an eye on those warnings or I can simply ignore them. Cheers! Trooned

1 0

New attack on the dir auths (Jan 27)
by Roger Dingledine 28 Jan '21

28 Jan '21

Hi folks, This afternoon a new traffic overload appeared, and this one looks definitely intentional. Somebody is trying to harm the Tor network. Maybe they're doing it to impact v3 onion services (because of the other bug, which is fixed but not everybody has the update) or maybe they're just trying to be jerks. We've missed several consensuses recently, and we should expect to miss more. I'm running a branch on moria1 that I think will work as a stopgap, but a majority of the dir auth operators are happily asleep right now, so on the theory that things won't burn down in the next hours, I'm going to let them finish sleeping and they can apply the patch on their Thursday. So expect things to remain uncomfortable for some hours yet. The fix as we currently imagine it will involve exit relays applying a patch too -- or if they can't patch easily, upgrading to a new upcoming version of Tor that has a fix in it. --Roger

1 1

v3 onion services unusable for a few hours today
by Roger Dingledine 28 Jan '21

28 Jan '21

Hi folks, The overload bugs on the directory authorities are continuing, but they also continue to be mostly manageable. But today, we spent several consensus intervals in a row without making a consensus -- which is supposed to be fine because Tor clients can do fine so long as they have a consensus from within the last 24 hours, but it exposed a bug in the v3 onion service implementation, where v3 onion clients and v3 onion services choose not to even attempt to fetch or publish v3 onion descriptors: Jan 10 11:32:24.364 [warn] No live consensus so we can't get the responsible hidden service directories. The fix imo is that we need to make v3 onion services work under the same time assumptions as the other parts of Tor -- if you have a usable consensus, then use it. I've opened this ticket for the network team: https://gitlab.torproject.org/tpo/core/tor/-/issues/40237 --Roger

1 1

Metrics Portal - REST API
by David Goulet 23 Nov '20

23 Nov '20

Hi Karsten! (Metrics and health team CCed) Network team has been working on a new "MetricsPort"[1] in tor which can expose counters of different metrics within "tor". It currently uses the Prometheus model [2] which then allows us to create proper monitoring graphs using tools like Grafana (see some example screenshots in #40063). The short term goal here also is to provide Grafana templates for monitoring a relay or onion service so people can just download them automatically from their marketplace and are ready to go. Then that made us think, what if we could have something similar on metrics.torproject.org. A page that we could query like "/prometheus" that would just give us a set of counters of the current state of the network. A bit like a REST API but less "API-issh". I do recall having seen at one point a REST API item on the metrics roadmap but I'm not entirely sure about my memory hence why I'm probing you about this. Likely at first, what such a page would expose is not different from what metrics has at the moment _but_ the difference is that it would allow anyone (most importantly us) to be able to aggregate visualization in one dashboard using latest visualization tech (Grafana for instance). This kind of page can usually handle thousands of requests a second without blinking so the load impact should be minimal since this is exposing an already existing state to the world rather than querying a state (like I assume Onionoo does?). Maybe the solution here could be to instead write an "exporter" that queries Onionoo and formats it nicely for a Prometheus server but I do fear the load that it could put on Onionoo if let say A LOT of metrics are queried every 5 seconds or so? The other thing is maybe the exporter idea is better, unclear, if we want to be more agile at integrating other types of metrics like let say monitoring the consensus like consensus-health does or extracting different data from extra-info. Thoughts? Cheers! David [1] https://gitlab.torproject.org/tpo/core/tor/-/issues/40063 [2] https://prometheus.io/docs/concepts/data_model/ -- 6/YYHbjbLDDZ9bG5utUI1Ts1W+SmKf0zyZc8rfr0FXY=

2 1

West Coast US Tor Relays Slow - Low Consensus Weight versus East Coast
by Neel Chauhan 06 Aug '20

06 Aug '20

Hi network-health@/tor-relays@ mailing lists, I noticed one thing: Tor relays on the West Coast US (and Canada) are generally slower than those on say the East Coast and in Europe. I moved to the West Coast this January, but this was not an issue in the past when looking at dedicated servers I had from the West Coast prior to this year. These are two middle relay instances on an Gigabit FTTH connection (Wave G in Redmond, WA): https://metrics.torproject.org/rs.html#details/B0F9BA27944FA59E3B1A182208FF… https://metrics.torproject.org/rs.html#details/DB710B14D7329B7289CFCC547F48… The Consensus Weight often is slower than the Advertised Bandwidth, and isn't ramping up despite having lots of bandwidth. I set RelayBandwidthRate to about 500 Mbps on each instance. This isn't just a Wave G problem, this affects almost every relay in the West Coast including other Gigabit ISPs such as AT&T (AS7018) and Sonic (AS46375), as well as hosting companies and colocation facilities. For instance, my Los Angeles-based Exit relays at Psychz Networks (AS40676) show this issue (however they are new): https://metrics.torproject.org/rs.html#details/156AAC3FAD1ACC8906316519DCB4… https://metrics.torproject.org/rs.html#details/A69CEB30328B1E85C6B167FECAF2… Even Emerald Onion has this issue: https://metrics.torproject.org/rs.html#details/09DCA3360179C6C8A5A20DDDE1C5… What is causing this issue and is there a solution? Is it backbone congestion due to COVID-19? The high load on dirauths? sbws regressions? Can I help fix this issue? I am a Core Tor contributor and am open to also working on sbws. Is there a way to optimize my relays (they run FreeBSD). -Neel === https://www.neelc.org/

1 0

No network health meeting today 27 April 2020
by Georg Koppen 27 Apr '20

27 Apr '20

Hi! Sorry for the late reply but we won't have a network health meeting this week. I am aiming to do a sync on Monday, May 4 2020 again at our usual time (1800 UTC). Georg

1 0

Status update for our bad relay processes
by Georg Koppen 08 Apr '20

08 Apr '20

Hi! I just sent a status update about our bad relays work to tor-relays[1]. It's focused on relay operators which is why I chose that list but, in case you are not reading it, I thought sending at least a note to this list as well as that mail essentially concerns network health questions (too). Georg [1] https://lists.torproject.org/pipermail/tor-relays/2020-April/018356.html

1 0

Next weekly meeting 20 April 2020
by Georg Koppen 08 Apr '20

08 Apr '20

Hello! I forgot to talk about it in our weekly sync earlier this week but as next Monday is a public holiday in a bunch of jurisdictions (Easter Monday) we'll skip that meeting. The next one will be held on 20 April same time, same place. See you there, Georg

1 0

Upgrade your 'tarragon' Tor relay?
by Roger Dingledine 06 Apr '20

06 Apr '20

Hi William, You are running a Tor relay, which is great: http://rougmnvswfsmd4dq.onion/rs.html#details/1D4AB6248E3D7DF9C4FBE40DCFAA3… First: that Tor version is obsolete, and because of old bugs, we will soon cut relays running those versions out of the network. Please consider upgrading! Second: let us know if there is any way to help you bump up the bandwidth that you are allocating to the Tor relay. :) You can find Tor packages for your distro / OS here: https://trac.torproject.org/projects/tor/wiki/TorRelayGuide#Platformspecifi… Ideally you will switch to keeping up with our stable releases, but if you need a stable that is especially stable, the Tor 0.3.5 branch will be maintained until Feb 2022: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkTeam/CoreTor… and you can see the lifetimes of other Tor versions on that table too. Let us know if we can do anything to make the process easier. And lastly, I am cc'ing the new network health mailing list (which has public archives), to help us stay synced: https://trac.torproject.org/projects/tor/wiki/org/teams/NetworkHealthTeam Thanks! --Roger

3 2

Re: [network-health] [tor-relays] Low bandwidth
by teor 01 Apr '20

01 Apr '20

Hi, I'm cc'ing the network health team, so they are aware of this issue. >>> On 30 Mar 2020, at 23:00, ha3ks ha3ks(a)protonmail.com wrote: >>> my relay dropped nearly all it’s bandwidth about a month ago, checking over it I can see 3 of the dir auths have given it a low consensus weighting, would that cause low bandwidth numbers? >>> I checked my ufw in Ubuntu and it’s allowing 443 and 80. >> >> Date: Tue, 31 Mar 2020 09:44:41 +1000 >> From: teor teor(a)riseup.net >> >> I have seen similar reports from a few other relay operators. >> >> What is your relay fingerprint? >> >> Have you followed the troubleshooting instructions here: >> https://trac.torproject.org/projects/tor/wiki/doc/MyRelayIsSlow > > On 31 Mar 2020, at 20:46, ha3ks <ha3ks(a)protonmail.com> wrote: > > The fingerprint is - 6E1DA4C0B0C05FB721B42329C47A20DA22908AEB > > I have followed some of this yes, though its a little over my head I have opened a ticket in sbws to follow up this issue: sbws measures some relays 100x lower than Torflow https://trac.torproject.org/projects/tor/ticket/33775 It could be related to these other issues: sbws does not detect changes in descriptor bandwidth values https://trac.torproject.org/projects/tor/ticket/30733 sbws bandwidth scans should require a minimum exit bandwidth https://trac.torproject.org/projects/tor/ticket/33009 T

2 2