tor-relays

tor-relays@lists.torproject.org

42 participants
4821 discussions

My bridge has no distribution mechanism
by Keifer Bly 30 Oct '22

30 Oct '22

https://metrics.torproject.org/rs.html#details/4D6E3CA2110FC36D3106C86940A1… Hey, So my bridge, listed above, says it has no distribution mechanism. Does this mean it is not currently being used? Thank you. --Keifer

1 0

DoS attacks -- status update
by Georg Koppen 28 Oct '22

28 Oct '22

Hello everyone! It's been a while since we last provided some update on this mailing list about our ongoing work fighting several DoS attacks. We can use the attached graph about detected overload over the last couple of months to show what is going on and what we do/plan to do about it. The first noteworthy incident on that graph is the sharp rise in overloaded non-exit nodes since the middle of July caused by a drop of onionskins[1] which the relay's available CPU/memory can't handle anymore. There are currently two ideas we are working on to cope with such a flood of onionskins: 1. Developing a Proof of Work (PoW) system to has a rate-limitation knob rejecting the load of onionskins while letting legitimate ones through.[2] We still need to solve some design issues (feel free to help!) but hope to have that feature integrated into Tor soon. 2. Relay operators started to experiment with iptables/nftables rules and having the right ones available might be a good stopgap approach against the onionskin related DoS. We are coordinating that effort[3], so we have something available to propose to the wider community, which is kept up-to-date and limits the risks of traffic overblocking. Feel free to help as well with that effort. The other noteworthy incident started around September 13 when exit nodes began to get overloaded (while the other DoS was and is still ongoing). Unfortunately, that exit related DoS is heavily impacting our users' experience as can be seen in our OnionPerf data[4]. While we are still investigating the nature of that DoS attack it turns out that blocking particular IP addresses with ExitPolicy rules seems to help on exit nodes this is tested on. The Artikel10 exit node operators provided even a script recently[5] to help with that (much appreciated, thanks!). This approach is highly experimental at this point and it might help us at least to come up with an actual design idea to counter that particular exit DoS. Thanks, Georg [1] For information about overload in general and what "drop of onionskins" means, see: https://support.torproject.org/relay-operators/#relay-bridge-overloaded. It contains as well a guide on how to enable MetricsPort monitoring yourself so you see the actual metrics of your own relay. [2] https://gitlab.torproject.org/tpo/core/tor/-/issues/40634 [3] https://gitlab.torproject.org/tpo/community/support/-/issues/40093 [4] https://metrics.torproject.org/torperf.html [5] https://lists.torproject.org/pipermail/tor-relays/2022-October/020848.html

1 0

MetricsPort: tor_relay_connections_total type confusion
by nat＠danwin1210.de 28 Oct '22

28 Oct '22

Hello David, again, thanks for your work on adding more metrics to tor's MetricsPort! Many relay operators will love this and documentation will be useful [1]. I reported https://gitlab.torproject.org/tpo/core/tor/-/issues/40699 which got closed yesterday, but there was likely a misunderstanding and the changes did not solve the underlying issue. The core issue is: The metric called tor_relay_connections(_total) [2][3] contains a mix of different types (gauge, counter). When mixing types in a single metric, the TYPE definition will always be wrong for one or the other value. For example grafana will show this if you use a counter metric without rate(): "Selected metric is a counter. Consider calculating rate of counter by adding rate()." It is a best practice to avoid mixing different types in a single metric. >From the prometheus best practices [4]: "As a rule of thumb, either the sum() or the avg() over all dimensions of a given metric should be meaningful (though not necessarily useful). If it is not meaningful, split the data up into multiple metrics. For example, having the capacity of various queues in one metric is good, while mixing the capacity of a queue with the current number of elements in the queue is not." An idea to address the underlying issue: One connection metric for counter and one for gauge: - tor_relay_connections_total for counters, like the current label state="created" - tor_relay_connections for gauge metrics, like the current label state="opened". "rejected" also appears to be a gauge metric. Another nice feature of these metrics would be to have a label for what type of system is connecting (src="relay", src="non-relay") - more on that in yesterday's email. A tool by toralf [4] also shows these and uses the source IP but tor itself does not need to look at the source IP to determine the type, something discussed in last week's relay operator meetup. best regards, nat [1] https://gitlab.torproject.org/tpo/web/support/-/issues/312 [2] https://gitlab.torproject.org/tpo/core/tor/-/commit/06a26f18727d3831339c138… [3] https://gitlab.torproject.org/tpo/core/tor/-/commit/6d40e980fb149549bbef5d9… [4] https://prometheus.io/docs/practices/naming/

2 2

Upcoming OpenSSL 3 security bugfix release
by Georg Koppen 28 Oct '22

28 Oct '22

Hello relay operators! If you are running OpenSSL 3 please be aware that you might need to upgrade it to 3.0.7 as fast as possible on next Tuesday (Nov 1). In an announcement[1] it got said 3.0.7 fixes a CRITICAL security issue. Operators running on the 1.1 series are fine as far as we know so far. Georg [1] https://www.openwall.com/lists/oss-security/2022/10/25/4

1 0

(Invitation) Join us! Tor Relay Operator Meetup - October 22, 2022 - 1900 UTC
by gus 24 Oct '22

24 Oct '22

Dear relay operators, The next Tor relay operator meetup will happen on Saturday, October 22 at 19.00 UTC. ## Where BigBlueButton: https://tor.meet.coop/gus-og0-x74-dzn ## Agenda (WIP) * Announcements * Tor and Snowflake in Iran * EOL rejection * State of DoS attack * New network health project * Q&A * Next Tor Relay Operator Meetup Meeting pad: https://pad.riseup.net/p/tor-relay-op-meetup-o22-keep Everyone is free to bring up additional questions or topics at the meeting itself. ## Registration No need for a registration or anything else, just use the room-link above. We will open the room 10 minutes before so you can test your mic setup. Please share with your friends, social media and other mailing lists! cheers, Gus -- The Tor Project Community Team Lead

4 5

Performance issues/DoS from outgoing Exit connections
by Alexander Dietrich 22 Oct '22

22 Oct '22

Hello, on the evening of 2022-10-18, we (Artikel10) started getting alerts about our Tor servers, while our traffic declined sharply. When we investigated, we found that there were hundreds of thousands of TCP connections (per server) open to a single address, orders of magnitude more than any other address. We blocked this address via "ExitPolicy reject", then another one, and since then things seem to have improved. I have thrown together a small Python script to detect this and generate "ExitPolicy reject" lines automatically: https://github.com/artikel10/surgeprotector This is still experimental, so if you decide to give the script a try, please keep an eye on it. Kind regards, Alexander -- PGP Key: https://dietrich.cx/pgp | 0x52FA4EE1722D54EB

3 3

List number of circuits per connection
by Logforme 20 Oct '22

20 Oct '22

I run the relay 8F6A78B1EA917F2BF221E87D14361C050A70CCC3 Like most relays mine has been targeted by the DoS attack. Hundreds of VPS IPs creating millions of IP connections. This I mitigated with rules in my firewall. Looking at the firewall counters it looks like that attack has now stopped. However the relay is still overloaded from lots of circuit creations. Normally my little relay reports around 100K circuits open in the log file but since the overloading started it's closer to 1M circuits open. All these circuit creations put a strain on the CPU, sometime pegging it at 100% (4 core i5-4670K CPU @ 3.40GHz). Worse, it seems to eat memory. Normally the tor process uses about 3GB (out of 8GB) but I have seen it quickly shoot up to using all memory and all swap. I assume this is because of the circuit creation DoS (it's new behavior) and what causes the oom killer to kill the tor process at least once a day or so. So, how do I mitigate the circuit creation DoS? My immediate thought is to identify if there are a few IPs responsible for the majority and add those to my firewall naughty list. Is there a way to query the tor process about number of open circuits mapped to IP addresses?

3 4

bridge down
by Anonforpeace 19 Oct '22

19 Oct '22

Hello: My Tor Bridge has been down for awhile as I was moving to a new home. I have been trying to bring it up again and have been receiving the messages you see below. I have checked the the tor project status and see that there is a Ddos attack affecting the network. Is that why I am getting this or am I doing something wrong? Thank you. darkhoodie@darkhoodie-HP-Compaq-Pro-6300-SFF:~$ sudo systemctl enable --now tor.service Synchronizing state of tor.service with SysV service script with /lib/systemd/systemd-sysv-install. Executing: /lib/systemd/systemd-sysv-install enable tor darkhoodie@darkhoodie-HP-Compaq-Pro-6300-SFF:~$ sudo systemctl restart tor.service darkhoodie@darkhoodie-HP-Compaq-Pro-6300-SFF:~$ journalctl -e -u tor@default Oct 16 00:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 00:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 01:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 01:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [59 similar message(s) suppressed in last 3540 > Oct 16 01:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 01:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 02:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 02:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 02:14:16 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: No circuits are opened. Relaxed timeout for circuit 875 (a Testing circuit 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the> Oct 16 02:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 02:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 03:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 03:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 03:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 03:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 04:00:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Heartbeat: Tor's uptime is 1 day 12:00 hours, with 0 circuits open. I've sent 6.47 MB and received 36.85 MB. I've received 0 connections on IPv4 and 0 on IPv6. I've made 161 c> Oct 16 04:00:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: While not bootstrapping, fetched this many bytes: 28998071 (server descriptor fetch); 2431916 (consensus network-status fetch); 310310 (microdescriptor fetch) Oct 16 04:00:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Heartbeat: Since last heartbeat message, I have seen 0 unique clients. Oct 16 04:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 04:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [59 similar message(s) suppressed in last 3540 > Oct 16 04:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 04:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 05:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 05:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 05:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 05:24:16 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: No circuits are opened. Relaxed timeout for circuit 920 (a Testing circuit 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the> Oct 16 05:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 06:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 06:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 06:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 06:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 07:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 07:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [59 similar message(s) suppressed in last 3540 > Oct 16 07:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 07:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 08:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 08:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 08:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 08:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 09:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 09:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 09:02:21 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your network connection speed appears to have changed. Resetting timeout to 60000ms after 18 timeouts and 103 buildtimes. Oct 16 09:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 09:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 10:00:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Heartbeat: Tor's uptime is 1 day 18:00 hours, with 0 circuits open. I've sent 7.75 MB and received 43.92 MB. I've received 0 connections on IPv4 and 0 on IPv6. I've made 188 c> Oct 16 10:00:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: While not bootstrapping, fetched this many bytes: 34600394 (server descriptor fetch); 2858305 (consensus network-status fetch); 361968 (microdescriptor fetch) Oct 16 10:00:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Heartbeat: Since last heartbeat message, I have seen 0 unique clients. Oct 16 10:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 10:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [59 similar message(s) suppressed in last 3540 > Oct 16 10:20:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 10:40:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 11:00:14 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Your server has not managed to confirm reachability for its ORPort(s) at 100.2.224.20:443. Relays do not publish descriptors until their ORPort and DirPort are reachable. Plea> Oct 16 11:00:15 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Unable to find IPv6 address for ORPort 443. You might want to specify IPv4Only to it or set an explicit address or set Address. [60 similar message(s) suppressed in last 3540 > Oct 16 11:05:13 darkhoodie-HP-Compaq-Pro-6300-SFF systemd[1]: Stopping Anonymizing overlay network for TCP... Oct 16 11:05:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Interrupt: we have stopped accepting new connections, and will shut down in 30 seconds. Interrupt again to exit now. Oct 16 11:05:13 darkhoodie-HP-Compaq-Pro-6300-SFF Tor[16182]: Delaying directory fetches: We are hibernating or shutting down. Sent with [Proton Mail](https://proton.me/) secure email.

4 4

Most useful bridge distribution
by Thoughts 12 Oct '22

12 Oct '22

Hi all - Running a bridge, and per some stats I saw earlier Moat seems to be the most popular distribution method. Just curious is something else is recommended? Some seem less secure than others, but I don't know the methods used to prevent abuse. Would appreciate guidance. Cheers, Kevin

2 1

Tor 0.4.6.x is unsupported, please upgrade
by Georg Koppen 09 Oct '22

09 Oct '22

Hello! To repeat what recently[1] got brought up: it's time to get relays upgraded in case they are running the EOL 0.4.6.x Tor series. We'll start reaching out to operators with valid contact information this week and plan to reject relays which are still on 0.4.6.x in about 5 weeks from now on at the end of September. You can follow along that process in our bug tracker[2] if you want. For the general processes around dealing with EOL relays in the Tor network see my mail from last October[3]. Feedback and improvements are welcome, as always. Georg [1] https://twitter.com/torproject/status/1557392816250441728 [2] https://gitlab.torproject.org/tpo/network-health/team/-/issues/252 [3] https://lists.torproject.org/pipermail/tor-relays/2021-October/019862.html

4 4

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

tor-relays