Hi,
I didn't see your original email and I don't currently have it in my inbox to reply to it directly but I can tell you by looking at your metrics history for "Alberta", indeed you had the same problem on Dec.18 that happened to my relays.
The attackers seem to be picking a group of relays at a time and move on to others and it may be a while before they come back again. However the problem gets amplified if you're running an exit relay without having large enough resources to deal with all this traffic.
I'm not running an exit relay, so I have to deal with the problem for a few hours, but all this outgoing traffic from my relay and other relays that are under attack will inevitably end up at exit relays. This means that even if they move on from my relay to another one, the traffic will still continue to poor into exit relays, so the duration of the attack for an exit relay may be much longer.
This shouldn't stop you from bringing your rely back up. You're not doing anything wrong and there's nothing wrong with your set up. If you're running other services on the same server of course the heavy load might disrupt those services as well, which is why it's always a good idea to only run Tor on the server and not to mix it with other services.
My suggestion for the time being, keep running my firewall scripts as they do help making the duration of the attack shorter by putting some of those IP addresses in the block list and help making it less painful and see how it goes. And in the long run, if you find that you don't have the resources or patience to deal with such heavy load, consider running a middle/Guard relay instead of an Exit. They're much easier to deal with during an attack. But don't quit. It'll pass.
Cheers.
On 1/15/2024 8:14 PM, denny.obreham@a-n-o-n-y-m-e.net wrote:
Sorry, but I'm going to vent a little bit. I'm not a network specialist, so 11 days ago I sent the following email to this mailing list asking for help because two of my Tor exit relays were completely frozen and I was unable to put them online again.
Nobody answered, not even a comment. No wait, there was one person - very active on this mailing list recently - who sent me an email personally to tell me that I was an idiot (referring to what, I don't know) who should kill himself. Adding furthermore that if I didn't commit suicide within 72 hours, he would personally come to my house and kill me with a Glock 9 mm. Fun stuff, very disturbing.
Anyway, no serious answers, someone calling m e an idiot: I tried to look as best as I could at what I did wrong. Couldn't find anything. My only available solution was to rebuild completely my server, something I wanted to do for a while because of other undesired quirks that were bothering me with my setup. I knew it would take a long time
- which I didn't really have - but I finally finished my new setup
yesterday. (Don't look for 25FC41154DCB2CAE3ABD74A8DFCD5B90D2CFFD57 or the bridge, they have been shut down for the moment.)
Today, I read a line from Chris Endiku-6 saying: "There’s something going on for a while and I haven’t seen any mentions of it." The exact problem I mentioned! He says it goes "as early as Dec.23"; my problem goes to Dec 18 as shown in my previous email. Also, not mentioned in my previous email, before I renewed my setup, my tor-ddos firewall rules (I use the ones f rom Endiku-6) had blocked about 5 times more IP than usual - if that can be useful information to anyone.
Sorry for the rent, I just needed to vent a little: There was something wrong, and I wasn't crazy or incompetent!
I still would like to know how to restart such a relay, if this happens again in the future - other than reinstalling the entire server, that is.
Thanks in advance.
On 01/06/2024 01:52 PM denny.obreham@a-n-o-n-y-m-e.net wrote ..
I manage the two following exit relays: * https://metrics.torproject.org/rs.html#details/25FC41154DCB2CAE3ABD74A8DFCD5B90D2CFFD57* https://metrics.torproject.org/rs.html#details/3B85067588C3F017D5CCF7D8F65B5881B7D4C97C They are both on the same IP and servers. A few hours ago they lost contact even though I have both Apache and I2P servers on the same machine that are reachable. The weird log messages seem to go back to Dec 18, after a restart: ``` Dec 18 07:49:47 a-n-o-n-y-m-e Tor-Alberta[904715]: Bootstrapped 100% (done): Done Dec 18 07:54:18 a-n-o-n-y-m-e Tor-Alberta[904715]: Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. Dec 18 07:57:11 a-n-o-n-y-m-e Tor-Alberta[904715]: Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [14254 similar message(s) suppressed in last 180 seconds] ``` There are a few more of these messages appearing afterward (My bandwidth is unlimited). This is the typical Heartbeat that comes afterward: ``` Dec 18 13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: Heartbeat: Tor's uptime is 6:00 hours, with 476 circuits open. I've sent 30.87 GB and received 9.04 GB. I've received 13401 connections on IPv4 and 0 on IPv6. I've made 46754 connections with IPv4 and 0 with IPv6. Dec 18 13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: While not bootstrapping, fetched this many bytes: 6412201 (server descriptor fetch); 1424 (server descriptor upload); 376002 (consensus network-status fetch); 57564 (microdescriptor fetch) Dec 18 13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: Circuit handshake stats since last time: 8/8 TAP, 1356688/6457307 NTor. Dec 18 13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: Since startup we initiated 0 and received 0 v1 connections; initiated 0 and received 0 v2 connections; initiated 0 and received 0 v3 connections; initiated 0 and received 429 v4 connections; initiated 285 and received 11137 v5 connections. Dec 18 13:49:42 a-n-o-n-y-m-e Tor-Alberta[904715]: Heartbeat: DoS mitigation since startup: 0 circuits killed with too many cells, 0 circuits rejected, 0 marked addresses, 0 marked addresses for max queue, 0 same address concurrent connections rejected, 0 connections rejected, 0 single hop clients refused, 0 INTRODUCE2 rejected. ``` Then I have this kind of message appearing: ``` Dec 18 14:13:23 a-n-o-n-y-m-e Tor-Alberta[904715]: No circuits are opened. Relaxed timeout for circuit 4487 (a Measuring circuit timeout 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. [4 similar message(s) suppressed in last 3300 seconds] ``` Then a few days later, this bug report: ``` Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: tor_bug_occurred_(): Bug: ../src/core/or/conflux.c:567: conflux_pick_first_leg: Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed. (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: Tor 0.4.8.10: Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed in conflux_pick_first_leg at ../src/core/or/conflux.c:567. Stack trace: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(log_backtrace_impl+0x5b) [0x55651f95b37b] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_bug_occurred_+0x18a) [0x55651f97294a] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(conflux_decide_next_circ+0x40e) [0x55651fa12afe] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(circuit_get_package_window+0x75) [0x55651fa12ec5] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x9ed63) [0x55651f908d63] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(connection_edge_package_raw_inbuf+0xae) [0x55651f90b80e] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(connection_edge_process_inbuf+0x6f) [0x55651fa2b9df] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x1c2fb4) [0x55651fa2cfb4] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x73ffc) [0x55651f8ddffc] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x1ff58) [0x7fcee899bf58] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x577) [0x7fcee899d8a7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(do_main_loop+0x127) [0x55651f8de7c7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_run_main+0x215) [0x55651f8e2805] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_main+0x4d) [0x55651f8e2c6d] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(main+0x1d) [0x55651f8d4dcd] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcee80a8d90] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fcee80a8e40] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(_start+0x25) [0x55651f8d4e25] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: Matching client sets: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_log_set(): Bug: Conflux 566550141B144136: 0 linked, 0 launched. Delivered: 75; teardown: 0; Current: (nil), Previous: (nil) (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: Matching server sets: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_log_set(): Bug: Conflux 566550141B144136: 0 linked, 0 launched. Delivered: 75; teardown: 0; Current: (nil), Previous: (nil) (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: End conflux set dump (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: circuit_get_package_window(): Bug: Conflux has no circuit to send on. Circuit 0x556536e76c20 idx 138 marked at line ../src/core/or/command.c:663 (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: tor_bug_occurred_(): Bug: ../src/core/or/conflux.c:567: conflux_pick_first_leg: Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed. (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: Tor 0.4.8.10: Non-fatal assertion !(smartlist_len(cfx->legs) <= 0) failed in conflux_pick_first_leg at ../src/core/or/conflux.c:567. Stack trace: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(log_backtrace_impl+0x5b) [0x55651f95b37b] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_bug_occurred_+0x18a) [0x55651f97294a] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(conflux_decide_next_circ+0x40e) [0x55651fa12afe] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(circuit_get_package_window+0x75) [0x55651fa12ec5] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x9ed63) [0x55651f908d63] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(connection_edge_package_raw_inbuf+0xae) [0x55651f90b80e] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(connection_edge_process_inbuf+0x6f) [0x55651fa2b9df] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x1c34dd) [0x55651fa2d4dd] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(+0x73ffc) [0x55651f8ddffc] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x1ff58) [0x7fcee899bf58] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x577) [0x7fcee899d8a7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(do_main_loop+0x127) [0x55651f8de7c7] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_run_main+0x215) [0x55651f8e2805] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(tor_main+0x4d) [0x55651f8e2c6d] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(main+0x1d) [0x55651f8d4dcd] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fcee80a8d90] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fcee80a8e40] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: Bug: /usr/bin/tor(_start+0x25) [0x55651f8d4e25] (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: Matching client sets: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_log_set(): Bug: Conflux 566550141B144136: 0 linked, 0 launched. Delivered: 75; teardown: 0; Current: (nil), Previous: (nil) (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: Matching server sets: (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_log_set(): Bug: Conflux 566550141B144136: 0 linked, 0 launched. Delivered: 75; teardown: 0; Current: (nil), Previous: (nil) (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: conflux_pick_first_leg(): Bug: End conflux set dump (on Tor 0.4.8.10 ) Dec 21 15:18:48 a-n-o-n-y-m-e Tor-Alberta[904715]: circuit_get_package_window(): Bug: Conflux has no circuit to send on. Circuit 0x556536e76c20 idx 138 marked at line ../src/core/or/command.c:663 (on Tor 0.4.8.10 ) ``` It is then an alternation of these three last messages (Heartbeat + "No circuits are opened" + bug) until yesterday where I get a series of these messages: ``` Jan 5 21:58:59 a-n-o-n-y-m-e Tor-Alberta[904715]: Failed to find node for hop #1 of our path. Discarding this circuit. Jan 5 21:58:59 a-n-o-n-y-m-e Tor-Alberta[904715]: Our circuit 0 (id: 355722) died due to an invalid selected path, purpose Unlinked conflux circuit. This may be a torrc configuration issue, or a bug. ``` At this point, I learned that my relays were down via Tor Weather. I restarted the relays - even rebooted the whole server - and both relays seem to be unable to connect to the network. Here's a sample: ``` Jan 6 09:44:46 a-n-o-n-y-m-e Tor-Alberta[5621]: 254 connections have failed: Jan 6 09:44:46 a-n-o-n-y-m-e Tor-Alberta[5621]: 254 connections died in state connect()ing with SSL state (No SSL object) Jan 6 09:44:46 a-n-o-n-y-m-e Tor-Alberta[5621]: Problem bootstrapping. Stuck at 5% (conn): Connecting to a relay. (Connection timed out; TIMEOUT; count 256; recommendation warn; host 65369D044C659CD299E35763914FFD0FC9AD4509 at 92.205.161.164:80) ``` I also have a bridge on this server (different IP) and it seems OK ( https://metrics.torproject.org/rs.html#details/7CEB9D16C5218FE1A9BAB8E8A6EA9471D2E1F9B8 ). Last messages after the server reboot are: ``` Jan 6 08:34:17 a-n-o-n-y-m-e Tor-Nestor[998]: Bootstrapped 100% (done): Done Jan 6 08:35:21 a-n-o-n-y-m-e Tor-Nestor[998]: Performing bandwidth self-test...done. Jan 6 09:54:22 a-n-o-n-y-m-e Tor-Nestor[998]: No circuits are opened. Relaxed timeout for circuit 527 (a Measuring circuit timeout 3-hop circuit in state doing handshakes with channel state open) to 60000ms. However, it appears the circuit has timed out anyway. ``` I have stopped both relays until I figure out what is going on. Help would be appreciated.