I've been running a relay for about months now. It runs on an 1.6 Ghz single core Atom with hyperthreading, 1GB of RAM. It's on my home connection; I advertise only 176 KB/sec.
Under normal conditions, it pumps around 100KB/sec, it has around 600 connections, it uses around 20% CPU. Everything looks very healthy.
Lately I have seen a couple of incidents where the number of connections suddenly goes up to over 3000, traffic increases heavily, CPU usage goes well over 150% (out of possible 200). Traffic can go up to between 500 and 1000 KB/sec for long periods of time. Sometimes it seems that my relay just can't take it anymore. In the log, the ratio of TAP handshakes goes wild, and I get clock jump warnings. My clock does not jump. This is Tor hanging while allocating memory.
Today I had a really bad episode, where the box started thrashing. When it became responsive again, Tor was left in a state where it was constantly downloading about 400KB/sec more than it was uploading. Normally I have a little bit more up than down, because I'm a directory server as well. I can not explain having a lot more down than up. I can fantasize that those hangs/clock jumps ("assuming established circuits no longer work") could leave 'half' circuits.
Finally I restarted my relay, (which I really don't like to do) and after a while it stabilized. At this point, my router shows a peak of almost 8000 NAT sessions.
Is this normal behavior of the network (esp. the sudden increase in connections) or is this another kind of attack/probe like what we've seen in early September? Is this because this machine is just too underpowered? Should I collect/provide any diagnostics? Have others seen similar events?
Thanks, -Job
Nov 16 22:39:52.000 [notice] Heartbeat: Tor's uptime is 36 days 6:00 hours, with 615 circuits open. I've sent 192.16 GB and received 171.74 GB. Nov 16 22:39:52.000 [notice] Average packaged cell fullness: 87.305% Nov 16 22:39:52.000 [notice] TLS write overhead: 9% Nov 16 23:39:28.000 [notice] Circuit handshake stats since last time: 6011/6013 TAP, 35/35 NTor. Nov 17 00:39:28.000 [notice] Circuit handshake stats since last time: 12521/12521 TAP, 47/47 NTor. Nov 17 01:24:45.000 [warn] Your system clock just jumped 138 seconds forward; assuming established circuits no longer work. Nov 17 01:39:28.000 [notice] Circuit handshake stats since last time: 49689/1454481 TAP, 51/51 NTor. Nov 17 02:28:44.000 [warn] Your system clock just jumped 174 seconds forward; assuming established circuits no longer work. Nov 17 02:33:12.000 [warn] Your system clock just jumped 268 seconds forward; assuming established circuits no longer work. Nov 17 02:35:17.000 [warn] Your system clock just jumped 125 seconds forward; assuming established circuits no longer work. Nov 17 02:39:28.000 [notice] Circuit handshake stats since last time: 55822/3477278 TAP, 37/37 NTor.
Nov 25 06:53:31.000 [notice] Circuit handshake stats since last time: 28194/28200 TAP, 32/32 NTor. Nov 25 07:29:57.000 [warn] Your system clock just jumped 114 seconds forward; assuming established circuits no longer work. Nov 25 07:31:58.000 [warn] Your system clock just jumped 121 seconds forward; assuming established circuits no longer work. Nov 25 07:53:31.000 [notice] Circuit handshake stats since last time: 24207/2692265 TAP, 42/42 NTor. Nov 25 08:53:31.000 [notice] Circuit handshake stats since last time: 76451/4271579 TAP, 49/49 NTor. Nov 25 09:53:31.000 [notice] Circuit handshake stats since last time: 84956/3666170 TAP, 42/42 NTor. Nov 25 10:45:15.000 [warn] Your system clock just jumped 302 seconds forward; assuming established circuits no longer work. Nov 25 11:01:59.000 [warn] Your system clock just jumped 1004 seconds forward; assuming established circuits no longer work. Nov 25 11:02:04.000 [notice] Circuit handshake stats since last time: 64816/3151447 TAP, 38/38 NTor. Nov 25 11:02:18.000 [notice] Heartbeat: Tor's uptime is 5 days 11:42 hours, with 2450 circuits open. I've sent 46.92 GB and received 46.11 GB. Nov 25 11:02:18.000 [notice] Average packaged cell fullness: 82.623% Nov 25 11:02:18.000 [notice] TLS write overhead: 8% Nov 25 11:53:31.000 [notice] Circuit handshake stats since last time: 18710/1099205 TAP, 193/202 NTor. Nov 25 13:22:03.000 [warn] Your system clock just jumped 1923 seconds forward; assuming established circuits no longer work. Nov 25 13:27:23.000 [notice] Circuit handshake stats since last time: 70795/3452514 TAP, 138/140 NTor. Nov 25 13:35:51.000 [warn] Your system clock just jumped 828 seconds forward; assuming established circuits no longer work. Nov 25 13:42:15.000 [warn] Your system clock just jumped 384 seconds forward; assuming established circuits no longer work. Nov 25 13:43:15.000 [notice] Received reload signal (hup). Reloading config and resetting internal state. Nov 25 13:43:16.000 [notice] Read configuration file "/opt/etc/tor/torrc". Nov 25 13:46:09.000 [notice] Interrupt: we have stopped accepting new connections, and will shut down in 30 seconds. Interrupt again to exit now. Nov 25 13:46:39.000 [notice] Clean shutdown finished. Exiting.