Probably to do with the lately regular spikes in load on nodes. You should configure tor to protect itself and the machine it is running on:
Limit its maxmeminqueue to <= 1gb (in torrc) limit virtual mem adressable by tor to <=2gb (limits on process) limit number open files to your usual load (limits on process) in firewall limit number of connections / ip in firewall limit new connection rate limit / ip
This might not be enough for tor to survive, but your system should.
There have been quite a few mails about that recently.
Regards
On Thu, 25 Jan 2018 at 20:30 Conrad Rockenhaus conrad@rockenhaus.com wrote:
Have you checked your kernel socket backlog limit? (net.core.somaxconn)
On Jan 25, 2018, at 1:19 PM, Christian Krbusek christian@ph3x.at
wrote:
Dear list,
recently I´m facing issues with my 4 middle relays:
B1E889E8EA604D81326F5071E0ABF1B2B16D5DAB FC9AC8EA0160D88BCCFDE066940D7DD9FA45495B BDB9EBCB1C2A424973A240C91EEC082C3EB61626 ACD889D86E02EDDAB1AFD81F598C0936238DC6D0
All of them running v0.3.2.9 on Debian stretch (kvm, 2 cores, 2GB RAM)
and stopping with the
log entries below. I can´t remember when this started but I´m quite sure
it did not happen
a few month ago.
There have not been any recent changes on the hardwarenode or on the VMs.
All of them running the "same" configuration created by ansible-relayor
- one example below.
The VMs are running at the time the relays go offline and even a network
restart doesn´t fix
the issue so I have to restart the VM to get the relays up and running
again.
Any hints would be much appreciated, if any more infos are needed just
ask.
Cheers, Christian
> CONFIG START <<<<<
# ansible-relayor generated torrc configuration file # Note: manual changes will be OVERWRITTEN on the next ansible-playbook
run
OfflineMasterKey 1 RunAsDaemon 0 Log notice syslog OutboundBindAddress 188.118.198.244 SocksPort 0 User _tor-188.118.198.244_443 DataDirectory /var/lib/tor-instances/188.118.198.244_443 ORPort 188.118.198.244:443
DirPort 188.118.198.244:80
SyslogIdentityTag 188.118.198.244_443
ControlSocket /var/lib/tor-instances/188.118.198.244_443/controlsocket
Nickname ph3x
Sandbox 1 ExitRelay 0 ExitPolicy reject *:*
ContactInfo 4096R/0x73538126032AD297 Christian Krbusek <abuse AT ph3x
DOT at> - 1NtneNxb8awwH8woJ7oL7UVj1SJDpAn8Kc
RelayBandwidthRate 15 MB RelayBandwidthBurst 30 MB NumCPUs 2
MyFamily
acd889d86e02eddab1afd81f598c0936238dc6d0,b1e889e8ea604d81326f5071e0abf1b2b16d5dab,bdb9ebcb1c2a424973a240c91eec082c3eb61626,fc9ac8ea0160d88bccfde066940d7dd9fa45495b
# end of torrc
> CONFIG END <<<<<
> LOGS START <<<<<
at-vie01a-tor01.ph3x.at:
Jan 25 17:58:24 at-vie01a-tor01 Tor-86.59.119.83_443[540]: Since
startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 8629 v4 connections; and receiv
ed 60 v1 connections, 4568 v2 connections, 9249 v3 connections, and
403685 v4 connections.
Jan 25 18:00:17 at-vie01a-tor01 kernel: [64938.876706] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.773119] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.774838] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786154] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786189] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786235] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786250] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786315] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786333] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786391] TCP: too many
orphaned sockets
Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786411] TCP: too many
orphaned sockets
Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053055] net_ratelimit: 2
callbacks suppressed
Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053056] TCP: too many
orphaned sockets
Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.301030] TCP: too many
orphaned sockets
Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.365099] TCP: too many
orphaned sockets
Jan 25 18:01:17 at-vie01a-tor01 kernel: [64998.557033] TCP: too many
orphaned sockets
Jan 25 18:01:18 at-vie01a-tor01 kernel: [64999.805082] TCP: too many
orphaned sockets
Jan 25 18:01:27 at-vie01a-tor01 kernel: [65009.277179] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533307] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533678] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533697] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533820] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.534615] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535200] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535460] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535708] TCP: too many
orphaned sockets
Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.536685] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677629] net_ratelimit: 11
callbacks suppressed
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677661] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677910] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677989] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680478] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680857] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682767] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682832] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682953] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683033] TCP: too many
orphaned sockets
Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683041] TCP: too many
orphaned sockets
Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590190] net_ratelimit: 6
callbacks suppressed
Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590194] TCP: too many
orphaned sockets
Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.597927] TCP: too many
orphaned sockets
Jan 25 18:05:15 at-vie01a-tor01 kernel: [65237.086196] TCP:
request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
Jan 25 18:17:01 at-vie01a-tor01 CRON[24416]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 18:57:09 at-vie01a-tor01 systemd[1]: Stopped target Timers.
at-vie01a-tor02.ph3x.at:
Jan 25 10:17:01 at-vie01a-tor02 CRON[19123]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 11:17:01 at-vie01a-tor02 CRON[23992]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Heartbeat:
Tor's uptime is 11:59 hours, with 38342 circuits open. I've sent 218.94 GB and received 192.87 GB.
Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Circuit
handshake stats since last time: 88566/90206 TAP, 12299213/12353484 NTor.
Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Since
startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 6295 v4 connections; and receiv
ed 60 v1 connections, 3085 v2 connections, 6040 v3 connections, and
300693 v4 connections.
Jan 25 12:17:01 at-vie01a-tor02 CRON[28856]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 12:18:28 at-vie01a-tor02 kernel: [44384.957658] TCP:
request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
Jan 25 13:17:01 at-vie01a-tor02 CRON[28998]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 14:17:01 at-vie01a-tor02 CRON[29061]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 14:36:57 at-vie01a-tor02 ntpd[490]: Soliciting pool server
185.144.161.170
Jan 25 14:37:05 at-vie01a-tor02 ntpd[490]: Soliciting pool server
2a02:1748:0:1500:3::2005
Jan 25 14:37:16 at-vie01a-tor02 ntpd[490]: error resolving pool
1.debian.pool.ntp.org: Temporary failure in name resolution (-3)
Jan 25 14:37:36 at-vie01a-tor02 ntpd[490]: error resolving pool
0.debian.pool.ntp.org: Temporary failure in name resolution (-3)
at-vie01a-tor03.ph3x.at:
Jan 24 13:21:18 at-vie01a-tor03 systemd[1]: apt-daily.timer: Adding 9h
35min 45.841113s random time.
Jan 24 13:24:24 at-vie01a-tor03 kernel: [62694.054136] TCP:
request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Heartbeat:
Tor's uptime is 17:59 hours, with 0 circuits open. I've sent 314.59 GB and received 317.28 GB.
Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Circuit
handshake stats since last time: 35924/36525 TAP, 10006455/10027375 NTor.
Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Since
startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 11906 v4 connections; and received 47 v1 connections, 4092 v2 connections, 8578 v3 connections, and 409960 v4 connections.
Jan 24 14:17:01 at-vie01a-tor03 CRON[21604]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 24 15:17:01 at-vie01a-tor03 CRON[21670]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 24 15:41:25 at-vie01a-tor03 ntpd[492]: error resolving pool
1.debian.pool.ntp.org: Temporary failure in name resolution (-3)
at-vie01a-tor04.ph3x.at:
Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Heartbeat:
Tor's uptime is 11:59 hours, with 36596 circuits open. I've sent 192.29 GB and received 194.32 GB.
Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Circuit
handshake stats since last time: 84713/86100 TAP, 11892935/11941101 NTor.
Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Since
startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 5938 v4 connections; and rec
eived 44 v1 connections, 2949 v2 connections, 5395 v3 connections, and
287856 v4 connections.
Jan 25 12:17:01 at-vie01a-tor04 CRON[29820]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 12:29:17 at-vie01a-tor04 kernel: [44936.694001] hrtimer:
interrupt took 12598111 ns
Jan 25 13:17:01 at-vie01a-tor04 CRON[2275]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 14:17:01 at-vie01a-tor04 CRON[7180]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 14:27:14 at-vie01a-tor04 kernel: [52013.316026] TCP:
request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
Jan 25 15:17:01 at-vie01a-tor04 CRON[7920]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 16:17:01 at-vie01a-tor04 CRON[8023]: (root) CMD ( cd / &&
run-parts --report /etc/cron.hourly)
Jan 25 16:42:38 at-vie01a-tor04 ntpd[492]: Soliciting pool server
81.16.38.161
Jan 25 16:42:46 at-vie01a-tor04 ntpd[492]: error resolving pool
3.debian.pool.ntp.org: Temporary failure in name resolution (-3)
> LOGS END <<<<<
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays