Have you checked your kernel socket backlog limit? (net.core.somaxconn)
On Jan 25, 2018, at 1:19 PM, Christian Krbusek christian@ph3x.at wrote:
Dear list,
recently I´m facing issues with my 4 middle relays:
B1E889E8EA604D81326F5071E0ABF1B2B16D5DAB FC9AC8EA0160D88BCCFDE066940D7DD9FA45495B BDB9EBCB1C2A424973A240C91EEC082C3EB61626 ACD889D86E02EDDAB1AFD81F598C0936238DC6D0
All of them running v0.3.2.9 on Debian stretch (kvm, 2 cores, 2GB RAM) and stopping with the log entries below. I can´t remember when this started but I´m quite sure it did not happen a few month ago.
There have not been any recent changes on the hardwarenode or on the VMs.
All of them running the "same" configuration created by ansible-relayor - one example below.
The VMs are running at the time the relays go offline and even a network restart doesn´t fix the issue so I have to restart the VM to get the relays up and running again.
Any hints would be much appreciated, if any more infos are needed just ask.
Cheers, Christian
CONFIG START <<<<<
# ansible-relayor generated torrc configuration file # Note: manual changes will be OVERWRITTEN on the next ansible-playbook run
OfflineMasterKey 1 RunAsDaemon 0 Log notice syslog OutboundBindAddress 188.118.198.244 SocksPort 0 User _tor-188.118.198.244_443 DataDirectory /var/lib/tor-instances/188.118.198.244_443 ORPort 188.118.198.244:443
DirPort 188.118.198.244:80
SyslogIdentityTag 188.118.198.244_443
ControlSocket /var/lib/tor-instances/188.118.198.244_443/controlsocket
Nickname ph3x
Sandbox 1 ExitRelay 0 ExitPolicy reject *:*
ContactInfo 4096R/0x73538126032AD297 Christian Krbusek <abuse AT ph3x DOT at> - 1NtneNxb8awwH8woJ7oL7UVj1SJDpAn8Kc RelayBandwidthRate 15 MB RelayBandwidthBurst 30 MB NumCPUs 2
MyFamily acd889d86e02eddab1afd81f598c0936238dc6d0,b1e889e8ea604d81326f5071e0abf1b2b16d5dab,bdb9ebcb1c2a424973a240c91eec082c3eb61626,fc9ac8ea0160d88bccfde066940d7dd9fa45495b # end of torrc
CONFIG END <<<<<
LOGS START <<<<<
at-vie01a-tor01.ph3x.at:
Jan 25 17:58:24 at-vie01a-tor01 Tor-86.59.119.83_443[540]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 8629 v4 connections; and receiv ed 60 v1 connections, 4568 v2 connections, 9249 v3 connections, and 403685 v4 connections. Jan 25 18:00:17 at-vie01a-tor01 kernel: [64938.876706] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.773119] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.774838] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786154] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786189] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786235] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786250] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786315] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786333] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786391] TCP: too many orphaned sockets Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786411] TCP: too many orphaned sockets Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053055] net_ratelimit: 2 callbacks suppressed Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053056] TCP: too many orphaned sockets Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.301030] TCP: too many orphaned sockets Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.365099] TCP: too many orphaned sockets Jan 25 18:01:17 at-vie01a-tor01 kernel: [64998.557033] TCP: too many orphaned sockets Jan 25 18:01:18 at-vie01a-tor01 kernel: [64999.805082] TCP: too many orphaned sockets Jan 25 18:01:27 at-vie01a-tor01 kernel: [65009.277179] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533307] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533678] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533697] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533820] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.534615] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535200] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535460] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535708] TCP: too many orphaned sockets Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.536685] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677629] net_ratelimit: 11 callbacks suppressed Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677661] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677910] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677989] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680478] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680857] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682767] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682832] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682953] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683033] TCP: too many orphaned sockets Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683041] TCP: too many orphaned sockets Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590190] net_ratelimit: 6 callbacks suppressed Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590194] TCP: too many orphaned sockets Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.597927] TCP: too many orphaned sockets Jan 25 18:05:15 at-vie01a-tor01 kernel: [65237.086196] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters. Jan 25 18:17:01 at-vie01a-tor01 CRON[24416]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 18:57:09 at-vie01a-tor01 systemd[1]: Stopped target Timers.
at-vie01a-tor02.ph3x.at:
Jan 25 10:17:01 at-vie01a-tor02 CRON[19123]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 11:17:01 at-vie01a-tor02 CRON[23992]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Heartbeat: Tor's uptime is 11:59 hours, with 38342 circuits open. I've sent 218.94 GB and received 192.87 GB. Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Circuit handshake stats since last time: 88566/90206 TAP, 12299213/12353484 NTor. Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 6295 v4 connections; and receiv ed 60 v1 connections, 3085 v2 connections, 6040 v3 connections, and 300693 v4 connections. Jan 25 12:17:01 at-vie01a-tor02 CRON[28856]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 12:18:28 at-vie01a-tor02 kernel: [44384.957658] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters. Jan 25 13:17:01 at-vie01a-tor02 CRON[28998]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 14:17:01 at-vie01a-tor02 CRON[29061]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 14:36:57 at-vie01a-tor02 ntpd[490]: Soliciting pool server 185.144.161.170 Jan 25 14:37:05 at-vie01a-tor02 ntpd[490]: Soliciting pool server 2a02:1748:0:1500:3::2005 Jan 25 14:37:16 at-vie01a-tor02 ntpd[490]: error resolving pool 1.debian.pool.ntp.org: Temporary failure in name resolution (-3) Jan 25 14:37:36 at-vie01a-tor02 ntpd[490]: error resolving pool 0.debian.pool.ntp.org: Temporary failure in name resolution (-3)
at-vie01a-tor03.ph3x.at:
Jan 24 13:21:18 at-vie01a-tor03 systemd[1]: apt-daily.timer: Adding 9h 35min 45.841113s random time. Jan 24 13:24:24 at-vie01a-tor03 kernel: [62694.054136] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters. Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Heartbeat: Tor's uptime is 17:59 hours, with 0 circuits open. I've sent 314.59 GB and received 317.28 GB. Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Circuit handshake stats since last time: 35924/36525 TAP, 10006455/10027375 NTor. Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 11906 v4 connections; and received 47 v1 connections, 4092 v2 connections, 8578 v3 connections, and 409960 v4 connections. Jan 24 14:17:01 at-vie01a-tor03 CRON[21604]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 24 15:17:01 at-vie01a-tor03 CRON[21670]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 24 15:41:25 at-vie01a-tor03 ntpd[492]: error resolving pool 1.debian.pool.ntp.org: Temporary failure in name resolution (-3)
at-vie01a-tor04.ph3x.at:
Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Heartbeat: Tor's uptime is 11:59 hours, with 36596 circuits open. I've sent 192.29 GB and received 194.32 GB. Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Circuit handshake stats since last time: 84713/86100 TAP, 11892935/11941101 NTor. Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 5938 v4 connections; and rec eived 44 v1 connections, 2949 v2 connections, 5395 v3 connections, and 287856 v4 connections. Jan 25 12:17:01 at-vie01a-tor04 CRON[29820]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 12:29:17 at-vie01a-tor04 kernel: [44936.694001] hrtimer: interrupt took 12598111 ns Jan 25 13:17:01 at-vie01a-tor04 CRON[2275]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 14:17:01 at-vie01a-tor04 CRON[7180]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 14:27:14 at-vie01a-tor04 kernel: [52013.316026] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters. Jan 25 15:17:01 at-vie01a-tor04 CRON[7920]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 16:17:01 at-vie01a-tor04 CRON[8023]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jan 25 16:42:38 at-vie01a-tor04 ntpd[492]: Soliciting pool server 81.16.38.161 Jan 25 16:42:46 at-vie01a-tor04 ntpd[492]: error resolving pool 3.debian.pool.ntp.org: Temporary failure in name resolution (-3)
LOGS END <<<<<
tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays