[tor-relays] Middle relays stopping because of "SYN flood"?

Thu Jan 25 19:33:54 UTC 2018

Have you checked your kernel socket backlog limit? (net.core.somaxconn)

> On Jan 25, 2018, at 1:19 PM, Christian Krbusek <christian at ph3x.at> wrote:
> 
> Dear list,
> 
> recently I´m facing issues with my 4 middle relays:
> 
> B1E889E8EA604D81326F5071E0ABF1B2B16D5DAB
> FC9AC8EA0160D88BCCFDE066940D7DD9FA45495B
> BDB9EBCB1C2A424973A240C91EEC082C3EB61626
> ACD889D86E02EDDAB1AFD81F598C0936238DC6D0
> 
> All of them running v0.3.2.9 on Debian stretch (kvm, 2 cores, 2GB RAM) and stopping with the
> log entries below. I can´t remember when this started but I´m quite sure it did not happen
> a few month ago.
> 
> There have not been any recent changes on the hardwarenode or on the VMs.
> 
> All of them running the "same" configuration created by ansible-relayor - one example below.
> 
> The VMs are running at the time the relays go offline and even a network restart doesn´t fix
> the issue so I have to restart the VM to get the relays up and running again.
> 
> Any hints would be much appreciated, if any more infos are needed just ask.
> 
> Cheers,
> Christian
> 
>>>>>> CONFIG START <<<<<
> 
> # ansible-relayor generated torrc configuration file
> # Note: manual changes will be OVERWRITTEN on the next ansible-playbook run
> 
> OfflineMasterKey 1
> RunAsDaemon 0
> Log notice syslog
> OutboundBindAddress 188.118.198.244
> SocksPort 0
> User _tor-188.118.198.244_443
> DataDirectory /var/lib/tor-instances/188.118.198.244_443
> ORPort 188.118.198.244:443
> 
> 
> DirPort 188.118.198.244:80
> 
> SyslogIdentityTag 188.118.198.244_443
> 
> ControlSocket /var/lib/tor-instances/188.118.198.244_443/controlsocket
> 
> Nickname ph3x
> 
> Sandbox 1
> ExitRelay 0
> ExitPolicy reject *:*
> 
> ContactInfo 4096R/0x73538126032AD297 Christian Krbusek <abuse AT ph3x DOT at> - 1NtneNxb8awwH8woJ7oL7UVj1SJDpAn8Kc
> RelayBandwidthRate 15 MB
> RelayBandwidthBurst 30 MB
> NumCPUs 2
> 
> MyFamily acd889d86e02eddab1afd81f598c0936238dc6d0,b1e889e8ea604d81326f5071e0abf1b2b16d5dab,bdb9ebcb1c2a424973a240c91eec082c3eb61626,fc9ac8ea0160d88bccfde066940d7dd9fa45495b
> # end of torrc
> 
>>>>>> CONFIG END <<<<<
> 
>>>>>> LOGS START <<<<<
> 
> at-vie01a-tor01.ph3x.at:
> 
> Jan 25 17:58:24 at-vie01a-tor01 Tor-86.59.119.83_443[540]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 8629 v4 connections; and receiv
> ed 60 v1 connections, 4568 v2 connections, 9249 v3 connections, and 403685 v4 connections.
> Jan 25 18:00:17 at-vie01a-tor01 kernel: [64938.876706] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.773119] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.774838] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786154] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786189] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786235] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786250] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786315] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786333] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786391] TCP: too many orphaned sockets
> Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786411] TCP: too many orphaned sockets
> Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053055] net_ratelimit: 2 callbacks suppressed
> Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053056] TCP: too many orphaned sockets
> Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.301030] TCP: too many orphaned sockets
> Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.365099] TCP: too many orphaned sockets
> Jan 25 18:01:17 at-vie01a-tor01 kernel: [64998.557033] TCP: too many orphaned sockets
> Jan 25 18:01:18 at-vie01a-tor01 kernel: [64999.805082] TCP: too many orphaned sockets
> Jan 25 18:01:27 at-vie01a-tor01 kernel: [65009.277179] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533307] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533678] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533697] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533820] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.534615] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535200] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535460] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535708] TCP: too many orphaned sockets
> Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.536685] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677629] net_ratelimit: 11 callbacks suppressed
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677661] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677910] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677989] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680478] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680857] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682767] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682832] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682953] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683033] TCP: too many orphaned sockets
> Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683041] TCP: too many orphaned sockets
> Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590190] net_ratelimit: 6 callbacks suppressed
> Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590194] TCP: too many orphaned sockets
> Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.597927] TCP: too many orphaned sockets
> Jan 25 18:05:15 at-vie01a-tor01 kernel: [65237.086196] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.
> Jan 25 18:17:01 at-vie01a-tor01 CRON[24416]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 18:57:09 at-vie01a-tor01 systemd[1]: Stopped target Timers.
> 
> 
> at-vie01a-tor02.ph3x.at:
> 
> Jan 25 10:17:01 at-vie01a-tor02 CRON[19123]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 11:17:01 at-vie01a-tor02 CRON[23992]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Heartbeat: Tor's uptime is 11:59 hours, with 38342 circuits open. I've sent 218.94 GB and received 192.87 GB.
> Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Circuit handshake stats since last time: 88566/90206 TAP, 12299213/12353484 NTor.
> Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 6295 v4 connections; and receiv
> ed 60 v1 connections, 3085 v2 connections, 6040 v3 connections, and 300693 v4 connections.
> Jan 25 12:17:01 at-vie01a-tor02 CRON[28856]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 12:18:28 at-vie01a-tor02 kernel: [44384.957658] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.
> Jan 25 13:17:01 at-vie01a-tor02 CRON[28998]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 14:17:01 at-vie01a-tor02 CRON[29061]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 14:36:57 at-vie01a-tor02 ntpd[490]: Soliciting pool server 185.144.161.170
> Jan 25 14:37:05 at-vie01a-tor02 ntpd[490]: Soliciting pool server 2a02:1748:0:1500:3::2005
> Jan 25 14:37:16 at-vie01a-tor02 ntpd[490]: error resolving pool 1.debian.pool.ntp.org: Temporary failure in name resolution (-3)
> Jan 25 14:37:36 at-vie01a-tor02 ntpd[490]: error resolving pool 0.debian.pool.ntp.org: Temporary failure in name resolution (-3)
> 
> 
> at-vie01a-tor03.ph3x.at:
> 
> Jan 24 13:21:18 at-vie01a-tor03 systemd[1]: apt-daily.timer: Adding 9h 35min 45.841113s random time.
> Jan 24 13:24:24 at-vie01a-tor03 kernel: [62694.054136] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.
> Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Heartbeat: Tor's uptime is 17:59 hours, with 0 circuits open. I've sent 314.59 GB and received 317.28 GB.
> Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Circuit handshake stats since last time: 35924/36525 TAP, 10006455/10027375 NTor.
> Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 11906 v4 connections; and received 47 v1 connections, 4092 v2 connections, 8578 v3 connections, and 409960 v4 connections.
> Jan 24 14:17:01 at-vie01a-tor03 CRON[21604]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 24 15:17:01 at-vie01a-tor03 CRON[21670]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 24 15:41:25 at-vie01a-tor03 ntpd[492]: error resolving pool 1.debian.pool.ntp.org: Temporary failure in name resolution (-3)
> 
> 
> at-vie01a-tor04.ph3x.at:
> 
> Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Heartbeat: Tor's uptime is 11:59 hours, with 36596 circuits open. I've sent 192.29 GB and received 194.32 GB.
> Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Circuit handshake stats since last time: 84713/86100 TAP, 11892935/11941101 NTor.
> Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Since startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 connections, and 5938 v4 connections; and rec
> eived 44 v1 connections, 2949 v2 connections, 5395 v3 connections, and 287856 v4 connections.
> Jan 25 12:17:01 at-vie01a-tor04 CRON[29820]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 12:29:17 at-vie01a-tor04 kernel: [44936.694001] hrtimer: interrupt took 12598111 ns
> Jan 25 13:17:01 at-vie01a-tor04 CRON[2275]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 14:17:01 at-vie01a-tor04 CRON[7180]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 14:27:14 at-vie01a-tor04 kernel: [52013.316026] TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies.  Check SNMP counters.
> Jan 25 15:17:01 at-vie01a-tor04 CRON[7920]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 16:17:01 at-vie01a-tor04 CRON[8023]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
> Jan 25 16:42:38 at-vie01a-tor04 ntpd[492]: Soliciting pool server 81.16.38.161
> Jan 25 16:42:46 at-vie01a-tor04 ntpd[492]: error resolving pool 3.debian.pool.ntp.org: Temporary failure in name resolution (-3)
> 
>>>>>> LOGS END <<<<<
> 
> _______________________________________________
> tor-relays mailing list
> tor-relays at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays