25 Jan
2018
25 Jan
'18
7:53 p.m.
Probably to do with the lately regular spikes in load on nodes. You should configure tor to protect itself and the machine it is running on: Limit its maxmeminqueue to <= 1gb (in torrc) limit virtual mem adressable by tor to <=2gb (limits on process) limit number open files to your usual load (limits on process) in firewall limit number of connections / ip in firewall limit new connection rate limit / ip This might not be enough for tor to survive, but your system should. There have been quite a few mails about that recently. Regards On Thu, 25 Jan 2018 at 20:30 Conrad Rockenhaus <conrad@rockenhaus.com> wrote: > Have you checked your kernel socket backlog limit? (net.core.somaxconn) > > > On Jan 25, 2018, at 1:19 PM, Christian Krbusek <christian@ph3x.at> > wrote: > > > > Dear list, > > > > recently I´m facing issues with my 4 middle relays: > > > > B1E889E8EA604D81326F5071E0ABF1B2B16D5DAB > > FC9AC8EA0160D88BCCFDE066940D7DD9FA45495B > > BDB9EBCB1C2A424973A240C91EEC082C3EB61626 > > ACD889D86E02EDDAB1AFD81F598C0936238DC6D0 > > > > All of them running v0.3.2.9 on Debian stretch (kvm, 2 cores, 2GB RAM) > and stopping with the > > log entries below. I can´t remember when this started but I´m quite sure > it did not happen > > a few month ago. > > > > There have not been any recent changes on the hardwarenode or on the VMs. > > > > All of them running the "same" configuration created by ansible-relayor > - one example below. > > > > The VMs are running at the time the relays go offline and even a network > restart doesn´t fix > > the issue so I have to restart the VM to get the relays up and running > again. > > > > Any hints would be much appreciated, if any more infos are needed just > ask. > > > > Cheers, > > Christian > > > >>>>>> CONFIG START <<<<< > > > > # ansible-relayor generated torrc configuration file > > # Note: manual changes will be OVERWRITTEN on the next ansible-playbook > run > > > > OfflineMasterKey 1 > > RunAsDaemon 0 > > Log notice syslog > > OutboundBindAddress 188.118.198.244 > > SocksPort 0 > > User _tor-188.118.198.244_443 > > DataDirectory /var/lib/tor-instances/188.118.198.244_443 > > ORPort 188.118.198.244:443 > > > > > > DirPort 188.118.198.244:80 > > > > SyslogIdentityTag 188.118.198.244_443 > > > > ControlSocket /var/lib/tor-instances/188.118.198.244_443/controlsocket > > > > Nickname ph3x > > > > Sandbox 1 > > ExitRelay 0 > > ExitPolicy reject *:* > > > > ContactInfo 4096R/0x73538126032AD297 Christian Krbusek <abuse AT ph3x > DOT at> - 1NtneNxb8awwH8woJ7oL7UVj1SJDpAn8Kc > > RelayBandwidthRate 15 MB > > RelayBandwidthBurst 30 MB > > NumCPUs 2 > > > > MyFamily > acd889d86e02eddab1afd81f598c0936238dc6d0,b1e889e8ea604d81326f5071e0abf1b2b16d5dab,bdb9ebcb1c2a424973a240c91eec082c3eb61626,fc9ac8ea0160d88bccfde066940d7dd9fa45495b > > # end of torrc > > > >>>>>> CONFIG END <<<<< > > > >>>>>> LOGS START <<<<< > > > > at-vie01a-tor01.ph3x.at: > > > > Jan 25 17:58:24 at-vie01a-tor01 Tor-86.59.119.83_443[540]: Since > startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 > connections, and 8629 v4 connections; and receiv > > ed 60 v1 connections, 4568 v2 connections, 9249 v3 connections, and > 403685 v4 connections. > > Jan 25 18:00:17 at-vie01a-tor01 kernel: [64938.876706] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.773119] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.774838] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786154] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786189] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786235] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786250] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786315] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786333] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786391] TCP: too many > orphaned sockets > > Jan 25 18:01:06 at-vie01a-tor01 kernel: [64987.786411] TCP: too many > orphaned sockets > > Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053055] net_ratelimit: 2 > callbacks suppressed > > Jan 25 18:01:15 at-vie01a-tor01 kernel: [64997.053056] TCP: too many > orphaned sockets > > Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.301030] TCP: too many > orphaned sockets > > Jan 25 18:01:16 at-vie01a-tor01 kernel: [64998.365099] TCP: too many > orphaned sockets > > Jan 25 18:01:17 at-vie01a-tor01 kernel: [64998.557033] TCP: too many > orphaned sockets > > Jan 25 18:01:18 at-vie01a-tor01 kernel: [64999.805082] TCP: too many > orphaned sockets > > Jan 25 18:01:27 at-vie01a-tor01 kernel: [65009.277179] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533307] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533678] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533697] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.533820] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.534615] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535200] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535460] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.535708] TCP: too many > orphaned sockets > > Jan 25 18:01:28 at-vie01a-tor01 kernel: [65009.536685] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677629] net_ratelimit: 11 > callbacks suppressed > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677661] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677910] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.677989] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680478] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.680857] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682767] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682832] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.682953] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683033] TCP: too many > orphaned sockets > > Jan 25 18:01:34 at-vie01a-tor01 kernel: [65015.683041] TCP: too many > orphaned sockets > > Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590190] net_ratelimit: 6 > callbacks suppressed > > Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.590194] TCP: too many > orphaned sockets > > Jan 25 18:02:13 at-vie01a-tor01 kernel: [65054.597927] TCP: too many > orphaned sockets > > Jan 25 18:05:15 at-vie01a-tor01 kernel: [65237.086196] TCP: > request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. > Check SNMP counters. > > Jan 25 18:17:01 at-vie01a-tor01 CRON[24416]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 18:57:09 at-vie01a-tor01 systemd[1]: Stopped target Timers. > > > > > > at-vie01a-tor02.ph3x.at: > > > > Jan 25 10:17:01 at-vie01a-tor02 CRON[19123]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 11:17:01 at-vie01a-tor02 CRON[23992]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Heartbeat: > Tor's uptime is 11:59 hours, with 38342 circuits open. I've sent 218.94 GB > and received 192.87 GB. > > Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Circuit > handshake stats since last time: 88566/90206 TAP, 12299213/12353484 NTor. > > Jan 25 11:59:10 at-vie01a-tor02 Tor-86.59.119.88_443[509]: Since > startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 > connections, and 6295 v4 connections; and receiv > > ed 60 v1 connections, 3085 v2 connections, 6040 v3 connections, and > 300693 v4 connections. > > Jan 25 12:17:01 at-vie01a-tor02 CRON[28856]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 12:18:28 at-vie01a-tor02 kernel: [44384.957658] TCP: > request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. > Check SNMP counters. > > Jan 25 13:17:01 at-vie01a-tor02 CRON[28998]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 14:17:01 at-vie01a-tor02 CRON[29061]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 14:36:57 at-vie01a-tor02 ntpd[490]: Soliciting pool server > 185.144.161.170 > > Jan 25 14:37:05 at-vie01a-tor02 ntpd[490]: Soliciting pool server > 2a02:1748:0:1500:3::2005 > > Jan 25 14:37:16 at-vie01a-tor02 ntpd[490]: error resolving pool > 1.debian.pool.ntp.org: Temporary failure in name resolution (-3) > > Jan 25 14:37:36 at-vie01a-tor02 ntpd[490]: error resolving pool > 0.debian.pool.ntp.org: Temporary failure in name resolution (-3) > > > > > > at-vie01a-tor03.ph3x.at: > > > > Jan 24 13:21:18 at-vie01a-tor03 systemd[1]: apt-daily.timer: Adding 9h > 35min 45.841113s random time. > > Jan 24 13:24:24 at-vie01a-tor03 kernel: [62694.054136] TCP: > request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. > Check SNMP counters. > > Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Heartbeat: > Tor's uptime is 17:59 hours, with 0 circuits open. I've sent 314.59 GB and > received 317.28 GB. > > Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Circuit > handshake stats since last time: 35924/36525 TAP, 10006455/10027375 NTor. > > Jan 24 13:59:57 at-vie01a-tor03 Tor-78.142.140.242_443[517]: Since > startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 > connections, and 11906 v4 connections; and received 47 v1 connections, 4092 > v2 connections, 8578 v3 connections, and 409960 v4 connections. > > Jan 24 14:17:01 at-vie01a-tor03 CRON[21604]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 24 15:17:01 at-vie01a-tor03 CRON[21670]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 24 15:41:25 at-vie01a-tor03 ntpd[492]: error resolving pool > 1.debian.pool.ntp.org: Temporary failure in name resolution (-3) > > > > > > at-vie01a-tor04.ph3x.at: > > > > Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Heartbeat: > Tor's uptime is 11:59 hours, with 36596 circuits open. I've sent 192.29 GB > and received 194.32 GB. > > Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Circuit > handshake stats since last time: 84713/86100 TAP, 11892935/11941101 NTor. > > Jan 25 12:00:52 at-vie01a-tor04 Tor-188.118.198.244_443[510]: Since > startup, we have initiated 0 v1 connections, 0 v2 connections, 0 v3 > connections, and 5938 v4 connections; and rec > > eived 44 v1 connections, 2949 v2 connections, 5395 v3 connections, and > 287856 v4 connections. > > Jan 25 12:17:01 at-vie01a-tor04 CRON[29820]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 12:29:17 at-vie01a-tor04 kernel: [44936.694001] hrtimer: > interrupt took 12598111 ns > > Jan 25 13:17:01 at-vie01a-tor04 CRON[2275]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 14:17:01 at-vie01a-tor04 CRON[7180]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 14:27:14 at-vie01a-tor04 kernel: [52013.316026] TCP: > request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. > Check SNMP counters. > > Jan 25 15:17:01 at-vie01a-tor04 CRON[7920]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 16:17:01 at-vie01a-tor04 CRON[8023]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > > Jan 25 16:42:38 at-vie01a-tor04 ntpd[492]: Soliciting pool server > 81.16.38.161 > > Jan 25 16:42:46 at-vie01a-tor04 ntpd[492]: error resolving pool > 3.debian.pool.ntp.org: Temporary failure in name resolution (-3) > > > >>>>>> LOGS END <<<<< > > > > _______________________________________________ > > tor-relays mailing list > > tor-relays@lists.torproject.org > > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays > > _______________________________________________ > tor-relays mailing list > tor-relays@lists.torproject.org > https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays >