Failing because we have 4063 connections already // Number of file descriptors
Since a couple of days ago, one of my relay nodes keeps logging messages like this: Tor[3534]: Failing because we have 4063 connections already. Please read doc/TUNING for guidance. [over 16000001 similar message(s) suppressed in last 21600 seconds] I found https://trac.torproject.org/projects/tor/ticket/16929 and an older mailing list thread (and doc/TUNING) that suggest increasing the maximum number of open file descriptors. I now use # /etc/security/limits.conf * - nofile 65535 to raise 'nofile' from 1024 to 65535, which does not seem to make any difference (the logged error message does not change). My relay uses Gentoo Linux kernel version 4.14.5 and Tor 0.3.2.6 alpha. I also tried older versions of the kernel and Tor 0.3.1.9, in several combinations, without success. I also other relays running just fine with the default number of file descriptors (1024). As I mentioned, the problems started a few days ago, around the time I upgraded from Gentoo system profile default/linux/amd64/13.0 to default/linux/amd64/17.0, rebuilding many system packages in the process. Unfortunately I cannot tell what exactly changed. Since the profile upgrade was deliberate and I cannot roll back, I wonder if I can overcome Tor's problems via configuration options only? -Ralph
Hi Please verify the effective limit used for your tor process: cat /proc/<pid>/limits with <pid> process id of the tor process. could be that your tuning is not being picked up by the distro. Regards On Fri, 15 Dec 2017 at 10:39 Ralph Seichter <m16+tor@monksofcool.net> wrote:
Since a couple of days ago, one of my relay nodes keeps logging messages like this:
Tor[3534]: Failing because we have 4063 connections already. Please read doc/TUNING for guidance. [over 16000001 similar message(s) suppressed in last 21600 seconds]
I found https://trac.torproject.org/projects/tor/ticket/16929 and an older mailing list thread (and doc/TUNING) that suggest increasing the maximum number of open file descriptors. I now use
# /etc/security/limits.conf * - nofile 65535
to raise 'nofile' from 1024 to 65535, which does not seem to make any difference (the logged error message does not change).
My relay uses Gentoo Linux kernel version 4.14.5 and Tor 0.3.2.6 alpha. I also tried older versions of the kernel and Tor 0.3.1.9, in several combinations, without success. I also other relays running just fine with the default number of file descriptors (1024).
As I mentioned, the problems started a few days ago, around the time I upgraded from Gentoo system profile default/linux/amd64/13.0 to default/linux/amd64/17.0, rebuilding many system packages in the process. Unfortunately I cannot tell what exactly changed. Since the profile upgrade was deliberate and I cannot roll back, I wonder if I can overcome Tor's problems via configuration options only?
-Ralph _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On 15.12.2017 10:45, r1610091651 wrote:
could be that your tuning is not being picked up by the distro.
Looks like you are right: # cat /proc/3534/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 3969 3969 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 3969 3969 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Only 4096 open files max. I did not expect that, because of tor $ ulimit -n 65535 tor $ cat /proc/sys/fs/file-max 101300 What part of the system's configuration could cause the 4096 open files limit, overriding the 65535 I specified in limits.conf? -Ralph
That depends on how tor is started and have different origins. What i know: * if started by systemd: the limit can be specified in the service descripton file /lib/systemd/system/tor@default.service: => LimitNOFILE=xxxx * if started by init: usually pam/security will be applied => using limits.conf and propagated by pam: /etc/pam.d/common-* There are probably also other paths. Regards On Fri, 15 Dec 2017 at 11:03 Ralph Seichter <m16+tor@monksofcool.net> wrote:
On 15.12.2017 10:45, r1610091651 wrote:
could be that your tuning is not being picked up by the distro.
Looks like you are right:
# cat /proc/3534/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 3969 3969 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 3969 3969 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Only 4096 open files max. I did not expect that, because of
tor $ ulimit -n 65535 tor $ cat /proc/sys/fs/file-max 101300
What part of the system's configuration could cause the 4096 open files limit, overriding the 65535 I specified in limits.conf?
-Ralph _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Fri, 15 Dec 2017, 11:13 r1610091651, <r1610091651@telenet.be> wrote:
That depends on how tor is started and have different origins. What i know: * if started by systemd: the limit can be specified in the service descripton file /lib/systemd/system/tor@default.service: => LimitNOFILE=xxxx * if started by init: usually pam/security will be applied => using limits.conf and propagated by pam: /etc/pam.d/common-*
There are probably also other paths.
Regards
On Fri, 15 Dec 2017 at 11:03 Ralph Seichter <m16+tor@monksofcool.net> wrote:
On 15.12.2017 10:45, r1610091651 wrote:
could be that your tuning is not being picked up by the distro.
Looks like you are right:
# cat /proc/3534/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 3969 3969 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 3969 3969 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
Only 4096 open files max. I did not expect that, because of
tor $ ulimit -n 65535 tor $ cat /proc/sys/fs/file-max 101300
What part of the system's configuration could cause the 4096 open files limit, overriding the 65535 I specified in limits.conf?
-Ralph _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
Short term you can modify limits on running process with prlimit
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 12/15/2017 10:38 AM, Ralph Seichter wrote:
My relay uses Gentoo Linux kernel version If you run a Gentoo system then take a look at this file :
# cat /etc/conf.d/tor # # Set the file limit rc_ulimit="-n 30000" - -- Toralf PGP C4EACDDE 0076E94E -----BEGIN PGP SIGNATURE----- iI0EAREIADUWIQQaN2+ZSp0CbxPiTc/E6s3eAHbpTgUCWjOgIhccdG9yYWxmLmZv ZXJzdGVyQGdteC5kZQAKCRDE6s3eAHbpThXAAPkBU2AvT/P8wI7LsLB89FX5c0sF SAHnfH94V+1fmSp4mgD/YmhdzdTf6B0+zYivdP2fcY44aZpKwsMACPp9FDI0sbE= =d9Fo -----END PGP SIGNATURE-----
On 15.12.2017 11:12, Toralf Förster wrote:
# cat /etc/conf.d/tor # # Set the file limit rc_ulimit="-n 30000"
Ah, thanks a lot! Limits were implied by openrc-run/start-stop-daemon, overriding my limits.conf entries. Turns out that /etc/conf.d/tor got deleted, although I have no idea how that happened. I built Tor from the official source tarball and created my own versions of /etc/init.d/tor and /etc/conf.d/tor (both files are part of Gentoo's net-vpn/tor package). I manually re-created /etc/conf.d/tor and verified that the max number of open file descriptors matches rc_ulimit. The relay has now been up for half an hour without logging errors. Thanks again, to both Toralf and r1610091651. -Ralph
participants (3)
-
r1610091651 -
Ralph Seichter -
Toralf Förster