[tor-relays] Platform diversity in Tor network [was: OpenBSD doc/TUNING]
Richard Johnson
rdump at river.com
Tue Dec 16 07:11:32 UTC 2014
On 2014-11-05 10:47, Libertas wrote:
> I appreciate your interest! Also, I hope I'm not speaking with too
> much authority. If anyone here has more OpenBSD experience than me,
> please send addendums or corrections.
Maybe call this an addendum? Some version of the following work in progress
is going onto our local documentation store for others maintaining our
OpenBSD relays.
It's a bit long-winded for inclusion in doc/TUNING per
https://trac.torproject.org/projects/tor/ticket/13702 , as it's intended to
educate *BSD and Linux sysadmins about a smidgen of the why behind the
tuning recommendations, as well as point at further exploration.
Richard
-------
Our OpenBSD tuning for Tor involves:
1) Certainly boosting the per-process allowed number of open files per
Tor relay to accommodate Tor circuits,
2) Probably increasing the system maximum number of open files to
accommodate Tor circuits,
3) Probably boosting the pf state table capacity to accommodate Tor
circuits, guard, and exit traffic, and
4) Probably running more than one relay on a host to use available CPU
cores and bandwidth.
1) Per-process open files (openfiles-max)
If you leave the per-process default of 1024 open files in place, even a
moderately busy Tor relay will eventually log messages telling you it's
using all available file descriptors, meaning relay throughput is
constrained artificially.
Here are a couple ways that work for boosting openfiles-max for Tor without
allowing other processes to run away.
a) At the end of /etc/login.conf:
-----8<-----
# For Tor, set an openfiles-max to override probable default openfiles-max
# of 1024
tor:\
:openfiles-max=8192:\
:tc=daemon:
-----8<-----
The name 'tor' for this stanza comes from the pkg_scripts entry in
/etc/rc.conf.local.
b) Before the line "rc_cmd $1" in /etc/rc.d/tor:
-----8<-----
ulimit -n 8192
-----8<-----
This line will need to be re-added when|if the /etc/rc.d/tor script is
replaced during the course a package upgrade.
2) System maximum open files (kern.maxfiles)
The system maximum number of open files allowed (kern.maxfiles) has a
default calculated per architecture (7030 for amd64 [1]). This default may
be exceeded on a moderately busy Tor relay. It can be adjusted with
sysctl(8).
Check 'sysctl kern.nfiles' under load to see how many file descriptors are
in use. We regularly find > 7000 open files in use on a long-running relay
that does nothing but Tor.
Roughly doubling the amd64 default kern.maxfiles to 2^14=16384 might be good
if you're hitting the limit.
To change it until reboot: sudo sysctl -w kern.maxfiles=16384
kern.maxfiles can be set persistently by adding a line to /etc/sysctl.conf as
follows:
-----8<-----
kern.maxfiles=16384 # bump kern.maxfiles from ${ARCH} default for Tor
-----8<-----
---
[1] Here's one description of the amd64 default kern.maxfiles calculation:
http://unix.stackexchange.com/questions/104929/does-openbsd-have-a-limit-to-the-number-of-file-descriptors/104948#104948
3) pf state table capacity
pf is on by default in OpenBSD, keeping state for network flows. Rather than
turning it off or writing rules with 'quick' and 'no state' for your Tor
ports (the state table will be faster than rule parsing), you may want to
adjust its capacity. This will be especially useful if you're running a
popular Tor relay or set of relays.
Examine your pf resource limits: sudo pfctl -sm
Examine your pf usage under load: sudo pfctl -si
If your 'state-limit', 'src-limit', and/or 'memory' counters are
incrementing, your host is almost certainly dropping packets from time to
time due to resource limits on state table storage.
For example, this kind of output from pfctl -si is probably bad for your
throughput:
memory 8986091 0.9/s
If your relay is used widely, especially as a guard, or you have multiple
relays sharing pf's limit, you may also find the "current entries" in your
state table (per pfctl -si) getting close to the default limit of 10k (per
pfctl -sm).
Boost the limits as needed. One suggestion for the options setting section
of your /etc/pf.conf is to boost the defaults by 10x or so:
-----8<-----
set limit { states 100000, src-nodes 100000, frags 8000 }
-----8<-----
Then continue monitoring the usage
If you're worried about memory use for the extra state table entries, 256 MB
should be more than enough for 500k entries. If your kernel is severely
memory constrained (e.g., you're running on older i386 hardware with a total
of 512 MB RAM), you can also explore expiring TCP states faster (see sudo
pfctl -st, maybe drop tcp.closed from 90s to 10s or the like).
For more info, here are writeups (the second with RRDTool monitoring
suggestions):
http://serverascode.com/2011/09/12/openbsd-pf-set-limit-states.html
http://www.skeptech.org/blog/2013/01/15/pf-limits-in-openbsd/
4) Loading more CPU cores
If you have one of your CPUs maxed out running a Tor relay, with the other
CPU(s) mostly idle (see top(1)), yet you have bandwidth to spare still, you
can run additional Tor instances to sop some of it up.
The sanest way to handle this is to make each relay a stand-alone entity
with a naming scheme to keep them straight. Here, we'll use "tor#" for every
relay past the first.
Make per-relay directories in /var owned by _tor:_tor mode 700
drwx------ 5 _tor _tor 512 Jan 13 18:52 /var/tor/
drwx------ 5 _tor _tor 512 Jan 13 22:39 /var/tor2/
drwx------ 5 _tor _tor 512 Jan 13 22:39 /var/tor3/
...
Copy the tor startup script /etc/rc.d/tor to match the naming scheme.
/etc/rc.d/tor2
/etc/rc.d/tor3
...
Copy the torrc from /etc/tor/torrc.
/etc/tor/torrc2
/etc/tor/torrc3
...
Modify /etc/tor/torrc2, /etc/tor/torrc3, ... so they refer to their
appropriate private DataDirectory and PidFile, listen on the appropriate
ports and IP addresses, and have the appropriate exit policies. (Remember
that the public Tor network will by design ignore more than two relays per
IP address.)
DataDirectory /var/tor2
PidFile /var/tor2/pid
ControlPort 9222
Address 10.2.2.2
ORPort 8222
DirPort 7222
...
DataDirectory /var/tor3
PidFile /var/tor3/pid
ControlPort 9333
Address 10.3.3.3
ORPort 8333
DirPort 7222
...
Set each relay to launch at system startup via the named /etc/rc.d scripts in
/etc/rc.conf.local's pkg_scripts.
tor_flags="${tor_flags} -f /etc/tor/torrc"
tor2_flags="${tor2_flags} -f /etc/tor/torrc2"
tor3_flags="${tor3_flags} -f /etc/tor/torrc3"
...
pkg_scripts=" ... tor tor2 tor3 ..."
Set openfiles-max for each named pkg_script from /etc/rc.conf.log in
/etc/login.conf.
tor:\
:openfiles-max=8192:\
:tc=daemon:
tor2:\
:openfiles-max=8192:\
:tc=daemon:
tor3:\
:openfiles-max=8192:\
:tc=daemon:
...
Remember to allow inbound traffic to the additional ports set in
/etc/tor/torrc[#] in your /etc/pf.conf.
More information about the tor-relays
mailing list