On 2014-11-05 10:47, Libertas wrote:
I appreciate your interest! Also, I hope I'm not speaking with too much authority. If anyone here has more OpenBSD experience than me, please send addendums or corrections.
Maybe call this an addendum? Some version of the following work in progress is going onto our local documentation store for others maintaining our OpenBSD relays.
It's a bit long-winded for inclusion in doc/TUNING per https://trac.torproject.org/projects/tor/ticket/13702 , as it's intended to educate *BSD and Linux sysadmins about a smidgen of the why behind the tuning recommendations, as well as point at further exploration.
Richard
------- Our OpenBSD tuning for Tor involves: 1) Certainly boosting the per-process allowed number of open files per Tor relay to accommodate Tor circuits, 2) Probably increasing the system maximum number of open files to accommodate Tor circuits, 3) Probably boosting the pf state table capacity to accommodate Tor circuits, guard, and exit traffic, and 4) Probably running more than one relay on a host to use available CPU cores and bandwidth.
1) Per-process open files (openfiles-max)
If you leave the per-process default of 1024 open files in place, even a moderately busy Tor relay will eventually log messages telling you it's using all available file descriptors, meaning relay throughput is constrained artificially.
Here are a couple ways that work for boosting openfiles-max for Tor without allowing other processes to run away.
a) At the end of /etc/login.conf: -----8<----- # For Tor, set an openfiles-max to override probable default openfiles-max # of 1024 tor:\ :openfiles-max=8192:\ :tc=daemon: -----8<----- The name 'tor' for this stanza comes from the pkg_scripts entry in /etc/rc.conf.local.
b) Before the line "rc_cmd $1" in /etc/rc.d/tor: -----8<----- ulimit -n 8192 -----8<----- This line will need to be re-added when|if the /etc/rc.d/tor script is replaced during the course a package upgrade.
2) System maximum open files (kern.maxfiles)
The system maximum number of open files allowed (kern.maxfiles) has a default calculated per architecture (7030 for amd64 [1]). This default may be exceeded on a moderately busy Tor relay. It can be adjusted with sysctl(8).
Check 'sysctl kern.nfiles' under load to see how many file descriptors are in use. We regularly find > 7000 open files in use on a long-running relay that does nothing but Tor.
Roughly doubling the amd64 default kern.maxfiles to 2^14=16384 might be good if you're hitting the limit.
To change it until reboot: sudo sysctl -w kern.maxfiles=16384
kern.maxfiles can be set persistently by adding a line to /etc/sysctl.conf as follows: -----8<----- kern.maxfiles=16384 # bump kern.maxfiles from ${ARCH} default for Tor -----8<-----
--- [1] Here's one description of the amd64 default kern.maxfiles calculation: http://unix.stackexchange.com/questions/104929/does-openbsd-have-a-limit-to-...
3) pf state table capacity
pf is on by default in OpenBSD, keeping state for network flows. Rather than turning it off or writing rules with 'quick' and 'no state' for your Tor ports (the state table will be faster than rule parsing), you may want to adjust its capacity. This will be especially useful if you're running a popular Tor relay or set of relays.
Examine your pf resource limits: sudo pfctl -sm Examine your pf usage under load: sudo pfctl -si
If your 'state-limit', 'src-limit', and/or 'memory' counters are incrementing, your host is almost certainly dropping packets from time to time due to resource limits on state table storage.
For example, this kind of output from pfctl -si is probably bad for your throughput: memory 8986091 0.9/s
If your relay is used widely, especially as a guard, or you have multiple relays sharing pf's limit, you may also find the "current entries" in your state table (per pfctl -si) getting close to the default limit of 10k (per pfctl -sm).
Boost the limits as needed. One suggestion for the options setting section of your /etc/pf.conf is to boost the defaults by 10x or so: -----8<----- set limit { states 100000, src-nodes 100000, frags 8000 } -----8<----- Then continue monitoring the usage
If you're worried about memory use for the extra state table entries, 256 MB should be more than enough for 500k entries. If your kernel is severely memory constrained (e.g., you're running on older i386 hardware with a total of 512 MB RAM), you can also explore expiring TCP states faster (see sudo pfctl -st, maybe drop tcp.closed from 90s to 10s or the like).
For more info, here are writeups (the second with RRDTool monitoring suggestions): http://serverascode.com/2011/09/12/openbsd-pf-set-limit-states.html http://www.skeptech.org/blog/2013/01/15/pf-limits-in-openbsd/
4) Loading more CPU cores
If you have one of your CPUs maxed out running a Tor relay, with the other CPU(s) mostly idle (see top(1)), yet you have bandwidth to spare still, you can run additional Tor instances to sop some of it up.
The sanest way to handle this is to make each relay a stand-alone entity with a naming scheme to keep them straight. Here, we'll use "tor#" for every relay past the first.
Make per-relay directories in /var owned by _tor:_tor mode 700 drwx------ 5 _tor _tor 512 Jan 13 18:52 /var/tor/ drwx------ 5 _tor _tor 512 Jan 13 22:39 /var/tor2/ drwx------ 5 _tor _tor 512 Jan 13 22:39 /var/tor3/ ... Copy the tor startup script /etc/rc.d/tor to match the naming scheme. /etc/rc.d/tor2 /etc/rc.d/tor3 ... Copy the torrc from /etc/tor/torrc. /etc/tor/torrc2 /etc/tor/torrc3 ... Modify /etc/tor/torrc2, /etc/tor/torrc3, ... so they refer to their appropriate private DataDirectory and PidFile, listen on the appropriate ports and IP addresses, and have the appropriate exit policies. (Remember that the public Tor network will by design ignore more than two relays per IP address.) DataDirectory /var/tor2 PidFile /var/tor2/pid ControlPort 9222 Address 10.2.2.2 ORPort 8222 DirPort 7222 ... DataDirectory /var/tor3 PidFile /var/tor3/pid ControlPort 9333 Address 10.3.3.3 ORPort 8333 DirPort 7222 ... Set each relay to launch at system startup via the named /etc/rc.d scripts in /etc/rc.conf.local's pkg_scripts. tor_flags="${tor_flags} -f /etc/tor/torrc" tor2_flags="${tor2_flags} -f /etc/tor/torrc2" tor3_flags="${tor3_flags} -f /etc/tor/torrc3" ... pkg_scripts=" ... tor tor2 tor3 ..." Set openfiles-max for each named pkg_script from /etc/rc.conf.log in /etc/login.conf. tor:\ :openfiles-max=8192:\ :tc=daemon: tor2:\ :openfiles-max=8192:\ :tc=daemon: tor3:\ :openfiles-max=8192:\ :tc=daemon: ... Remember to allow inbound traffic to the additional ports set in /etc/tor/torrc[#] in your /etc/pf.conf.