July 2015 - tor-relays - lists.torproject.org

Giving away some "pre-warmed" relay keys for adoption
by Roman Mamedov 29 Jul '15

29 Jul '15

Hello, If anyone is planning to spin up a new VM or dedi to run a Tor relay and want it to be put instantly into good use (without wasting couple of weeks to a month for the whole "unmeasured relay/measured/guard/stable" cycle to complete) or if you are having trouble with your current relay being measured properly, in a few days I'm considering to give away several fingerprints+keys from my relays that I will be shutting down. If you install Tor on a new machine and unpack one of these into your /var/lib/tor, it starts up something like this (see below); i.e. in a couple of hours to a full 100% utilization. eth0 16:59 ^ t | t t rt rt rt rt | rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt rt rt rt | rt rt rt rt rt rt rt rt rt rt rt -+---------------------------------------------------------------------------> | 17 18 19 20 21 22 23 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 h rx (KiB) tx (KiB) h rx (KiB) tx (KiB) h rx (KiB) tx (KiB) 17 0 0 01 0 0 09 34135840 34964598 18 0 0 02 0 0 10 37991652 38890814 19 0 0 03 0 0 11 38710878 39749495 20 0 0 04 0 0 12 39168357 40164982 21 0 0 05 50433 1472 13 42351426 43626118 22 0 0 06 10904079 10927593 14 42826283 43928095 23 0 0 07 15691530 15839439 15 41240931 42507372 00 0 0 08 27079433 27403035 16 41941981 43139517 Drop me an E-Mail if you want one of these (probably about 2-3 available for now). -- With respect, Roman

8 14

Re: [tor-relays] pinning relay keys to IPs (or not)
by starlight.2015q2＠binnacle.cx 29 Jul '15

29 Jul '15

>(where a lot of IPs changed their AS from >IANA to Digital Ocean) A couple of minor notes regarding ASNs: 1) many IPs fall under a hierarchy of ASs where a large core-network provider (e.g. Level3) advertises a block and a second client leaf-AS advertises a sub- block. Sometimes the core AS advertises the smaller blocks though that has diminished with the CIDR route consolidation initiative. Also some ASs advertise bocks and sub-blocks. This shows up often with the CYMRU lookup data dig +short D.C.B.A.origin.asn.cymru.com txt and DNS will rotate the multiple advertisements, so one should sort the list by CIDR size and select the smallest block (i.e. largest CIDR "/" value). Possibly MaxMind takes care of all this in their data. 2) one can likely ignore AS changes when the IP has not changed, thus avoiding problems caused by network restructuring 3) perhaps many dynamic allocations where the IP changes to different AS can be detected by examining the AS owner identifiers and looking for a match I agree guards are special and perhaps should not be allowed to change ASs at all without loosing the flag, maybe even should stay glued to one IP to avoid any failed client connections and the negative impact that may have on anonymity. It seems reasonable to allow dynamic-IP middle and exit relays.

2 2

pinning relay keys to IPs (or not)
by nusenu 29 Jul '15

29 Jul '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 [split from 'Giving away some "pre-warmed" relay keys for adoption'] > I'm of the opinion that it may be worth adding code to pin relay > identities to IP addresses on the DirAuth side so that consensus > weight and flag assignment gets totally reset if the ORPort IP > changes, but if there's too much churn already it may cause more > trouble than it's worth. I hope such code will not be added, because it renders relays on dynamic IPs basically useless. In the past ~week only there were >1000 fingerprints (<3% cw fraction) using more than one IP address (in that timeframe) > I'm somewhat torn on the whole key pinning thing, because I think > an individual operator moving their relay around is sort of ok > (though in an ideal world the consensus weight should get reset and > rapidly re-measured), but giving away the private component of a > relay's identity key is putting users at risk, and is behavior that > should be discouraged if not outright prohibited if possible (and > key pinning would be a heavy handed way to rule out this sort of > stupidity). -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJVtOqsAAoJEFv7XvVCELh0N2oQAJYXVGM0hHJzBl3i7tkfZx8E hEd9PgyX0v+nXnlAU06lsFltK12VaCkXp9Gft8TlmshP6NqsdHk+KPrtumUE7IA4 MmYmmVi5Tb8AyN241nr/RbYOUS+9nz2VvtkhaikkvbFxwzua678I13P1PC/QPki3 S4KwQ0YYYlUrG529GhLaGkm5byX3N+MAdGQn04xzCDzBW/j3GxbZYBj4+tqgHONR 9iAs5/p00MFVGp3Ex8xJKBVRW0NRT//fKRrZtRDBaNxduUg7F0EccO4zlPjTUBAM e+g8tergT4ICixddhRih0iGA3ysw3/mI1KyvOMegZXkUQ0Yo+puR4O7S2w4WoWRh 1tkkbcFWAh4sCKDPLlD/sqWKFDkHLtP3AOG+KGgybkhULNU1GgAftV39QVrsABx/ oP/dhC9d+MablSZg6qS8dxTV3L+E5EdE2RX226+BPjRkUSqphojEYjDpiaR3/COU 2BmACmPIuiS60DfaiqygKQaCwvgDl6q96o5oZEmPHem2SIxh5o3DD1wLJgKKEsiL WgmoRHTLfqc2Uq0442oi7KMhgGcIffS8xKbb1xw/9yIF2WiKrluFFWau5k6cQYA2 0buDAz7WWi7kxUCnn39NcyrtWoOWMziRH+rjv5esoRN7W5k2GNtBcylZ6kyzcudr hy7j6rYMLpUHsVc2NHKd =8W7p -----END PGP SIGNATURE-----

5 9

Very unbalanced in/out connection ratio
by mattia 29 Jul '15

29 Jul '15

Hello, I've been running a middle relay [1] for months and I have always noticed an unbalance between incoming and outgoing connections (monitored through arm). The incoming connections were always in between 0-10% more than the outgoing ones. This morning though the unbalance has reached so far unseen levels: arm reports more than 2500 incoming connections while outgoing are little more than 200. Is this normal and, if so, why? Thank you, Mattia [1] https://globe.torproject.org/#/relay/0D14A4498E7D1854696CC07548653EA4CD3CB2…

2 2

Re: [tor-relays] longclaw BWauth is broken -- no measurements since 7/22
by starlight.2015q2＠binnacle.cx 28 Jul '15

28 Jul '15

Longclaw is broken again. Has not updated 55% of the the relay BS measurments for forty hours. Updated the ticket below with details and raise it to "major" issue. At 16:42 7/25/2015 -0400, you wrote: >Realized not everyone monitors this >list so I opened a Trac ticket: > >https://trac.torproject.org/projects/tor/ticket/16667 > > > >At 11:19 7/25/2015 -0700, you wrote: >>Interesting. Thanks starlight for the head's up! I'm not >>aware of any known issues with longclaw right now >>so looping in its operator. >> >> >>On Sat, Jul 25, 2015 at 10:09 AM, >><starlight.2015q2(a)binnacle.cx> wrote: >>> Subject says it. >>> >>> Thought it was strange that its measurement >>> of my relay has not been moving. Pulled the >>> vote history and grep'ed a dozen relay >>> measurements and none of them have changed. >>>

1 0

pinning relay keys to IPs (or not)
by starlight.2015q2＠binnacle.cx 26 Jul '15

26 Jul '15

Perhaps a way to do it is reset the consensus for a relay if its IP address moves to a different Autonomous System. Is rare that dynamic IP causes relays to hop ASs (e.g. possibly SBC/ATT), and list of exceptions could be created for the few cases where it causes trouble. CYMRU has a dynamic service for looking up AS from IP.

3 2

Re: [tor-relays] longclaw BWauth is broken -- no measurements since 7/22
by starlight.2015q2＠binnacle.cx 25 Jul '15

25 Jul '15

Realized not everyone monitors this list so I opened a Trac ticket: https://trac.torproject.org/projects/tor/ticket/16667 At 11:19 7/25/2015 -0700, you wrote: >Interesting. Thanks starlight for the head's up! I'm not >aware of any known issues with longclaw right now >so looping in its operator. > > >On Sat, Jul 25, 2015 at 10:09 AM, ><starlight.2015q2(a)binnacle.cx> wrote: >> Subject says it. >> >> Thought it was strange that its measurement >> of my relay has not been moving. Pulled the >> vote history and grep'ed a dozen relay >> measurements and none of them have changed. >>

1 0

longclaw BWauth is broken -- no measurements since 7/22
by starlight.2015q2＠binnacle.cx 25 Jul '15

25 Jul '15

Subject says it. Thought it was strange that its measurement of my relay has not been moving. Pulled the vote history and grep'ed a dozen relay measurements and none of them have changed.

2 1

Re: [tor-relays] How to Run High Capacity Tor Relays
by Green Dream 24 Jul '15

24 Jul '15

Pardon the thread necromancy, but I'm wondering if this document ever made its way off this mailing list and onto a blog? Or perhaps there is some other modern doc covering this topic? I've recently setup a relay on a Gb/s fiber connection, and am struggling to understand how to optimize performance. It's not clear 5 years later which if any of the tweaks listed below are still relevant. I'm running a modern debian-based system. Thanks in advance. > After talking to Moritz and Olaf privately and asking them about their > nodes, and after running some experiments with some high capacity > relays, I've begun to realize that running a fast Tor relay is a > pretty black art, with a lot of ad-hoc practice. Only a few people > know how to do it, and if you just use Linux and Tor out of the box, > your relay will likely underperform on 100Mbit links and above. > In the interest of trying to help grow and distribute the network, my > ultimate plan is to try to collect all of this lore, use Science to > divine out what actually matters, and then write a more succinct blog > post about it. > However, that is a lot of work. It's also not totally necessary to do > all this work, when you can get a pretty good setup with a rough > superset of all of the ad-hoc voodoo. This post is thus about that > voodoo. > Hopefully others will spring forth from the darkness to dump their own > voodoo in this thread, as I suspect there is one hell of a lot of it > out there, some (much?) of which I don't yet know. Likewise, if any > blasphemous heretic wishes to apply Science to this voodoo, they > should yell out, "Stand Back, I'm Doing Science!" (at home please, not > on this list) and run some experiments to try to eliminate options > that are useless to Tor performance. Or cite academic research papers. > (But that's not Science, that's computerscience - which is a religion > like voodoo, but with cathedrals). > Anyway, on with the draft: > > == Machine Specs == > First, you want to run your OS in x64 mode because openssl should do > crypto faster in 64bit. > Tor is currently not fully multithreaded, and tends not to benefit > beyond 2 cores per process. Even then, the benefit is still marginal > beyond just 1 core. 64bit Tor nodes require about one 2Ghz Xeon/Core2 > core per 100Mbit of capacity. > Thus, to fill an 800Mbit link, you need at least a dual socket, quad > core cpu config. You may be able to squeeze a full gigabit out of one > of these machines. As far as I know, no one has ever done this with > Tor, on any one machine. > The i7's also just came out in this form factor, and can do > hyperthreading (previous models may list 'ht' in cpuinfo, but actually > don't support it). This should give you a decent bonus if you set > NumCPUs to 2, since ht tends to work better with pure integer math > (like crypto). We have not benchmarked this config yet though, but I > suspect it should fill a gigabit link fairly easily, possibly > approaching 2Gbit. > At full capacity, exit node Tor processes running at this rate consume > about 500M of ram. You want to ensure your ram speed is sufficient, > but most newish hardware is good. Using on this chart: > > https://secure.wikimedia.org/wikipedia/en/wiki/List_of_device_bandwidths#Me… > you can do the math and see that with a dozen memcpys in each > direction, you come out needing DDR2 to be able to push 1Gbit full > duplex. > As far as ethernet cards, the Intel e1000e *should* be theoretically > good, but they seem to fail at properly irq balancing across multiple > CPUs on recent kernels, which can cause you to bottleneck at 100% CPU > on one core. At least that has been Moritz's experience. In our > experiments, the RTL-8169 works fine (once tweaked, see below). > > == System Tweakscript Wibbles and Config Splatters == > First, you want to ensure that you run no more than 2 Tor instances > per IP. Any more than this and clients will ignore them. > Next, paste the following smattering into the shell (or just read it > and make your own script): > # Set the hard limit of open file descriptors really high. > # Tor will also potentially run out of ports. > ulimit -SHn 65000 > # Set the txqueuelen high, to prevent premature drops > ifconfig eth0 txqueuelen 20000 > # Tell our ethernet card (interrupt found from /proc/interrupts) > # to balance its IRQs across one whole CPU socket (4 cpus, mask 0f). > # You only want one socket for optimal ISR and buffer caching. > # > # Note that e1000e does NOT seem to obey this, but RTL-8169 will. > echo 0f > /proc/irq/17/smp_affinity > # Make sure you have auxiliary nameservers. I've seen many ISP > # nameservers fall over under load from fast tor nodes, both on our > # nodes and from scans. Or run caching named and closely monitor it. > echo "nameserver 8.8.8.8" >> /etc/resolv.conf > echo "nameserver 4.2.2.2" >> /etc/resolv.conf > # Load an amalgam of gigabit-tuning sysctls from: > # http://datatag.web.cern.ch/datatag/howto/tcp.html > # http://fasterdata.es.net/TCP-tuning/linux.html > # http://www.acc.umu.se/~maswan/linux-netperf.txt > # http://www.psc.edu/networking/projects/tcptune/#Linux > # and elsewhere... > # We have no idea which of these are needed yet for our actual use > # case, but they do help (especially the nf-contrack ones): > sysctl -p << EOF > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_wmem = 4096 65536 16777216 > net.core.netdev_max_backlog = 2500 > net.ipv4.tcp_no_metrics_save = 1 > net.ipv4.tcp_moderate_rcvbuf = 1 > net.core.rmem_max = 1048575 > net.core.wmem_max = 1048575 > net.ipv4.ip_local_port_range = 1025 61000 > net.ipv4.tcp_synack_retries = 3 > net.ipv4.tcp_tw_recycle = 1 > net.ipv4.tcp_max_syn_backlog = 10240 > net.ipv4.tcp_fin_timeout = 30 > net.ipv4.tcp_keepalive_time = 1200 > net.netfilter.nf_conntrack_tcp_timeout_established=7200 > net.netfilter.nf_conntrack_checksum=0 > net.netfilter.nf_conntrack_max=131072 > net.netfilter.nf_conntrack_tcp_timeout_syn_sent=15 > net.ipv4.tcp_keepalive_time=60 > net.ipv4.tcp_keepalive_time = 60 > net.ipv4.tcp_keepalive_intvl = 10 > net.ipv4.tcp_keepalive_probes = 3 > net.ipv4.ip_local_port_range = 1025 65530 > net.core.netdev_max_backlog=300000 > net.core.somaxconn=20480 > net.ipv4.tcp_max_tw_buckets=2000000 > net.ipv4.tcp_timestamps=0 > vm.min_free_kbytes = 65536 > net.ipv4.ip_forward=1 > net.ipv4.tcp_syncookies = 1 > net.ipv4.tcp_synack_retries = 2 > net.ipv4.conf.default.forwarding=1 > net.ipv4.conf.default.proxy_arp = 1 > net.ipv4.conf.all.rp_filter = 1 > net.ipv4.conf.default.send_redirects = 1 > net.ipv4.conf.all.send_redirects = 0 > EOF > # XXX: ethtool wibbles > # You may also have to tweak some parameters with ethtool, possibly > # also enabling some checksum offloading or irq coalescing options to > # spare CPU, but for us this hasn't been needed yet. > > == Setting up the Torrc == > Basically you can just read through the stock example torrc, but there > are some as-yet undocumented magic options, and options that need new > defaults. > # NumCPUs doesn't provide any benefit beyond 2, and setting it higher > # may cause cache misses. > NumCPUs 2 > # These options have archaic maximums of 2-5Mbyte > BandwidthRate 100 MB > BandwidthBurst 200 MB > > == Waiting for the Bootstrap and Measurement Process == > Perhaps the most frustrating part of this setup is how long it takes > for you to acquire traffic. If you are starting new at an ISP, I would > consider only 200-400Mbit for your first month. Hitting that by the > end of the month may be a challenge, mostly because their may be dips > and total setbacks along the way. > The slow rampup is primarily due to limitations in Tor's ability to > rapidly publish descriptor updates, and to measure relays. > It ends up taking about 2-3 days to hit an observed bandwidth of > 2Mbyte/sec per relay, but it can take well over a week or more (Moritz, do > you have a better number?) to reach 8-9Mbyte/relay. This is for an > Exit node. A middle node will likely gather traffic slower. Also once > you crash, you lose it. This bug is about that issue: > https://trac.torproject.org/projects/tor/ticket/1863 > There is also a potential dip when you get the Guard flag, as our load > balancing formulas try to avoid you, but no clients have chosen you > yet. Changes to the authority voting on Guards in Tor 0.2.2.15 should > make this less drastic. It is also possible that your observed > bandwidth will be greater than without it. However, it will still take > up to 2 months for clients to choose you as their new Guard. > > == Running temporary auxiliary nodes == > One way to shortcut this process and avoid paying for bandwidth you > don't use is to spin up a bunch of temporary nodes to utilize the CPU > and quickly gather that easy first 2MB/sec of observed bandwidth. > But you need the spare IPs to do this. > > == Monitoring == > Personally, I prefer console-based options like nload, top, and > Damian's arm (http://www.atagar.com/arm/) because I don't like the > idea of running extra services to publish my monitoring data to the > world. > Other people have web-based monitoring using things like munin and > mrtg. It would be nice to get a script/howto for that too. > > == Current 1-Box Capacity Record == > Our setup has topped at 450Mbit, but averages between 300-400Mbit. We > are currently having uptime issues due to heat (melting, poorly > ventilated harddrives). It is likely that once we resolve this, we > will continually increase to our CPU ceiling. > I believe Moritz and Olaf also push this much capacity, possibly a bit > more, but with less nodes (4 as opposed to our 8). I hear Jake is also > ramping up some Guard nodes (or maybe I didn't? Did I just betray you > again Jake?) > > == Did I leave anything out? == > Well, did I? > > -- > Mike Perry > Mad Computer Scientist > fscked.org evil labs

8 11

Running a relay in the Netherlands
by TorOps 24 Jul '15

24 Jul '15

Hello, I am about to move to the Netherlands and will, as part of the move, decommission the server my relay is currently running on. I'd like to be able to continue to help, though, so I wonder how the legal climate is for Tor relays and/or exit nodes in the Netherlands? On a related topic, how smart/efficient/possible is it to run a relay/exit on a Raspberry Pi? Thanks for any info. -- Regards, StairNetTor Operator

6 7