Raspberry Pi Relay Node Performance and future Plans on Documentation and more
Hi Gordon and Matthias, I've split your discussion from the original thread "Running exit-node in Germany" and created a new one. I fully agree with you that the Raspberry Pi is the perfect device to let others run a Tor Relay Node very easily. What follows is a long mail about my experiences and more thoughts about the Pi as relay. On Thu, Aug 1, 2013 at 5:29 PM, Gordon Morehouse <gordon@morehouse.me>wrote:
Matthias Redies:
Ok that is good to know. Right know I will probably run it on 1-1.5 Mbps and later on 3-4 Mbps. What is the maximum your raspberry is capable to do? Please let me know if you publish your tutorial.
I had it pushing about 1.5Mbps and crashing only about once a week before I started having TCP connect floods and had to take it offline until I could pay attention for a while. I'm still tuning it. It crashed much, much more often before some basic tuning, though.
I'm running a relay node on a Raspberry Pi at my VDSL home connection for several weeks now (provider "Deutsche Telekom", connection is 50mbps down, 10 up) and I didn't run into such issues so far. There're already several howtos out there which describe how to setup a relay node on the Raspberry Pi. I used this German one as starting point: http://blog.weezerle.de/2013/01/11/raspberry-pi-als-tor-server/ My current configuration has only the following two non-default options: DisableDebuggerAttachment 0 AvoidDiskWrites 1 (Plus, I run it as relay only by disabling the socks proxy with "SocksPort 0".) Here's my node: https://atlas.torproject.org/#details/5AB2F49A1B79148E3A8F52808A00451DA00841... http://globe.rndm.de/#/relay/5AB2F49A1B79148E3A8F52808A00451DA00841B1 As you can see, it forwards on average between 2-4mbps (256-512 kB/sec) and there're peaks as high as 5.5mbps (~700 kB/s). The throughput varies depending on the day of week and time of day and therefore I believe the low average isn't the Raspberry's fault. I guess, if I set a higher advertised value, then I would see a more constant throughput closer to the peak value. Currently, it's set to 10mbps: RelayBandwidthRate 1024 KB RelayBandwidthBurst 1024 KB (Note that the Pi is definitely not able to forward 10mbps of Tor traffic, but it would be great to max it out 24/7.) The logged throughput is also consistent with what I saw with the console traffic monitor "nload" (suggested command line options for nicer units and less refreshes: nload -u K -U G -t 3000). I guess, I saw even higher peaks there. Monitoring everything with "arm" is also nice but far too CPU intensive. More information about my setup: - I have the B revision with 512 MB memory - running stock Raspbian - installed "tor" package 0.2.3.25-1 from the Raspbian repository - CPU is overclocked using "raspi-config" to 950 MHz Before I overclocked the Pi, I saw a similar average throughput. When you search for "rasp" at Tor Atlas or Globe (e.g. http://globe.rndm.de/#/search/query=rasp) , you can see that there're already more than 40 devices running. My node performs quite well compared to others and I would argue that it's in the top 5 among all Raspberry Pi nodes so far. To track down possible CPU problems, we should also compare the openssl benchmark results. The Raspberry Pi wiki has some reference values: http://elinux.org/RPi_Performance#Results_3 I've attached the summary output of "openssl speed" for my overclocked Pi (950 MHz) to this mail. As you can see, the numbers are higher than the reference one (I guess probably due to overclocking and different OpenSSL version). I've also attached the log of notices of the last 4 days (unfortunately no longer logs available). Within four days there was only one "Your computer is too slow to handle this many circuit creation requests!" warning and one "Failed to hand off onionskin.". In general I would say, I haven't seen any serious issues so far.
And my plan is to publish my results to the entire list, because at $35, Raspberry Pis can make *great* relays for slower home broadband, but they need a little tender loving care first. :)
I totally agree with you. Recently, there has been quite some buzz about the OnionPi howto (http://learn.adafruit.com/onion-pi/overview) but its main goal wasn't providing a long-running relay node. In my opinion, this should get addressed separately and made as simple as possible for novices. My guess is that many people want to contribute bandwidth, but they do not want to deal with Linux specifics (and definitely do not want to get into any trouble with the police). They need a plug-and-forget solution: attach the Pi to the router and leave it running in the closet for months. If it has to be restarted, Tor Weather can send them an email. What do you think about documenting the setup of the relay in the Tor wiki? https://trac.torproject.org/projects/tor/wiki I'm not very familiar with the Tor Project yet, but to me it looks like the wiki will be the best place. Other people can contribute as well and it will be visible enough. It would be good if there's an always up-to-date wiki page which could become the reference for new users. The idea of the Raspberry Pi as relay node could be even further expanded: - Matthias already mentioned that there should be a specific image for it. I agree! - New users would prefer a simple webinterface to configure the relay (basically a Vidalia as webinterface :-) - Such a webinterface should also show statistics e.g., "Your node has forwarded 1 TB in the last 4 weeks." This will show people that their contribution made a difference and it will make them happy and more confident about it. Eventually, they'll convince others to run a relay as well. I won't have the time to realize these ideas, but maybe others want to jump in. Further work and testing could be coordinated on this mailing list? I hope to have something up in a week or two, I need to watch it for a
while and continue to tweak, and maybe develop a solution for the TCP storms that can bring down a lot of consumer routers, before publishing for all.
I run a regular Linux PC as router which forwards all the traffic of the Pi. The machine has more than enough CPU power and therefore I haven't seen any issues there. Since the two months that I run the relay node on the Pi, my Linux PC router has frozen twice (which didn't happen before). But I'm not sure that it can be related to the forwarding of the relay node traffic. For a broader adaption it's definitely necessary to look into the behavior of the most popular routers e.g., the ones given out by the major providers. These things could be documented on the Tor Wiki as well. Best regards, Michael
Michael Berlin:
Hi Gordon and Matthias,
I've split your discussion from the original thread "Running exit-node in Germany" and created a new one.
I fully agree with you that the Raspberry Pi is the perfect device to let others run a Tor Relay Node very easily. What follows is a long mail about my experiences and more thoughts about the Pi as relay.
Just wanted to say that I've seen this and will reply in depth soon. I've just got my Pi relay running again and am experimenting with it and attempting to gather all the tips in the dozens of blog posts about running a Tor relay or a Bittorrent daemon (turns out, there are some characteristics these applications share), figure out the useful ones, and gather them all together. I was hoping some people from the list could help me with the "figure out the useful ones" and it looks like you got started before I could ask. :) One thing I'm particularly concerned with, if these Pi relays are going to be dropped onto non-technical but morally supportive folks' broadband connections by their friends and relatives: congestion control. The relay MUST NOT (following RFC definitions, heh) interfere with Netflix or Youtube or this won't work out so well. I also found the Adafruit HOWTO and was sad that it didn't mention relaying much. More later, -Gordon
Michael Berlin:
The logged throughput is also consistent with what I saw with the console traffic monitor "nload" (suggested command line options for nicer units and less refreshes: nload -u K -U G -t 3000). I guess, I saw even higher peaks there. Monitoring everything with "arm" is also nice but far too CPU intensive.
A quick thought - can we pull together data on tor monitoring utilities and then figure out the best bang for armhf CPU buck? 'nload' is good, but doesn't tell me much about *tor's* condition (other than it's probably not dead if there's still traffic flowing). It'd be nice to have some notion of circuits open, circuit creation rate, and tor's memory and CPU usage all on one screen, like 'arm', but lighter. I agree about 'arm', BTW. I use it for tuning on my x86-based relays, but it's buggy and it is really too heavy for the Pi's CPU. Maybe I should try to see if I can get 'arm' to run under PyPy[1] - I wonder if PyPy is faster on armhf like it can be on x86 and amd64. [1] http://pypy.org/ Best, -Gordon
On Sun, Aug 4, 2013 at 9:24 PM, Gordon Morehouse <gordon@morehouse.me> wrote:
Michael Berlin:
The logged throughput is also consistent with what I saw with the console traffic monitor "nload" (suggested command line options for nicer units and less refreshes: nload -u K -U G -t 3000). I guess, I saw even higher peaks there. Monitoring everything with "arm" is also nice but far too CPU intensive.
A quick thought - can we pull together data on tor monitoring utilities and then figure out the best bang for armhf CPU buck?
'nload' is good, but doesn't tell me much about *tor's* condition (other than it's probably not dead if there's still traffic flowing). It'd be nice to have some notion of circuits open, circuit creation rate, and tor's memory and CPU usage all on one screen, like 'arm', but lighter.
I agree about 'arm', BTW. I use it for tuning on my x86-based relays, but it's buggy and it is really too heavy for the Pi's CPU. Maybe I should try to see if I can get 'arm' to run under PyPy[1] - I wonder if PyPy is faster on armhf like it can be on x86 and amd64.
[1] http://pypy.org/
Best, -Gordon
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
In case you are wondering if anyone is following this, please continue! I have an aging Mac Mini that's running a Tor relay and one day it's going to go belly-up. I have an RPi that I could easily re-purpose to replace my relay. And while the Mini is low-powered the Pi is lower still so any tuning advice will be appreciated! Thanks!
Bill Waggoner:
On Sun, Aug 4, 2013 at 9:24 PM, Gordon Morehouse <gordon@morehouse.me> wrote:
Michael Berlin: [snip]
there. Monitoring everything with "arm" is also nice but far too CPU intensive.
A quick thought - can we pull together data on tor monitoring utilities and then figure out the best bang for armhf CPU buck?
'nload' is good, but doesn't tell me much about *tor's* condition (other than it's probably not dead if there's still traffic flowing). It'd be nice to have some notion of circuits open, circuit creation rate, and tor's memory and CPU usage all on one screen, like 'arm', but lighter.
I agree about 'arm', BTW. I use it for tuning on my x86-based relays, but it's buggy and it is really too heavy for the Pi's CPU. Maybe I should try to see if I can get 'arm' to run under PyPy[1] - I wonder if PyPy is faster on armhf like it can be on x86 and amd64.
[1] http://pypy.org/
I've submitted a ticket to the Tor Project trac about this, full well realizing that my current main role is spewing ideas around with basically zero time to help implement them. :( https://trac.torproject.org/projects/tor/ticket/9403 -Gordon
Michael Berlin:
Hi Gordon and Matthias,
I've split your discussion from the original thread "Running exit-node in Germany" and created a new one.
I fully agree with you that the Raspberry Pi is the perfect device to let others run a Tor Relay Node very easily. What follows is a long mail about my experiences and more thoughts about the Pi as relay. [snip] I've also attached the log of notices of the last 4 days (unfortunately no longer logs available). Within four days there was only one "Your computer is too slow to handle this many circuit creation requests!" warning and one "Failed to hand off onionskin.". In general I would say, I haven't seen any serious issues so far.
Here's why I'm still working on tuning. After discovering that my relay on the Pi was being rate-limited by an error in QoS in my router, I fixed that and the traffic immediately spiked (used up the BandwidthBurst bucket, I presume), then settled down - but after I went to bed, the following happened. It's a classic example of what I call a "circuit creation storm," which my bigger VPS nodes can handle, but the Pi can't. After the final log message, as far as I can tell, the process was killed for eating too much memory. Note the serendipitous heartbeat message in the middle, which mentions 2680 circuits open. I freed the Pi up to push about 2.6 Mbps. On my bigger VPS relays, *most* of the time a 5Mbps relay will stabilize well below 2500 circuits in my experience. At the last line, the log ends, and tor was not running when I woke up, so without checking further I'm going to assume the OS killed it as it chewed through all available memory. I'm kind of amazed that it didn't crash my router, but I did some tuning on the router the last time it did so. It's a consumer router running an alt firmware, but the NAT table only has 4096 entries or so - after the prior times where these "circuit creation storms" crashed my *router* I set timeouts on half-closed and abandoned TCP connections very aggressively, and it seems to survive now. The point, though, is that for sticking these things on the broadband of friends and family: * Tor shouldn't crash * Tor shouldn't crash their routers, which are often less forgiving than mine with the NAT table space and default timeouts * Tor shouldn't make them notice any degradation in streaming video services. I'm serious. If we can't manage that *on the Tor box itself* (possibly by using an alternate, highly conservative TCP congestion avoidance algo[1]?) then a lot of people will end up taking them off their network eventually. I will continue to research ways to avoid this - MaxAdvertisedBandwidth is a very crude tool, and I swear I read someplace about a MaxOpenCircuits or MaxCircuitRequests type setting slated to go in soon...? For now (and please feel free to shoot this full of holes, I'm by no means an iptables wizard, and this was cooked up before *any* caffeine this morning), I'm going to try limiting all SYNs with iptables: iptables -A INPUT -p tcp --syn -m limit --limit 4/s --limit-burst 10 -j ACCEPT iptables -A INPUT -p tcp --syn -j LOG iptables -A INPUT -p tcp --syn -j REJECT [1] http://arstechnica.com/information-technology/2012/05/codel-buffer-managemen... ... or for easy pasting, http://v.gd/An7s4B Aug 06 23:46:19.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. Aug 06 23:46:19.000 [warn] Failed to hand off onionskin. Closing. Aug 06 23:49:21.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [1337 similar message(s) suppressed in last 60 seconds] Aug 06 23:50:42.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [121 similar message(s) suppressed in last 60 seconds] Aug 06 23:51:03.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [445 similar message(s) suppressed in last 60 seconds] Aug 06 23:51:43.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [5248 similar message(s) suppressed in last 60 seconds] [snipping more of the same] Aug 06 23:55:49.000 [notice] Heartbeat: Tor's uptime is 2 days 12:00 hours, with 2680 circuits open. I've sent 5.67 GB and received 5.32 GB. Aug 06 23:56:28.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [2580 similar message(s) suppressed in last 60 seconds] Aug 06 23:58:26.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [271 similar message(s) suppressed in last 60 seconds] Aug 06 23:58:30.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [938 similar message(s) suppressed in last 60 seconds] Aug 06 23:59:30.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [1014 similar message(s) suppressed in last 60 seconds] Aug 07 00:00:30.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [1511 similar message(s) suppressed in last 60 seconds]
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 I still have the really weird circuit creation storms going on. I'm trying to figure out how to *eliminate* the possibility with some kind of iptables throttling, but limiting SYNs to 4 per second bursting to 10 didn't do anything at all. I know about the MaxAdvertisedBandwidth trick but it seems like a hacky workaround to me. I'd rather just advertise the bandwidth I have and either be able to handle it or, if possible, gracefully degrade during a storm, if I can detect it, by throttling circuit creation requests or TCP SYNs or whatever does the job. I happened to pop in and take a peek at the Pi during a "storm," which I noticed because there were some messages in the logs pretty recently with lots of "your computer is too slow to handle this many circuit creation requests!" with astronomical (seeming) numbers: Aug 12 00:43:45.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [369 similar message(s) suppressed in last 60 seconds] Aug 12 00:44:26.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [2514 similar message(s) suppressed in last 60 seconds] Aug 12 00:45:25.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [3196 similar message(s) suppressed in last 60 seconds] Aug 12 00:48:03.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [350 similar message(s) suppressed in last 60 seconds] The machine was receiving only 30KB/sec sustained Ethernet traffic and replying with the same, but system load was 0.00 and Tor appeared to be dead. So, I restarted it. Here are some logs. After the restart, notice the instant it's bootstrapped 100%, it gets slamed with circuit requests *again:* Aug 12 01:01:20.000 [notice] We now have enough directory information to build circuits. Aug 12 01:01:20.000 [notice] Bootstrapped 80%: Connecting to the Tor network. Aug 12 01:01:21.000 [notice] Self-testing indicates your ORPort is reachable from the outside. Excellent. Publishing server descriptor. Aug 12 01:01:23.000 [notice] Heartbeat: Tor's uptime is 0:00 hours, with 17 circuits open. I've sent 35 kB and received 28 kB. Aug 12 01:01:23.000 [notice] Bootstrapped 85%: Finishing handshake with first hop. Aug 12 01:01:24.000 [notice] Bootstrapped 90%: Establishing a Tor circuit. Aug 12 01:01:26.000 [notice] Tor has successfully opened a circuit. Looks like client functionality is working. Aug 12 01:01:26.000 [notice] Bootstrapped 100%: Done. Aug 12 01:01:26.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. Aug 12 01:01:26.000 [warn] Failed to hand off onionskin. Closing. Bandwidth before and after the restart... Slammed immediately. Actually, my max relay bandwith when bursting is around ~350KB/sec, but how much of this is legit and how much is what appears to be either thousands of creation requests or a logging bug about said requests? Either way, Tor *will* crash (and make my router sad) if left to its own devices for a day or two on the Pi, as it stands now. Device eth0 [192.168.1.2] (1/2): ===================================================== Incoming: . |...##|# . |.. ##|######## . |||#..################## ||##|########################## .################################# ################################### #################################### Curr: 283 kByte/s .#################################### Avg: 99 kByte/s ##################################### Min: 7.79 kByte/s . |##################################### Max: 292 kByte/s ####|.|........###################################### Ttl: 3.00 GByte Outgoing: |. ||.||#|#### . | ..#|#|#|########### ....###|##################### ..|############################## Curr: 203 kByte/s .################################## Avg: 71 kByte/s .|################################### Min: 0.52 kByte/s ##################################### Max: 214 kByte/s ####|.|........|##################################### Ttl: 3.22 GByte And logs as I was adjusting the bandwidth paste (I let it continue) ... note the bit about the nameserver, that's my *router* (WRT54G running Tomato) getting hammered hard enough by something - number of connections? - to start having problems. The last message has 7069 suppressed repeats. WTF. One additional clue, if Tor is dead and I restart it, the 30KB/sec sustained traffic you see at the lower left of the graph above drops off immediately. That's when I *start* the Tor process. WTF. Aug 12 01:02:26.000 [notice] Self-testing indicates your DirPort is reachable from the outside. Excellent. Aug 12 01:04:09.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [12350 similar message(s) suppressed in last 60 seconds] Aug 12 01:04:11.000 [warn] eventdns: All nameservers have failed Aug 12 01:04:11.000 [notice] eventdns: Nameserver 192.168.1.1:53 is back up Aug 12 01:04:11.000 [warn] eventdns: All nameservers have failed Aug 12 01:04:11.000 [notice] eventdns: Nameserver 192.168.1.1:53 is back up Aug 12 01:04:45.000 [warn] eventdns: All nameservers have failed Aug 12 01:04:45.000 [notice] eventdns: Nameserver 192.168.1.1:53 is back up Aug 12 01:05:10.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [9647 similar message(s) suppressed in last 60 seconds] Aug 12 01:06:11.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [6107 similar message(s) suppressed in last 60 seconds] Aug 12 01:06:13.000 [notice] Tried for 121 seconds to get a connection to [scrubbed]:993. Giving up. (waiting for circuit) Aug 12 01:07:09.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [7069 similar message(s) suppressed in last 60 seconds] What is going on here?! And, how do I throttle it? I've had to shut it down for the time being once again. - -Gordon Gordon Morehouse: ... or for easy pasting, http://v.gd/An7s4B -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSCJ4BAAoJED/jpRoe7/uj+QcIAJw/kyFifuMIhN3blVRWswvf hQvTe2r5Z0lHH0ockL6KcHt1u14Yq/3ED2sEVFGs+q01XYKLao1OVDG4NO6wsZxq CX+Lzug/BrhS8hDJlTTMSSDY4S0iHzyF9iMOlLW3iHgztDnfP/WhqjDnLV29qhNV 1iQVbhVYBzf6X3UAgiebbV07gLPlsY70bWBSL7JSon07aa5ZsI7lt/9rWRAiHeRp Pk8lftIobq9t1Vx2NVsbRkJ8pmRMRELcX58SmHYjlD4+qgugObJKrG4pOcuj4Z2C Z/Yz/VuiLv6AY+glxJtsSFZrukQr6isS2MW6f8bJwEXkQHOcLYj2Pf28TPdRMzk= =njNC -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Gordon Morehouse:
I still have the really weird circuit creation storms going on. I'm trying to figure out how to *eliminate* the possibility with some kind of iptables throttling, but limiting SYNs to 4 per second bursting to 10 didn't do anything at all.
Looking at what's draining out of my router's connection table (after it came back to life when I shut Tor down), it appears that there are hundreds of *outbound* connections *from* the Tor box to a wide variety of IPs. Weeeeeeeird. - -Gordon -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSCJ86AAoJED/jpRoe7/ujVggIAMTdqkCESrnCK6WfWtpg41GC l9fEEuo8yr8yh8Y4e3WolhPcW7b5EHrOIa5Yts0S+AJqGYuZvDDYI4ODRRKozbip RvLH5KO929f6STAUSm/RAgJCA/6gtKDDwHTrMVJY/klxI+OSHkYR1X98hiCZWKTc LWiVj67kT/NmoHeaFAPzxHt+v1xv9ETLjNDpt5ybRhz9NW10z6NZi3rJ0QWBBB7V EZnkpCetdQRuxHv2LArHdpP+mBEvtrLvN0cMbH/OSWWErAlYJTikhHi+zgX39ZQJ pYt2w8H7y5ogkeBhjXs9owH+XIZqSh52w0Sl0hOsHjW9/JR/tYPHTD1bkeSFeUQ= =XtQy -----END PGP SIGNATURE-----
On Mon, Aug 12, 2013 at 4:34 AM, Gordon Morehouse <gordon@morehouse.me> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
I still have the really weird circuit creation storms going on. I'm trying to figure out how to *eliminate* the possibility with some kind of iptables throttling, but limiting SYNs to 4 per second bursting to 10 didn't do anything at all.
I know about the MaxAdvertisedBandwidth trick but it seems like a hacky workaround to me. I'd rather just advertise the bandwidth I have and either be able to handle it or, if possible, gracefully degrade during a storm, if I can detect it, by throttling circuit creation requests or TCP SYNs or whatever does the job.
Circuit creation happens within the Tor protocol. How many circuit creation requests you get at once is a function of how much bandwidth you appear to have. How many you can handle is a function of how fast your CPU is, and how fast your crypto implementation is. You can decrease how much bandwidth you appear to have with "MaxAdvertisedBandwidth", but you already knew that. One thing that you should try is seeing whether the latest 0.2.4.x release does any better for you. In particular, I'd recommend trying the just-released 0.2.4.16-rc, with openssl 1.0.1e, and make sure that openssl 1.0.1e was built with the -enable-ec_nistp_64_gcc_128 option if possible. (I see you're already using 1.0.1e, but it doesn't appear to have been built with that option.) Using 0.2.4.x should let Tor use a faster circuit extension handshake to clients that support it. It will also have Tor use an improved algorithm for deciding how long is too long for a circuit queue. Instead of limiting the queue to a fixed number, it limits the size of the queue based on the expected time to clear it. (Another thing to look at would be the output of ./src/test/bench in the 0.2.4.x package.) yrs, -- Nick
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Nick Mathewson:
Circuit creation happens within the Tor protocol. How many circuit creation requests you get at once is a function of how much bandwidth you appear to have. How many you can handle is a function of how fast your CPU is, and how fast your crypto implementation is.
Slow and slow, for the Raspberry Pi. I have started gathering student hand-optimized crypto routines from around the net that are written from ARMv6, but that's a looooong ways away.
You can decrease how much bandwidth you appear to have with "MaxAdvertisedBandwidth", but you already knew that.
Yep. I just don't understand why it's so spiky.
One thing that you should try is seeing whether the latest 0.2.4.x release does any better for you. In particular, I'd recommend trying the just-released 0.2.4.16-rc, with openssl 1.0.1e, and make sure that openssl 1.0.1e was built with the -enable-ec_nistp_64_gcc_128 option if possible. (I see you're already using 1.0.1e, but it doesn't appear to have been built with that option.)
Yeah, I just noticed this morning that because I'd installed the Pi a few months ago before I upgraded to 0.2.4.x everywhere (figured I might as well when I upgraded my bridges to obfs3), it's still on 0.2.3.x.
Using 0.2.4.x should let Tor use a faster circuit extension handshake to clients that support it. It will also have Tor use an improved algorithm for deciding how long is too long for a circuit queue. Instead of limiting the queue to a fixed number, it limits the size of the queue based on the expected time to clear it.
I'll try it in the next day or two.
(Another thing to look at would be the output of ./src/test/bench in the 0.2.4.x package.)
And I'll try that and post it. - -Gordon -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSCQ4oAAoJED/jpRoe7/ujw6IIAKMJZr+h+VLfoTd4VjKehsrF qNNw3nstx82bAcaehKEFPSrvkIH2FRO+FqYR6ai+pWHTV8zZI593GA46Mb+9ElUU Gpifg/D0oJtitsjwzB1MQ+MqNfqa7/13jQ45nm5J9ip51KqbyUwgwR4XAhFRxQ8H J7Uv47SRqSd4sV0YQSC2rBOYFStLz0tYsN89L4ArC6mph83aNqpM0W2vopkRbO2R xC8//NmWQ93wsqkRFHpc3jZVsTuURx3YmgiCF4mpn/me7X/o0j9bhP8ZOmWW/vfB 6kdGG7a0tBYcciQk7VWqNHsGMh+PEIJCPAOS+sbeeds+0ZN8BPbXSRwHj/0OQ8s= =C2cg -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Quick addendum: Nick Mathewson:
Circuit creation happens within the Tor protocol. How many circuit
Is my observation of hundreds/thousands of outbound TCP connections from the Tor node simply because I "hot restarted" it when it already had a lot of circuits open (without much bandwidth going through them, oddly, I've seen it with 1200 circuits and 40KB/sec going through) *before* the "storm" hit? In other words, a circuit creation storm may not have anything to do with number or rate of inbound TCP connections (or outbound)?
You can decrease how much bandwidth you appear to have with "MaxAdvertisedBandwidth", but you already knew that.
I get that, but I wonder if MaxAvailableCircuits could/should be a thing. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSCRE+AAoJED/jpRoe7/ujiaMIAIN0+3k4uGItoJfiAoeX00QD x3W9nYHwVapxYkfQlDq5dkpY/x5MuK/yj3qFcdtnBehUGhi8yl+0+hSQ/EDkRLkx B6USPWK69MU26fxitfTQBQK3lFjEf68oNcJOVFbJJ1PwNy1kHDTJEC1jO5rX/Bsa RK4fJp/V9tSANoH6P1NBYExgj/5hSraRNV3JChRm2KfIx2wXzsFlEOGILtefYBBU ymEuNN9LEBysoEtyVZbtl3v3GVNtXSLbpSfZijx1zR53z+1RWoNKmEq0ZoSvebVj yCyxYZQPSU+dp9K4roYU9OMoOb8ImpQ8FSmwqxyBlBHP71IfZ4+Ganad9LwqL/k= =UojE -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Nick Mathewson:
(Another thing to look at would be the output of ./src/test/bench in the 0.2.4.x package.)
Here's the output of that, run from the built Debian source package, on a minimally loaded Raspberry Pi which appears to overclock to 900MHz under load while running at 700MHz while idle[1]: ~/debs $ `find . -name bench` ===== dmap ===== nbits=65536 digestmap_set: 1051.23 ns per element digestmap_get: 860.90 ns per element digestset_add: 211.21 ns per element digestset_contains: 199.60 ns per element. Hits == 32882688 False positive rate on digestset: 0.21% ===== aes ===== 1 bytes: 175.66 nsec per byte 2 bytes: 121.08 nsec per byte 4 bytes: 88.51 nsec per byte 8 bytes: 72.52 nsec per byte 16 bytes: 65.66 nsec per byte 32 bytes: 62.29 nsec per byte 64 bytes: 60.02 nsec per byte 128 bytes: 59.14 nsec per byte 256 bytes: 58.65 nsec per byte 512 bytes: 58.27 nsec per byte 1024 bytes: 58.06 nsec per byte 2048 bytes: 58.07 nsec per byte 4096 bytes: 57.89 nsec per byte 8192 bytes: 58.09 nsec per byte ===== onion_TAP ===== Client-side, part 1: 11374.294922 usec. Server-side, key guessed right: 28724.287109 usec Server-side, key guessed wrong: 37697.994141 usec. Client-side, part 2: 9042.044922 usec. ===== onion_ntor ===== Client-side, part 1: 4804.935547 usec. Server-side: 14673.099609 usec Client-side, part 2: 9837.501953 usec. ===== cell_aes ===== 509 bytes, misaligned by 0: 56.80 nsec per byte 509 bytes, misaligned by 1: 56.75 nsec per byte 509 bytes, misaligned by 2: 56.86 nsec per byte 509 bytes, misaligned by 3: 56.73 nsec per byte 509 bytes, misaligned by 4: 56.89 nsec per byte 509 bytes, misaligned by 5: 57.05 nsec per byte 509 bytes, misaligned by 6: 56.93 nsec per byte 509 bytes, misaligned by 7: 57.07 nsec per byte 509 bytes, misaligned by 8: 56.84 nsec per byte 509 bytes, misaligned by 9: 56.92 nsec per byte 509 bytes, misaligned by 10: 56.84 nsec per byte 509 bytes, misaligned by 11: 56.70 nsec per byte 509 bytes, misaligned by 12: 56.95 nsec per byte 509 bytes, misaligned by 13: 56.87 nsec per byte 509 bytes, misaligned by 14: 56.92 nsec per byte 509 bytes, misaligned by 15: 56.87 nsec per byte ===== cell_ops ===== Inbound cells: 29332.05 ns per cell. (57.63 ns per byte of payload) Outbound cells: 29962.20 ns per cell. (58.86 ns per byte of payload) ===== dh ===== Complete DH handshakes (1024 bit, public and private ops): 36.593397 millisec each. ===== ecdh_p256 ===== Complete ECDH P-256 handshakes (2 public and 2 private ops): 33.855275 millisec each. ===== ecdh_p224 ===== Complete ECDH P-224 handshakes (2 public and 2 private ops): 26.402733 millisec each. [1] 'cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq' performed during 'bench' runs and while nothing is happening report 900MHz during 'bench' and 700MHz idle. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSCaUkAAoJED/jpRoe7/ujSNMIAIokuGUDLBxgo3bWbgnHU4s2 8Zj65/YRoT+U0Px1E7zg4nVb4chuoeEQDuFllXKHSIbzQn5ULHBloCAlN11XDx9F 6gp91USOB4/ZWAVLO4trfiMikYSOMDva0CQIKWM5rG1+AQXrcdvkcCzR4q5ZKeON lQrfeNzT+jVoHnNir0Fd4tXrsrv13WfsDTlO6dBlHBKkP6kJdPnqHFF8gl6nPg6b 55V45MLT87JLFYCeeW8N/BVJi5REBe42wjFxq2YKEU/GDNS2jNLQZjpKV0MCnMER zD2QWqlMtWclB4e65BVSpFxuPXo7pFoE0cGFltMTMqTmx5jDuQQtBDFc+cBUkQ0= =dAli -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Nick Mathewson:
One thing that you should try is seeing whether the latest 0.2.4.x release does any better for you. In particular, I'd recommend trying the just-released 0.2.4.16-rc, with openssl 1.0.1e, and make sure that openssl 1.0.1e was built with the -enable-ec_nistp_64_gcc_128 option if possible. (I see you're already using 1.0.1e, but it doesn't appear to have been built with that option.)
For any Raspberry Pi Tor node operators breathlessly following this thread :P I succeeded in building 0.2.4.16-rc on the Pi. We will see how it performs now vs the circuit creation storms. This is not a simple Debian-type binary package install, as the packages present in the Tor Project experimental repos are built for *Debian* wheezy - that is, ARMv7 - and not *Raspbian* which was built to support the ARMv6 CPU on the Pi. So for those wishing to run experimental versions, nightlies etc of Tor on the RasPi, you will need to follow the instructions here: https://www.torproject.org/docs/debian.html.en#source Note that this may require over a GiB of dependency downloads, and the compile takes a long time - I wasn't paying close attention, but maybe 40-60 minutes. After building, when I installed I left out installing the debugging symbols (the -dbg .deb). I have not tried rebuilding a custom OpenSSL .deb yet. I also noticed in the scroll that the build process appeared to be looking around for libnacl-dev[1], which isn't present in Raspbian. I don't know if building NaCl would help with Tor performance on slow machines or not, but it's a thought. Based on some of their statements, they're thinking about the ARM architecture, though given that they mentioned NEON (ARMv7+), ARMv6 and older may be left out - I haven't bothered to dig through yet and see. [1] http://nacl.cr.yp.to/ - -Gordon -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSCar2AAoJED/jpRoe7/uj5EoH/21Bwa+oqhUZmU9cYeZm1Z+u bi6fnUsHylyvYKVKkU4wrTVg50t4ql3qWuK6efd2JdAWY+YhdLv6FxfBMCAJfY1y SLIn6GKeO124Rp7+wqNM0PBgDGxDvzbhGvArj1xo9XLlazFQr1d3zioZ4W+DM6Ll kmivDupsC4L+wOXM2PoP4n+P0lqPWeKVkcWnCUfLdn53+ted1Q1o/UsI0/PcpR3N I0XCtFY0xFb+ZngIajiACWDqJtpS79c1pyLkwNC5Mw962VDno6W5VK80ZxxEBvT7 QM269bbtGLI6b2Wiho8FQiq7fW18Ej40LjgDwJgwLQWqRd6962uJKRJKqevuxbk= =xvjz -----END PGP SIGNATURE-----
participants (4)
-
Bill Waggoner -
Gordon Morehouse -
Michael Berlin -
Nick Mathewson