A bit more evidence on circuit creation storms
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 So, I started having "tubes clogged" problems this evening and realized, finally, that my Raspberry Pi powered relay had been weathering a circuit creation storm since about 18:11 my time. tl;dr main dev-related questions at bottom Aug 29 18:11:03.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. Aug 29 18:12:02.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [2692 similar message(s) suppressed in last 60 seconds] First off, I'm happy to note that the Pi is far, FAR more able to weather these storms running 0.2.4.x, even without the perfectly optimal OpenSSL (all jokes aside about that particular combination of three words)... Secondly, the number of messages escalated. Here is the worst: Aug 29 18:38:04.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [18258 similar message(s) suppressed in last 60 seconds] What on earth is causing so many circuit creation requests in such a short timespan? Again, a Pi-specific note - on 0.2.3.x, the Pi would have crashed or Tor been killed before this number ever reached five digits, so 0.2.4.x is better! I has a metric! However, note that whatever is going on starts plugging up the works in Tor itself: Aug 29 18:18:36.000 [notice] Tried for 129 seconds to get a connection to [scrubbed]:993. Giving up. (waiting for circuit) Aug 29 18:19:04.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [7190 similar message(s) suppressed in last 60 seconds] Aug 29 18:19:14.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60s after 18 timeouts and 172 buildtimes. I assure you, my network connection speed didn't change. I still don't know how in the Tor protocol a circuit creation request is made, but maybe it was them gobbling bandwidth. Probably not, though? There was an increase in average TX/RX bandwidth during this incident, but not to levels that are in any way out of the ordinary for how I use my Internet connection. However, it's still causing trouble with my router (which is running Tomato). Check this out: Aug 29 18:56:14.000 [warn] Your computer is too slow to handle this many circuit creation requests! Please consider using the MaxAdvertisedBandwidth config option or choosing a more restricted exit policy. [13867 similar message(s) suppressed in last 60 seconds] Aug 29 18:56:40.000 [warn] eventdns: All nameservers have failed [snipped similar "too slow to handle" messages] Aug 29 19:05:19.000 [notice] eventdns: Nameserver 192.168.1.1:53 is back up My main question: How do circuit creation requests on one's Tor relay cause load on one's network infrastructure? Is it DNS requests? Is it TCP connection state entries? It's not bandwidth, we observed that above, and my router can handle far faster pipes than the one it's on currently. The DNS failing is a sign that the router is under severe stress. Back in the 0.2.3.x days, I often had to reboot the *router* after one of these storms, not just the Tor relay. And again - do we really know what is causing this? Something seems seriously wrong with the kind of numbers I'm seeing coming in to a node with MaxAdvertisedBandwidth 250KB. I think a bug should be opened about this, if there isn't one already. Regards, - -Gordon M. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSIAT1AAoJED/jpRoe7/ujuK4H/06MxbDHlN2uHjae6XurDib1 3yxXhHAUXC5pcP8smLQgqqtazTweeQDfIOT78QS9I0MB5z7ZnAYYsjs1/tJ1AKhb GkHtYXnlaZCxuP/oHlZYVsEhKXGGRSrXQM+y9f4uoukCn7KyVlCPF9oz2wBtSLqE qCENCZydn7nPUL6mRxveCoUvVXcGfBvm9uk17SmfuV2e6eDr5NMM4x2IxsZAYzLq hfYOUs8xLMqldsFqIaIV5J8sKwurI040P+TpZNVe7anB1ZF7uNRPfiyGrG7/JgTq aiGTyPArEdLnqYcRwGW+HbXBhQBjqF5yLt5Jj+kXXG5qBvaLQ3h/7IT7CJFr8GI= =eaXx -----END PGP SIGNATURE-----
On 13-08-29 10:35 PM, Gordon Morehouse wrote:
What on earth is causing so many circuit creation requests in such a short timespan?
One possibility, if i recall correctly, is that the Tor that comes with the PirateBrowser bundle is configured to build single hop circuits. Make sure that these defaults are still set in your relay: AllowSingleHopExits 0 AllowSingleHopCircuits 0 You can also try RefuseUnknownExits 1 but maybe auto is better And these may help sketch the circuit storms: EntryStatistics 1 ExitPortStatistics 1 ConnDirectionStatistics 1 Which is all to say, i dont know either...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 krishna e bera:
On 13-08-29 10:35 PM, Gordon Morehouse wrote:
What on earth is causing so many circuit creation requests in such a short timespan?
One possibility, if i recall correctly, is that the Tor that comes with the PirateBrowser bundle is configured to build single hop circuits.
Make sure that these defaults are still set in your relay:
The DDOS - because that's what it obviously is - managed to kill my Pi-based node last night, so I've finally restarted with all these except RefuseUnknownExits 1, just because of your caveat. I had some of the statistics already enabled, so it's continuing to collect those. Is there a way to give Tor a hard memory cap, so it won't chew up all available RAM on the system?
AllowSingleHopExits 0 AllowSingleHopCircuits 0
You can also try RefuseUnknownExits 1 but maybe auto is better
And these may help sketch the circuit storms: EntryStatistics 1 ExitPortStatistics 1 ConnDirectionStatistics 1
Best, Gordon M. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSIjJoAAoJED/jpRoe7/ujuicH/Au5NXr/q5MTYH54mPPuE/OJ 9jOkT/M34O0+U9gqYH8ja2gzTFf3CdxESraS6A7A+r2DWUX9R6l+zia5YX/SYCUu dWWNB843vXhcjNqhw01h05c70QgKStKrm6sLCjliVxhcovfMnkmMxLxk3EmQ3OzW nOdHQT2QGV+xCXqYz7FS9OtCnRmjTjI3bb8PJ1xcL76IjPGlCKr5vt7cDO3Y7x80 o0hnPJxMhYs0MhS5sNXfHIifDNT6LlCuZvIT0GLe3w9Gg15BUYKgn5bi1iNEtoSV J/2DbxvmT23Tv6E2WnpxEoOu/yupbHAiDcYbwIT1ig4mePA/xCgjdm7mEdqrXpE= =AiLg -----END PGP SIGNATURE-----
You could modify the tor init script to limit the memory usable by /usr/sbin/tor as described here: http://jlebar.com/2011/6/15/Limiting_the_amount_of_RAM_a_program_can_use.htm... But I don’t know if this works on RaspPi platform and what happens when the tor process hits the memory limit. Regards, Torland On Saturday 31 August 2013 11:14:04 Gordon Morehouse wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
krishna e bera:
On 13-08-29 10:35 PM, Gordon Morehouse wrote:
What on earth is causing so many circuit creation requests in such a short timespan?
One possibility, if i recall correctly, is that the Tor that comes with the PirateBrowser bundle is configured to build single hop circuits.
Make sure that these defaults are still set in your relay: The DDOS - because that's what it obviously is - managed to kill my Pi-based node last night, so I've finally restarted with all these except RefuseUnknownExits 1, just because of your caveat.
I had some of the statistics already enabled, so it's continuing to collect those.
Is there a way to give Tor a hard memory cap, so it won't chew up all available RAM on the system?
AllowSingleHopExits 0 AllowSingleHopCircuits 0
You can also try RefuseUnknownExits 1 but maybe auto is better
And these may help sketch the circuit storms: EntryStatistics 1 ExitPortStatistics 1 ConnDirectionStatistics 1
Best, Gordon M.
-----BEGIN PGP SIGNATURE-----
iQEcBAEBCgAGBQJSIjJoAAoJED/jpRoe7/ujuicH/Au5NXr/q5MTYH54mPPuE/OJ 9jOkT/M34O0+U9gqYH8ja2gzTFf3CdxESraS6A7A+r2DWUX9R6l+zia5YX/SYCUu dWWNB843vXhcjNqhw01h05c70QgKStKrm6sLCjliVxhcovfMnkmMxLxk3EmQ3OzW nOdHQT2QGV+xCXqYz7FS9OtCnRmjTjI3bb8PJ1xcL76IjPGlCKr5vt7cDO3Y7x80 o0hnPJxMhYs0MhS5sNXfHIifDNT6LlCuZvIT0GLe3w9Gg15BUYKgn5bi1iNEtoSV J/2DbxvmT23Tv6E2WnpxEoOu/yupbHAiDcYbwIT1ig4mePA/xCgjdm7mEdqrXpE= =AiLg -----END PGP SIGNATURE----- _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 tor-admin:
You could modify the tor init script to limit the memory usable by /usr/sbin/tor as described here:
http://jlebar.com/2011/6/15/Limiting_the_amount_of_RAM_a_program_can_use.htm...
But I don’t know if this works on RaspPi platform and what happens when the tor process hits the memory limit.
Thank you, I may mess around with this - what I'd dearly prefer is a torrc setting though. Maybe I should search for existing feature requests and/or submit a new one. [snip]
Is there a way to give Tor a hard memory cap, so it won't chew up all available RAM on the system?
Best, - -Gordon M. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSJMWtAAoJED/jpRoe7/ujEQAIAKWnkP3OfemDbxcmhJJr/rIW 65Hpa8gQl9S5z1OoQXLKHEnlIziYj8UJbWM2+ADsJJReAHtWSMAb8ZL7MLSn5wR8 hoTtsQJ+ZjXPP0M0Yiqgwb2HRUcWYuarGOseLzWUwsBAy3g0ucsDrnDhwj9cGPdB AAa4g4JJ/k5jW1N8clS0WIVJLnzDs4xYbxZzaLc54yQqawS4bybJUqjghLQekVdo wETMVK4Ajoip2Le1gDGevcO+0a6Nl15QLNdIkGSY7n8vRjQ1gvNpMIDt2CM+miko lwiWMy4OMbEQkDqKN7nkR15gLDoHsjgp37EbJn5im5Nfp/aNGMNGj8bWv0oi8Is= =G9Rh -----END PGP SIGNATURE-----
On Thu, Aug 29, 2013 at 11:30:33PM -0400, krishna e bera wrote:
On 13-08-29 10:35 PM, Gordon Morehouse wrote:
What on earth is causing so many circuit creation requests in such a short timespan?
One possibility, if i recall correctly, is that the Tor that comes with the PirateBrowser bundle is configured to build single hop circuits.
As far as I can tell from the configuration files people have posted here, this is not so. The Pirate Browser makes normal 3-hop circuits. It just includes a bunch of config lines that make you think it's doing 1-hop circuits -- but we haven't written any Tor config options to make you build 1-hop circuits. You'd have to hack your controller on the client side (or the Tor software directly), and even then you could only use the handful of relays that have allowed it. As for the circuit create storm phenomenon... if this is in fact a botnet signing up the million new users, and they're connecting to a hidden service C&C site, then I would expect even fiercer create storms. See also https://trac.torproject.org/projects/tor/ticket/9574 --Roger
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Roger Dingledine:
On Thu, Aug 29, 2013 at 11:30:33PM -0400, krishna e bera wrote:
On 13-08-29 10:35 PM, Gordon Morehouse wrote:
What on earth is causing so many circuit creation requests in such a short timespan?
[snip]
As for the circuit create storm phenomenon... if this is in fact a botnet signing up the million new users, and they're connecting to a hidden service C&C site, then I would expect even fiercer create storms.
See also https://trac.torproject.org/projects/tor/ticket/9574
Hi Roger, and thanks for taking the time to respond. I've definitely seen an increase in the storms compared to the baseline I established on my Raspberry Pi after upgrading to 0.2.4.16-rc; but so far only one crash. There has been a ripple or two on my bigger VPS relays. If I could bug you for a sec, I do have some questions about how circuit creation works in Tor. I hope you have a moment to answer. The full message is at: https://lists.torproject.org/pipermail/tor-relays/2013-August/002589.html Here is the main set of questions (the "numbers" are in the full message):
My main question: How do circuit creation requests on one's Tor relay cause load on one's network infrastructure? Is it DNS requests? Is it TCP connection state entries? It's not bandwidth, we observed that above, and my router can handle far faster pipes than the one it's on currently. The DNS failing is a sign that the router is under severe stress. Back in the 0.2.3.x days, I often had to reboot the *router* after one of these storms, not just the Tor relay.
And again - do we really know what is causing this? Something seems seriously wrong with the kind of numbers I'm seeing coming in to a node with MaxAdvertisedBandwidth 250KB.
Thanks much, - -Gordon M. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSIqe3AAoJED/jpRoe7/ujFKsH/A6zDg7QnvWF+n+hqFFao890 RO6BqcVnwgfqYaHNvgeQy+8IX16OT+4ccdmXWqAqa7LoJKTzXkAG9X2pQ60V5wNx PfeXR/sf9N0ZEiQFrC4mhJB+n7RImjiDflpgMKZnVUnGdalRG/b8ht2hpcAf/PfE RCqPn0DzEwOQUav8sG1pkc0Ii3iu1Xuo288lHjwElXEBo6SSMZ21X2EPUxJPCFtd Qp4TJ+pVYf6RJPX1amYdEbVcAGlIYELVXo4opSiqc4Bklng2vrZYwwpUU2KRWjqx Fq3CcKa+naXFm4JZXn8FtqrEb8LZhObOFDrjD6MlxPuwBL0GN9SOrV205ByJ+Sk= =s+9T -----END PGP SIGNATURE-----
On Thu, 29 Aug 2013 19:35:37 +0000, Gordon Morehouse wrote: ...
Aug 29 18:19:14.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60s after 18 timeouts and 172 buildtimes.
Random data point: I had these yesterday on a VPS-based relay.
My main question: How do circuit creation requests on one's Tor relay cause load on one's network infrastructure? Is it DNS requests? Is it TCP connection state entries? It's not bandwidth, we observed that above, and my router can handle far faster pipes than the one it's on currently. The DNS failing is a sign that the router is under severe stress.
Possibly your uplink is full (supposing you're on some DSL), and is starting to build up ping time; then DNS requests to the outside can start to timeout. Andreas -- "Totally trivial. Famous last words." From: Linus Torvalds <torvalds@*.org> Date: Fri, 22 Jan 2010 07:29:21 -0800
On 8/29/2013 11:09 PM, Andreas Krey wrote:
On Thu, 29 Aug 2013 19:35:37 +0000, Gordon Morehouse wrote: ...
Aug 29 18:19:14.000 [notice] Your network connection speed appears to have changed. Resetting timeout to 60s after 18 timeouts and 172 buildtimes. Random data point: I had these yesterday on a VPS-based relay.
My main question: How do circuit creation requests on one's Tor relay cause load on one's network infrastructure? Is it DNS requests? Is it TCP connection state entries? It's not bandwidth, we observed that above, and my router can handle far faster pipes than the one it's on currently. The DNS failing is a sign that the router is under severe stress. Possibly your uplink is full (supposing you're on some DSL), and is starting to build up ping time; then DNS requests to the outside can start to timeout.
Andreas
Over roughly the same time frame I received an incredibly high number of spam e-mails in one e-mail account that normally gets 20 or so a day on quiet days. Perhaps this is another example of mal-ware in action. David C
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Andreas Krey:
My main question: How do circuit creation requests on one's Tor relay cause load on one's network infrastructure? Is it DNS requests? Is it TCP connection state entries? It's not bandwidth, we observed that above, and my router can handle far faster pipes than the one it's on currently. The DNS failing is a sign that the router is under severe stress.
Possibly your uplink is full (supposing you're on some DSL), and is starting to build up ping time; then DNS requests to the outside can start to timeout.
Andreas
Unlikely - I do pseudo-QoS such that my uplink should never be full. It's possible, but I believe it's more likely that my router is thrashing for whatever reason and starts to be real slow at responding to DNS. -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJSIi9ZAAoJED/jpRoe7/ujL2EH/0O7HxgB/g5nEKFxgz114cYC gnlNzKXc2xUYZ92vv7BnNwuZ6TuFvW1qNBXT+9T1ObYWngnn/e3Jz8QOF6axmTd3 C+l2asKurD6M4dqkY53miLzrd783JTkgiPYaW6bNdFrmy+hzYjTbHuShccFYgncb SWRXIMk6OXDqLhCyMAUjlR6Dd0yq6uJMAefmgLyEc9stvrRcrK8de0xD2/qVnHCi RNJVQXAjKk5NvOIiONBilDl1jQsGgUkENz0Rh9naN9tjOE+Ev0lWbhUZpA09+2is T2/IbXQXS8m9c4KfL6Z+aSufD9ALjQrYLNQBs6vzk+SEl4ghyCoMOMzPMGS2JvE= =z+Zl -----END PGP SIGNATURE-----
participants (6)
-
Andreas Krey -
David Carlson -
Gordon Morehouse -
krishna e bera -
Roger Dingledine -
tor-admin