[tor-relays] Circuit creation "storms" overwhelming Raspberry Pi?
Thomas Hand
th6045 at gmail.com
Wed Jun 5 17:08:49 UTC 2013
I'm also having some problems with my rpi node going down every few days
due to lack of resouces and needing a reset. Can you mail me with some of
the alterations you made which might make it more stable? Thanks. T
On Jun 5, 2013 10:42 AM, <temp5 at tormail.org> wrote:
> I've been seeing these storms as well on my relay. I average something
> like 100 connections for weeks and weeks per the tor logs, but then
> suddenly it will jump into the thousands and I'll see the "Failed to hand
> off onionskin." and "Your computer is too slow to handle this many circuit
> creation requests!" messages.
>
> I wonder if it's some type of DDOS too.
>
> I thought about using this method
> (http://www.debian-administration.org/articles/187) on the relay and dir
> ports, but I'm not sure what sort of limits to set. Like does 1 Tor
> circuit = 1 iptables connection? Or if a user hits a webpage with 100 ads
> on it, maybe it would be 1 Tor circuit = 100 iptables connections?
>
> That's about as far as I got. I didn't want to break things by trying to
> fix another.
>
> > I did a lot of tuning on the Raspberry Pi and it's now much, much more
> > stable as a Tor relay, but just now I had another "circuit creation
> > storm." Interestingly, the Pi remained up, and my *router* crashed.
> > I've also seen huge bursts of circuit creation on a relay I run on a
> > VPS, but as it's a much more powerful box it rarely complains (and thus
> > I rarely notice).
> >
> > This many circuits and outbound connections is highly unusual for the
> > small relay I'm running on the Pi. And this behavior definitely occurs
> > in bursts. Is this an outbound DDOS or an attack on Tor itself? If the
> > former (or maybe the latter), is there some way I could perhaps use
> > iptables to temporarily "clamp" the ability to open TCP connections when
> > Tor (or anything on the Pi) opens a number over some threshold in some
> > short period of time?
> >
> > Here's log output (via 'arm') from the relay after my router crashed
> > twice, I went to the admin panel and noted hundreds of outbound
> > connections from my Tor box. Time is America/Los_Angeles.
> >
> > │ 13:55:00 [ARM_NOTICE] Relay unresponsive (last heartbeat: Sat May
> > 4 13:54:14 2013)
> > │ 13:52:25 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [404 similar message(s)
> > suppressed
> > in last 60 seconds]
> > │ 13:51:07 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [75 similar message(s)
> > suppressed in
> > last 60 seconds]
> > │ 13:50:52 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [601 similar message(s)
> > suppressed
> > in last 60 seconds]
> > │ 13:48:39 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [99 similar message(s)
> > suppressed in
> > last 60 seconds]
> > │ 13:47:34 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [22 similar message(s)
> > suppressed in
> > last 60 seconds]
> > │ 13:46:17 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [253 similar message(s)
> > suppressed
> > in last 60 seconds]
> > │ 13:43:47 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [1396 similar message(s)
> > suppressed
> > in last 60
> > │ seconds]
> > │ 13:42:48 [WARN] Your computer is too slow to handle this many
> > circuit
> > creation
> > │ requests! Please consider using the MaxAdvertisedBandwidth
> > config
> > option or choosing
> > │ a more restricted exit policy. [16 similar message(s)
> > suppressed in
> > last 60 seconds]
> >
> > Here's how it crashed my router (blowing ip_conntrack limits is
> > sufficient only to mess up many of my TCP connections, but eventually
> > the router runs out of memory and starts killing processes):
> >
> > May 4 13:51:24 dedmaus user.warn kernel: ip_conntrack: table full,
> > dropping packet.
> > May 4 13:51:24 dedmaus user.warn kernel: ip_conntrack: table full,
> > dropping packet.
> > May 4 13:51:24 dedmaus user.warn kernel: ip_conntrack: table full,
> > dropping packet.
> > May 4 13:51:25 dedmaus user.warn kernel: ip_conntrack: table full,
> > dropping packet.
> > May 4 13:51:29 dedmaus user.warn kernel: NET: 152 messages suppressed.
> > May 4 13:51:29 dedmaus user.warn kernel: ip_conntrack: table full,
> > dropping packet.
> > May 4 13:51:34 dedmaus user.warn kernel: NET: 193 messages suppressed.
> > May 4 13:51:34 dedmaus user.warn kernel: ip_conntrack: table full,
> > dropping packet.
> > May 4 13:51:39 dedmaus user.warn kernel: NET: 227 messages suppressed.
> >
> > ...ad infinitum with the number of messages suppressed per 5 sec
> > increasing until the router crashes.
> >
> >
> >
> > On Mon, Mar 18, 2013, at 06:18 PM, torsion at ftml.net wrote:
> >> I'm also seeing occasional messages like this on the Pi (it never lasts
> >> long):
> >>
> >> 18:13:24 [ARM_NOTICE] Relay resumed
> >> 18:13:18 [ARM_NOTICE] Relay unresponsive (last heartbeat: Mon Mar 18
> >> 18:13:04 2013)
> >> 17:28:43 [ARM_NOTICE] Relay resumed
> >> 17:28:38 [ARM_NOTICE] Relay unresponsive (last heartbeat: Mon Mar 18
> >> 17:28:25 2013)
> >> 14:12:26 [ARM_NOTICE] Relay resumed
> >> 14:12:20 [ARM_WARN] Deduplication took too long. Its current
> >> implementation has difficulty handling large logs so disabling it to
> >> keep the interface responsive.
> >> 14:12:20 [ARM_NOTICE] Relay unresponsive (last heartbeat: Mon Mar 18
> >> 14:12:06 20
> >>
> >> On Mon, Mar 18, 2013, at 01:00 PM, torsion at ftml.net wrote:
> >> > Hi there, I just joined the mailing list and apologized if this has
> >> been
> >> > discussed before. I did find discussion of a similar issue in January
> >> > 2013's archive:
> >> >
> >> >
> https://lists.torproject.org/pipermail/tor-relays/2013-January/001809.html
> >> >
> >> > It's important to note that I believe I've seen (but didn't save logs)
> >> a
> >> > couple "circuit creation burst" events on my established relay (about
> >> > 5Mbps, stable, guard, non-exit) which was mostly able to handle it
> >> > without crashing as it has plenty of RAM and the above-mentioned
> >> > messages - "Your computer is too slow to handle this many circuit
> >> > creation requests! Please consider using the MaxAdvertisedBandwidth
> >> > config option or choosing a m ore restricted exit policy." - appear
> >> only
> >> > with the relay is under load for other reasons AND a large number of
> >> > circuits are being suddenly created.
> >> >
> >> > I wondered if this was some kind of DOS attempt but didn't think much
> >> of
> >> > it because my fast relay continued working fine.
> >> >
> >> > However, I've just set up a Raspberry Pi, the 512MB model, as a relay
> >> on
> >> > a slower connection. Here are the relevant settings on this relay:
> >> >
> >> > RelayBandwidthRate 130 KB
> >> > RelayBandwidthBurst 340 KB
> >> >
> >> > The Pi has a fairly slow CPU, so I'd occasionally get messages about
> >> log
> >> > deduplication being too slow or something, but didn't think much of
> >> it.
> >> > I finally got the relay up and left it up for over 24 hours. When I
> >> > woke up this morning it had crashed. Here are the relevant log
> >> messages
> >> > - note the huge jump in number of circuits between 22:35 and 04:35
> >> > (maybe I got the Stable flag), then the storm of circuit open requests
> >> > starting at 05:49. Eventually I believe the Pi ran out of memory and
> >> > killed the tor process.
> >> >
> >> > What's very interesting here is that my fast VPS relay with a
> >> > RelayBandwidthRate over 5x faster is almost never handling much more
> >> > than 1000 circuits, so why all of a sudden the demand on the Pi that's
> >> > advertising a lower bandwidth rate?
> >> >
> >> > Mar 17 22:35:00.000 [notice] Heartbeat: Tor's uptime is 1 day 0:00
> >> > hours, with 26 circuits open. I've sent 974.13 MB and received 969.92
> >> > MB.
> >> > Mar 18 04:35:00.000 [notice] Heartbeat: Tor's uptime is 1 day 6:00
> >> > hours, with 972 circuits open. I've sent 1.61 GB and received 1.59 GB.
> >> > Mar 18 05:49:44.000 [warn] Your computer is too slow to handle this
> >> many
> >> > circuit creation requests! Please consider using the
> >> > MaxAdvertisedBandwidth config option or choosing a more restricted
> >> exit
> >> > policy.
> >> > Mar 18 05:49:44.000 [warn] Failed to hand off onionskin. Closing.
> >> > Mar 18 05:50:44.000 [warn] Your computer is too slow to handle this
> >> many
> >> > circuit creation requests! Please consider using the
> >> > MaxAdvertisedBandwidth config option or choosing a more restricted
> >> exit
> >> > policy. [5817 similar message(s) suppressed in last 60 seconds]
> >> > Mar 18 05:52:30.000 [warn] Your system clock just jumped 101 seconds
> >> > forward; assuming established circuits no longer work.
> >> > Mar 18 05:53:51.000 [warn] Your computer is too slow to handle this
> >> many
> >> > circuit creation requests! Please consider using the
> >> > MaxAdvertisedBandwidth config option or choosing a more restricted
> >> exit
> >> > policy. [1055 similar message(s) suppressed in last 60 seconds]
> >> > Mar 18 05:55:14.000 [warn] Your computer is too slow to handle this
> >> many
> >> > circuit creation requests! Please consider using the
> >> > MaxAdvertisedBandwidth config option or choosing a more restricted
> >> exit
> >> > policy. [329 similar message(s) suppressed in last 60 seconds]
> >> >
> >> > I'd like to figure out just how much the Raspberry Pi is capable of,
> >> > because it could be a cheap way to build out the relay network by
> >> people
> >> > who want to donate bandwidth - but of course it needs to be stable,
> >> and
> >> > something about my setup is not.
> >> >
> >> > Also:
> >> >
> >> > Mar 16 20:55:33.000 [notice] No AES engine found; using AES_*
> >> functions.
> >> >
> >> > I have no idea if the Broadcom BCM2835 SoC (ARM1176JZF-S CPU) in the
> >> Pi
> >> > has any AES capability, but it'd be great to find out.
> >
> >
> >
>
>
> _______________________________________________
> tor-relays mailing list
> tor-relays at lists.torproject.org
> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.torproject.org/pipermail/tor-relays/attachments/20130605/83a8a62e/attachment.html>
More information about the tor-relays
mailing list