[tor-talk] (D)DOS over Tor network ? Help !

Wed Dec 3 11:48:46 UTC 2014

On 2014-12-02 21:45, l.m wrote:
>> Perhaps the new implementation of the hidden services will be better
> ?
>> How is it going ?
> 
> I don't see anything in the improvements suggested for hidden services
> that would help this situation. Though I would be grateful for being
> corrected.
> 
> First, I just want to say I only meant sheep(s) to emphasize that you
> don't know how many black sheep you have participating. I mentioned
> the part about this potentially being an attack external to Tor out of
> concern for your participation in a de-anonymizing attack on your
> hosted HS. I see your HS's are offline while you troubleshoot this so
> that's good. Next, I'm confused by what you describe. Sorry I deleted
> your previous email so I may repeat some things you already said.
> 
> - no evidence of any HS being flooded from logs. No evidence of a load
> on any particular HS.
> and
> - next to no bandwidth consumption. Does this include no processor
> use? I don't recall if that was mentioned before.
> 
> - no evidence is apparent from checking REND_QUERY=HS. So no request
> to rendezvous. Might make sense given little/no traffic.
> 
> - your guards go offline. This is contradictory. If the attack is
> within Tor via a HS it implies the HS traffic *reliably* makes it to
> at least your guard before you experience the symptomatic overload and
> timeout. Meaning there must be traffic you can detect. Otherwise the
> attacker would likely lose their connection to the rendezvous point
> (at least sometimes) by committing to the attack. What I mean is in
> order for this to be an attack via malicious HS it would need to
> succeed in not timing out until the traffic reaches your guard and
> server. That's two circuits that must work before failing at your
> guard. Not to mention you already tried changing the guards. It just
> seems implausible to occur reliably enough to take your server down.
> This assumes little/no traffic and no heavy cpu usage.
> 
> -- Now I don't know how you setup your logging but I assume you would
> know if there was a load on any particular site or flood. I can
> suggest beefing up this part of the auditing trail. You could use
> proxy (on the same server) in your server blocks (Nginx?) for each HS
> (or in batches). Then you can use SPI to analyses the traffic of each
> proxy for a build up of use that might be causing your timeouts.
> Though I don't see the use if your logging is as good as you
> mentioned.
> 
> If there's no traffic, no cpu usage, no evidence of HS load except
> your guards are timing out--I'm back to the implausible. Two circuits
> that reliably take down your guards. There *has* to be traffic on your
> side you can measure or some load indicator. Either that or the attack
> is external to Tor. On the other hand you could reply and say 'yeh
> lots of cpu use'. In which case sorry for wasting your time. If there
> is alot of cpu use the VM-partitioning solution is the best solution
> as it would guarantee at least some guards available to your other
> HS's. It also provides you more granular control over hardware
> allocation. Either way you have to assume at some point you will be
> targeted externally (from Tor) to de-anonymize your HS's. Shared
> hosting.. many HS's.. you're an eavesdropping goldmine.
> 
> -- leeroy bearr

Hi, thanks for supporting.

As a reminder, here is the archive for this thread: 
https://lists.torproject.org/pipermail/tor-talk/2014-December/035807.html

And that's December: 
https://lists.torproject.org/pipermail/tor-talk/2014-December/thread.html

Problem is still going on, let's recap:

Tor goes 100% CPU only when (re)starting and publishing the HS, this 
takes around 5 minutes,
after that, it uses about 1% - 5% of 1 core.

We _are_ able to see the access.log and error.log of each virtual host, 
since they are simple nginx vhosts.
We also have a script that records the access.log size on mysql, we use 
that script to delete unused accounts.
We confirm that both access.log and error.log doesn't move of an inch, 
basically there is no http traffic.

Bandwidth usage is consistently around 5KB - 15KB / second.

Currently we disabled from torrc the most visited vhosts, have a look at 
this:

http://fuckyouhotwkd3xh.onion/ (working)
http://ijaw6sx25rzose3c.onion/ (working)
http://3xsrosyv52vefh2l.onion/ (working)
http://v5uhidj456rn4cra.onion/ (working)
http://za3uovobxijq4grb.onion/ (not working)
http://4vxnpigkblvmud4i.onion/ (not working)
http://3jfbvyg5pggg7mfg.onion/ (not working)

this is a list of the HS we are currently hosting. We tried more 
addresses but in short, it looks like only 10% of the HS are working.

Those are the Tor logs of the current session:

Dec 03 XXX [notice] Bootstrapped 90%: Establishing a Tor circuit.
Dec 03 XXX [notice] Tor has successfully opened a circuit. Looks like 
client functionality is working.
Dec 03 XXX [notice] Bootstrapped 100%: Done.
Dec 03 XXX [notice] Your Guard regar42 (XXX) is failing more circuits 
than usual. Most likely this means the Tor network is overloaded. 
Success counts are 120/181. Use counts are 453/453. 120 circuits 
completed, 0 were unusable, 0 collapsed, and 0 timed out. For reference, 
your timeout cutoff is 91 seconds.
Dec 03 XXX [warn] Your Guard regar42 (XXX) is failing a very large 
amount of circuits. Most likely this means the Tor network is 
overloaded, but it could also mean an attack against you or potentially 
the guard itself. Success counts are 120/241. Use counts are 453/453. 
120 circuits completed, 0 were unusable, 0 collapsed, and 0 timed out. 
For reference, your timeout cutoff is 91 seconds.
Dec 03 XXX [warn] Your Guard regar42 (XXX) is failing an extremely large 
amount of circuits. This could indicate a route manipulation attack, 
extreme network overload, or a bug. Success counts are 60/241. Use 
counts are 453/453. 60 circuits completed, 0 were unusable, 0 collapsed, 
and 0 timed out. For reference, your timeout cutoff is 91 seconds.
Dec 03 XXX [notice] We'd like to launch a circuit to handle a 
connection, but we already have 32 general-purpose client circuits 
pending. Waiting until some finish.
Dec 03 XXX [notice] Your Guard TorKuato (XXX) is failing more circuits 
than usual. Most likely this means the Tor network is overloaded. 
Success counts are 105/151. Use counts are 81/81. 105 circuits 
completed, 0 were unusable, 0 collapsed, and 2063 timed out. For 
reference, your timeout cutoff is 60 seconds.
(after 5 minutes)
Dec 03 XXX [notice] We'd like to launch a circuit to handle a 
connection, but we already have 32 general-purpose client circuits 
pending. Waiting until some finish. [749254 similar message(s) 
suppressed in last 600 seconds]
(after another 5 minutes)
Dec 03 XXX [notice] We'd like to launch a circuit to handle a 
connection, but we already have 32 general-purpose client circuits 
pending. Waiting until some finish. [398 similar message(s) suppressed 
in last 600 seconds]
(after 20 minutes)
Dec 03 XXX [notice] We'd like to launch a circuit to handle a 
connection, but we already have 32 general-purpose client circuits 
pending. Waiting until some finish. [6 similar message(s) suppressed in 
last 600 seconds]

We are running all this on a small VPS with 1GB of RAM.
The RAM finishes quickly but there is plenty of free swap.
The service was running fine with much more hidden services, on the same 
hardware.
We could increase the RAM but doubt that's the problem.

Again .. any Tor developer willing to dig into this ?
Ours is an experimental project, there is nothing business-oriented 
here.
If we are pushing the limits of hidden services capabilities, or really 
are under a low-level Tor based attack, isn't this a good chance to 
improve Tor's hidden service code
or just improve Tor's hidden service _logging_ code ?

Perhaps we should post this on tor-dev ?