[tor-bugs] #14006 [Core Tor/Tor]: Hidden service error: "We'd like to launch a circuit to handle a connection, but we already have 32 general-purpose client circuits..."

[Tor Bug Tracker & Wiki]

#14006: Hidden service error: "We'd like to launch a circuit to handle a
connection, but we already have 32 general-purpose client circuits..."
-----------------------------------------------+---------------------------
 Reporter:  asn                                |          Owner:  (none)
     Type:  defect                             |         Status:
                                               |  needs_information
 Priority:  Medium                             |      Milestone:  Tor:
                                               |  0.3.3.x-final
Component:  Core Tor/Tor                       |        Version:
 Severity:  Normal                             |     Resolution:
 Keywords:  tor-hs circuit-management scaling  |  Actual Points:
Parent ID:                                     |         Points:
 Reviewer:                                     |        Sponsor:
-----------------------------------------------+---------------------------

Comment (by dgoulet):

 Replying to [comment:20 asn]:
 > I think what I would prefer here is for Tor to rate-limit itself when
 building onion service circuits. Especially so when it has multiple onion
 services, but maybe even when it has only a single one. So instead of
 building all its onion circuits (IPs + hsdir circs) at once, it waits a
 randomized time (around a second?) before building each one.

 The problem with adding a random delay at startup is that it won't solve
 the "32 general purpose circuits are pending" issue. If those circuits a
 really stuck being built, the delay won't help much as they will all end
 up queued up and stuck at some point.

 A wise rate limit is probably what we want so we never go above that 32
 limit and thus no need for a cryptic warning that makes it that you just
 can't do anything but wait or/and panic.

 Now ok, looking a bit closely to the logs above, notice:

 {{{
 Jan 21 10:53:40.000 [warn] Giving up launching first hop of circuit to
 rendezvous point
 $9844B981A80B3E4B50897098E2D65167E6AEF127~$9844B981A80B3E4B50 at
 62.138.7.171 for service eb3w4t.....
 }}}

 The above is a service trying to open a circuit to a rendezvous point...
 So I think the bigger issue here is that we have 32 circuits stuck in a
 non OPEN state and just never expire for some reasons? Or they do but we
 open 32 new ones very quickly and they get stalled again in a non OPEN
 state.

 My money is on the later due to the *amount* of suppressed log (see
 below). This looks to me like a service getting a ridiculous amount of
 rendezvous requests, the Guard is chocking so we keep reaching that 32
 limit.

 {{{
 Jan 21 10:37:42.000 [notice] We'd like to launch a circuit to handle a
 connection, but we already have 32 general-purpose client circuits
 pending. Waiting until some finish. [215959 similar message(s) suppressed
 in last 600 seconds]
 }}}

 Quick skim, I don't see anything in `circuit_expire_building()` that would
 make a circuit be ignored in the GUARD_WAIT state so they should in theory
 expire even though they are waiting for the guard to be usable?

 I'm getting indeed more and more convinced that we need a rate limit both
 client and service side. That is a bit like we do with DoS mitigation now
 (#24902) which is some per second rate with burst. Busy hidden service
 will suffer reachability but at least won't break the network. The point
 is that DoS mitigation will prevent as much as possible a client DDoS
 towards a single service and that service will by itself prevent to DDoS
 the network.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/14006#comment:21>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online