<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><span></span></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div><span></span></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div><div>Hi,</div><div><br></div><div>I didn't give much detail here:</div><div><br><blockquote type="cite"><div dir="auto"><div><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">On 2 Feb 2018, at 11:08, Alec Muffett <<a href="mailto:alec.muffett@gmail.com">alec.muffett@gmail.com</a>> wrote:<br></span></font></div><blockquote type="cite"><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="m_888101483028181594quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br>The current limit is 2 connections per IP address.<br>This affects single onion services, because they don't use guards.<br><br>Can you please make sure that you only have one or two Single Onion<br>Services on each outbound IP address?</span></font></blockquote></div></div></div></blockquote></div></blockquote></div></div><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div><span style="background-color: rgba(255, 255, 255, 0);">So I'm going to quote from my email to tor-relays earlier today:</span></div><span style="background-color: rgba(255, 255, 255, 0);"><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div></span><blockquote type="cite"><span style="background-color: rgba(255, 255, 255, 0);">Here are the mitigations again:<br><br></span><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">o Major features:<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> - Give relays some defenses against the recent network overload. We<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> start with three defenses (default parameters in parentheses).<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> First: if a single client address makes too many connections<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> (>100), hang up on further connections. Second: if a single client<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> address makes circuits too quickly (more than 3 per second, with<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> an allowed burst of 90) while also having too many connections open<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> (3), refuse new create cells for the next while (1-2 hours).<br></span></font></blockquote><span style="background-color: rgba(255, 255, 255, 0);"><br>We could patch clients so they never exceed this number of circuits<br>by default. But that would penalise large clients that have a dedicated IP<br>address.<br></span></blockquote><span style="background-color: rgba(255, 255, 255, 0);"><br>For Alec's home development use case, limiting client bursts may be helpful.</span><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"></span></font></blockquote><blockquote type="cite"><font color="#000000"></font></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> Third: (DoS mitigation)<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> if a client asks to establish a rendezvous point to you directly,<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> ignore the request. These defenses can be manually controlled<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> by new torrc options, but relays will also take guidance from<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> consensus parameters, so there's no need to configure anything<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"> manually. Implements ticket 24902.</span></font></blockquote></blockquote><div><br></div>This shuts down Tor2web, which is a major source of the current load.<br><span style="background-color: rgba(255, 255, 255, 0);"><br></span><blockquote type="cite" __apple_fixed_attribute="true"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"></span></font></blockquote><blockquote type="cite"><blockquote type="cite" __apple_fixed_attribute="true"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">On 2 Feb 2018, at 03:04, David Goulet <<a href="mailto:dgoulet@torproject.org" dir="ltr" x-apple-data-detectors="true" x-apple-data-detectors-type="link" x-apple-data-detectors-result="2">dgoulet@torproject.org</a>> wrote:<br><br></span></font></blockquote><blockquote type="cite" __apple_fixed_attribute="true"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">On 01 Feb (04:01:10), grarpamp wrote:<br></span></font></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">Applications that use a lot of resources will have to rate-limit themselves.<br></span></font></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">Otherwise, relays will rate-limit them.<br></span></font></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">It's possible if relays figure that stuff by #2 might not be<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">an attack per se, but could be user activities... that relays<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">might push back on that one by...<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">- Seeking significantly higher default values committed<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">- Seeking default action committed as off<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">- Setting similar on their own relays if commits don't<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">work. And by not being default off, it should be prominently<br></span></font></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">documented if #2 affects users activities [1].<br></span></font></blockquote></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">That I agree. We've set up default values for now and they will probably be<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">adapted over time so for now this is experimental to see how much we make<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">people unhappy (well except for the people doing the DoS ;).<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">But I do agree that we should document some "real life" use cases that could<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">trigger defenses at the relay in some very public way (blog post or wiki)<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">before this goes wide in the network. Large amount of tor clienst behind NAT<br></span></font></blockquote><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">is one I have in mind, IPv6 as well...<br></span></font></blockquote><span style="background-color: rgba(255, 255, 255, 0);"><br>We did some analysis when we were choosing these figures.<br><br>It takes a few hundred clients behind an IP address, to have a 50% probability<br>of 3 clients choosing the same large guard. That's unusual. And if clients see their<br>guard timeout, they will move to another guard.<br><br>Here are some other scenarios:<br><br>Peer-to-peer clients like Ricochet, when the user has >90 contacts, but only<br>when there are hundreds of other clients on the same IP address.<br><br>Any Tor client that doesn't use guards. For example:<br><br>Bridges with multiple users, but only when there are >=3 bridges per outbound<br>IP address. (This is unlikely, because bridges need their own IPv4 address.<br>If you have multiple bridges using the default route, and multiple IP addresses,<br>set OutboundBindAddress on each bridge.) We will need to consider this issue<br>when we allow IPv6 bridges without a public IPv4 address. Perhaps an appropriate solution is to make bridge clients use vanguards.<br><br>Multiple (>=3) Tor2web or single onion services using separate tor instances,<br>behind a single IP address, making large numbers of circuits. This is a likely<br>source of our current issues.</span></blockquote><div><div><br></div><div>Alec wrote:</div><div><br></div><blockquote type="cite"><div><div dir="auto"><div dir="auto">I think the NYT is okay (separate IPs?)</div></div></div></blockquote><div><br></div><div>I hear Facebook is ok as well.</div><br><blockquote type="cite"><div><div dir="auto"><div dir="auto">but if I understand this right, this is going to hamper EOTK development, since I have ~ 12 worker onions spread over 6 quad-core machines, and then publish up to 10 additional "service" addresses via OnionBalance ... all behind my single DSL NAT firewall that protects them from inbound traffic.</div></div></div></blockquote></div></div><div><br></div><div><div><span style="background-color: rgba(255, 255, 255, 0);">For IP addresses with 3 or more connections to a single guard, the guard imposes</span></div><div><span style="background-color: rgba(255, 255, 255, 0);">a limit of 1 circuit every 3 seconds, with a 90 circuit burst allowance.</span></div></div><div><br></div><div>Are these single onion services or do they use guards?</div><div><br></div><div>If they are single onion services, <span style="background-color: rgba(255, 255, 255, 0);">then the probability that 3 of your 13? instances</span></div><div><span style="background-color: rgba(255, 255, 255, 0);">will choose </span><span style="background-color: rgba(255, 255, 255, 0);">the same relays for their directory, HSDir, intro or rend points is small</span></div><div><span style="background-color: rgba(255, 255, 255, 0);">(< 10%). </span><span style="background-color: rgba(255, 255, 255, 0);">If it happens, and if they build more than 90 circuits to the same relay,</span></div><div><span style="background-color: rgba(255, 255, 255, 0);">the defence </span><span style="background-color: rgba(255, 255, 255, 0);">will trigger. Then both instances will try another relay.</span></div><div><span style="background-color: rgba(255, 255, 255, 0);">You should probably limit your circuit construction rate if this happens a lot.</span></div><div><br></div><div>If they use guards, then the probability that 3 of your 13? instances will choose</div><div>the same guard is negligible (1 in 1 million). The defence will not trigger.</div><div><br></div><div><div><span style="background-color: rgba(255, 255, 255, 0);">Aaron wrote:</span></div><div><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div><blockquote type="cite"><font color="#000000"><span style="background-color: rgba(255, 255, 255, 0);">Because the circuit-creation limit is applied at the guard, wouldn’t this affect hidden sevices instead of single onion services?</span></font></blockquote><br></div></div><div>It will only trigger if hundreds of guard-using clients are behind a single IP address.</div><div>See above.</div><div><br></div><div>Alec wrote:</div><div><br></div><div><div dir="auto"></div><blockquote type="cite"><div dir="auto"><span style="background-color: rgba(255, 255, 255, 0);">I am not going to pretend that I fully understand the DDoS mitigations yet, but experience at two jobs has taught me that at least three entire countries essentially present themselves from behind small numbers of heavily NATed addresses, so I hope that the mitigations are NAT-friendly.</span></div><div dir="auto"><span style="background-color: rgba(255, 255, 255, 0);"><br></span></div><div dir="auto"><span style="background-color: rgba(255, 255, 255, 0);">ISTR that UAE and Singapore are two such, I forget the third?</span></div></blockquote></div><div><br></div><div>They are as NAT-friendly as possible under the circumstances.</div></div></div><div>We may have issues if thousands of users are behind a single IP.</div><div><br></div><div>In this case, we could recommend bridges, or increase the DDoS mitigation limits.</div><div><br></div><div>T</div></body></html>