<html><head></head><body><div class="ydp70c789f7yahoo-style-wrap" style="font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:16px;"><div></div>
<div>David,</div><div><br></div><div>This is exactly the type of information I was hoping for. You should make this an article and link it to the overloaded support page.</div><div><br></div><div>I guess I assumed that Tor preformed external timeout monitoring apposed to relay reported resource monitoring.</div><div><br></div><div>It's interesting that you mention loadbalancing Tor as that is precisely what my recent efforts have been geared toward.</div><div><br></div><div>I'm fairly confident that my last overloaded state was due to migrating one of my Tor relay nodes onto a previously provisioned BotFarm node and forgetting to kill the existing bot processes; thus, having competing resources. I can confirm that when loadbalancing Tor relay nodes that the whole is only as good as the weakest link; thus, it's important to have identical Tor relay nodes to evenly distribute circuits and maintain consensus.</div><div><br></div><div>In this paradigm, I was hoping to be able to define a timeout value associated with the overloaded state and tune the loadbalancer to redistribute to different upstream nodes should a Tor relay node reach such a value. However, it seems this is a moot point, after reading your summary of the reporting process.</div><div><br></div><div>At present, I have the upstream, loadbalancing timeout values disabled and let the Tor nodes build or teardown circuits based on available resources per node. I do see spikes alternate through various nodes throughout the day. It would be nice to find an upstream timeout value to better manage those spikes.</div><div><br></div><div>Any recommendations would be greatly appreciated.</div><div><br></div><div>Respectfully,</div><div><br></div><div><br></div><div>Gary</div><div><br></div><div>P.S. This is all being done on ASUSWRT-Merlin using AiMesh nodes, but isn't limited to that architecture. I hope to publish a tutorial, after ironing out all the kinks.</div><div><br></div><div><br></div>
<div id="ydp70c789f7yahoo_quoted_3282162806" class="ydp70c789f7yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
On Tuesday, September 28, 2021, 7:01:04 AM MDT, David Goulet <dgoulet@torproject.org> wrote:
</div>
<div><br></div>
<div><br></div>
<div>On 27 Sep (14:23:34), Gary C. New via tor-relays wrote:<br clear="none"><br clear="none">> George,<br clear="none">> The referenced support article provides recommendations as to what might be<br clear="none">> causing the overloaded state, but it doesn't provide the metric(s) for how<br clear="none">> Tor decides whether a relay is overloaded. I'm trying to ascertain the<br clear="none">> later. I would assume the overloaded state metric(S) is/are a maximum<br clear="none">> timeout value and/or reoccurrence value, etc. By knowing what the<br clear="none">> overloaded state metric is, I can tune my Tor relay just short of it. Thank<br clear="none">> you for your reply. Respectfully,<br clear="none"><br clear="none">Hi Gary!<br clear="none"><br clear="none">I'll try to answer the best I can from what we've have worked on for these<br clear="none">overload metrics.<br clear="none"><br clear="none">Essentially, there few places within a Tor relay that we can easily notice an<br clear="none">"overloaded" state. I'll list them and tell you how we decide:<br clear="none"><br clear="none">1. Out-Of-Memory invocation<br clear="none"><br clear="none"> Tor has its own OOM and it is invoked when 75% of the total memory tor<br clear="none"> thinks it can use is reached. Thus, let say tor thinks it can use 2GB in<br clear="none"> total then at 1.5GB of memory usage, it will start freeing memory. That, is<br clear="none"> considered an overload state.<br clear="none"><br clear="none"> Now the real question here is what is the memory "tor thinks" it has.<br clear="none"> Unfortunately, it is not the greatest estimation but that is what it is.<br clear="none"> When tor starts, it will use MaxMemInQueues for that value or will look at<br clear="none"> the total RAM available on the system and apply this algorithm:<br clear="none"><br clear="none"> if RAM >= 8GB {<br clear="none"> memory = RAM * 40%<br clear="none"> } else {<br clear="none"> memory = RAM * 75%<br clear="none"> }<br clear="none"> /* Capped. */<br clear="none"> memory = min(memory, 8GB) -> [8GB on 64bit and 2GB on 32bit)<br clear="none"> /* Minimum value. */<br clear="none"> memory = max(250MB, memory)<br clear="none"><br clear="none"> Why we picked those numbers, I can't tell you that, these come from the very<br clear="none"> early days of the tor software and I can't tell you why.<br clear="none"><br clear="none"> And so to avoid such overload state, clearly run a relay above 2GB of RAM on<br clear="none"> 64bit should be the bare minimum in my opinion. 4GB would be much much<br clear="none"> better. In DDoS circumstances, there is a whole lot of memory pressure.<br clear="none"><br clear="none"> One keen observer can notice that this approach also has the problem that it<br clear="none"> doesn't shield tor from being called by the OS OOM itself. Reason is that<br clear="none"> because we take the total memory on the system when tor starts, if the<br clear="none"> overall system has many other applications running using RAM, we end up<br clear="none"> eating too much memory and the OS could OOM tor without tor even noticing<br clear="none"> memory pressure. Fortunately, this is not a problem affecting the overload<br clear="none"> status situation.<br clear="none"><br clear="none">2. Onionskins processing<br clear="none"><br clear="none"> Tor is sadly single threaded _except_ for when the "onion skins" are<br clear="none"> processed that is the cryptographic work that needs to be done on the famous<br clear="none"> "onion layers" in every circuits.<br clear="none"><br clear="none"> For that we have a thread pool and outsource all of that work to that pool.<br clear="none"> It can happen that this pool starts dropping work due to back pressure and<br clear="none"> that in turn is an overload state.<br clear="none"><br clear="none"> Why this can happen, essentially CPU pressure. If your server is running at<br clear="none"> capacity and it is not only your tor, then this is likely to trigger.<br clear="none"><br clear="none">3. DNS Timeout<br clear="none"><br clear="none"> This applies only to Exits. If tor starts noticing DNS timeouts, you'll get<br clear="none"> the overload flag. This might not be because your relay is overloaded in<br clear="none"> terms of resources but it signals a problem on the network.<br clear="none"><br clear="none"> And DNS timeouts at the Exits are a _huge_ UX problem for tor users and so<br clear="none"> Exit operators really need to be on top of those to help. It is not clear<br clear="none"> with the overload line but at least if an operator notices the overload<br clear="none"> line, it can then investigate DNS timeouts in case there is no resources<br clear="none"> pressure.<br clear="none"><br clear="none">4. TCP port exhaustion<br clear="none"><br clear="none"> This should be extremely rare though. The idea about this one is that you<br clear="none"> ran out of TCP ports which is a range that is usually, on Linux, 32768-60999<br clear="none"> and so having that many connections would lead to the overload state.<br clear="none"><br clear="none"> However, I think (I might be wrong though) that nowadays this range is per<br clear="none"> source IP and not process wide so it would likely have to be deliberate from<br clear="none"> someone to put your relay in that state.<br clear="none"><br clear="none"><br clear="none">There are two other overload lines that tor relay report:<br clear="none">"overload-ratelimits" and "overload-fd-exhausted" but they are not used yet<br clear="none">for the overload status on Metrics. But you can find them in your relay<br clear="none">descriptor[0] if you are curious.<br clear="none"><br clear="none">They are about when your relay reaches its connection global limit too often<br clear="none">and when your relay runs out of file descriptors.<br clear="none"><br clear="none">Hope this helps but overall as you can see, a lot of factor can influence<br clear="none">these metrics and so the ideal ideal ideal situation for a tor relay is that<br clear="none">it runs alone on a fairly good machine. Any kinds of pullback from a tor relay<br clear="none">like being overloaded has cascading effects on the network both in terms of UX<br clear="none">but also in terms of load balancing which tor is not yet very good at (but we<br clear="none">are working on hard on making it much better!!).<br clear="none"><br clear="none">Cheers!<br clear="none">David<div class="ydp70c789f7yqt1920372800" id="ydp70c789f7yqtfd52452"><br clear="none"><br clear="none">[0]</div><br clear="none"><br clear="none"><a shape="rect" href="https://collector.torproject.org/recent/relay-descriptors/server-descriptors/?C=M;O=D" rel="nofollow" target="_blank">https://collector.torproject.org/recent/relay-descriptors/server-descriptors/?C=M;O=D</a><br clear="none"><br clear="none">-- <br clear="none">+7Xz1XCshqTyudrO7K4kGBEl+NghDNbqiTGYZpSOw4U=<div class="ydp70c789f7yqt1920372800" id="ydp70c789f7yqtfd68229"><br clear="none"></div><div class="ydp70c789f7yqt1920372800" id="ydp70c789f7yqtfd87976">_______________________________________________<br clear="none">tor-relays mailing list<br clear="none"><a shape="rect" href="mailto:tor-relays@lists.torproject.org" rel="nofollow" target="_blank">tor-relays@lists.torproject.org</a><br clear="none"><a shape="rect" href="https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays" rel="nofollow" target="_blank">https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays</a><br clear="none"></div></div>
</div>
</div></div></body></html>