Re: How is this attack causing 900 Mbps download and 100 Mbps upload?
Hi, I run three hobby relays and I seem to be seeing the same behavior on one of them. I compared two similar ARM, 2-core, non-exit Guard+HSDir relays, both running Tor 0.4.9.13. On the affected relay I measured 41,240 BEGIN_DIR streams in 61 seconds (~676/s). The Tor main thread reaches about 100% CPU, ntor onionskins are being dropped, and traffic is strongly asymmetric — roughly 2.2 MB/s read vs 7.1 MB/s written. On my other Guard+HSDir relay, measured at roughly the same time, I saw only 148 BEGIN_DIR streams in 60 seconds (~2.5/s). That relay carries much more traffic (~15 MB/s in both directions) and appears healthy. So in my case, simply having the HSDir flag does not seem sufficient to trigger the issue. One relay is affected, while another Guard+HSDir relay is not. I’m only a hobby relay operator, so I’m posting these numbers mainly in case they are useful for comparison. Regards, KhanKrum
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 The attack seems to be changing. It's now significantly worse and is sustained for 12-24 hours at a time. This needs to be addressed soon. Regards, forest -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQQtr8ZXhq/o01Qf/pow+TRLM+X4xgUCasSonAAKCRAw+TRLM+X4 xjv/AQCrUEM2LOPYDvOlNXBfFAeqQfZ/BoauQV0XXCED1JzHjwEA8Hr0JVwe/Xle Aq0dqJ60uE4HUGId7r8PhR23nic4QA0= =eUdr -----END PGP SIGNATURE-----
Hi, I run a small non-exit Guard/HSDir relay (FreeBSD, Tor 0.4.9.13). A little over a week ago it experienced a clearly abnormal event where outbound traffic became extremely asymmetric and briefly reached several hundred Mbit/s while inbound traffic remained much lower. FreeBSD also logged mbuf/jumbo cluster exhaustion, and Tor eventually stopped. This weekend the relay had another strange event. The VPS itself remained running and FreeBSD booted normally. The interface was up and the local provider gateway was reachable, but the host could not reach any external IPv4 address and the public IP/SSH became unreachable. This condition lasted overnight and connectivity returned the following morning without any configuration changes on the relay. Unfortunately I did not capture BEGIN_DIR counts or CPU usage before connectivity disappeared, so I cannot say whether this was the same attack, an upstream mitigation/null-route, or an unrelated provider network issue. I am posting this mainly because the timing overlaps with the reports of longer 12–24 hour attacks. If it happens again I will try to capture BEGIN_DIR rate, CPU usage, Tor logs and traffic asymmetry before the host becomes unreachable. Regards, Sent with Proton Mail secure email. On Tuesday, October 6th, 2026 at 10:52 AM, forest via tor-relays <tor-relays@lists.torproject.org> wrote:
The attack seems to be changing. It's now significantly worse and is sustained for 12-24 hours at a time. This needs to be addressed soon.
Regards, forest
Hi, A follow-up to my previous message, because the situation has developed in an interesting way. In my original comparison, KhanKrumCloud3 was the affected relay, while KhanKrumCloud2, a similar 2-core ARM Guard+HSDir relay hosted by a different provider, appeared completely healthy and served as a useful comparison. KhanKrumCloud3 later recovered spontaneously. I deliberately did not restart Tor and made no configuration changes. The abnormal BEGIN_DIR activity and the strong traffic asymmetry disappeared suddenly, while Tor continued running with the same PID. Now KhanKrumCloud2, the relay that was previously healthy, has developed very similar behaviour. KhanKrumCloud2 is hosted at Hetzner. During a two-hour monitoring run I observed BEGIN_DIR rates frequently in the range of approximately 1,400-2,300 per second. At the same time inbound traffic was usually only around 4-8 MB/s, while outbound traffic reached approximately 70-115 MB/s. The Tor main thread was saturated and Tor repeatedly logged warnings about too many circuit creation requests. The behaviour was also clearly intermittent. At approximately 18:44 UTC, KhanKrumCloud2 was still processing more than 2,000 BEGIN_DIR streams per second and writing more than 100 MB/s. Less than a minute later BEGIN_DIR dropped to almost zero and outbound traffic fell below 1 MB/s. The relay then remained mostly quiet for about 75 minutes. At the end of the monitoring period, the behaviour returned very suddenly: 19:59:50 UTC - BEGIN_DIR 0/s, outbound about 1 MB/s, 2,991 open circuits 20:00:55 UTC - BEGIN_DIR 1,226/s, outbound about 57 MB/s, 5,502 open circuits 20:02:05 UTC - BEGIN_DIR 2,345/s, outbound about 115 MB/s, 9,029 open circuits Tor remained the same process during these transitions. I then upgraded KhanKrumCloud2 from Tor 0.4.9.13 to Tor 0.4.9.14, updated the kernel and rebooted the server. After reboot, KhanKrumCloud2 came back normally with all its previous flags: Fast, Guard, HSDir, Running, Stable, V2Dir, Valid Unfortunately, the abnormal workload was still present. A new 60-second measurement after the upgrade and reboot showed 31,353 BEGIN_DIR streams, or 522.55 BEGIN_DIR/s. At the same time Tor was using around 130% CPU. Inbound traffic was only around 1-2 MB/s, while outbound traffic was again pinned at the configured relay bandwidth limit. I have now reduced the bandwidth limits on KhanKrumCloud2 substantially: RelayBandwidthRate 35 Mbits RelayBandwidthBurst 50 Mbits MaxAdvertisedBandwidth 34 Mbits I did this mainly to protect the server's traffic quota. This is becoming a practical problem for cloud-hosted relay operators, not only an interesting traffic anomaly. Most cloud providers meter or charge outbound traffic. During the worst periods, KhanKrumCloud2 was sending approximately 100-115 MB/s while receiving only a few MB/s. Sustained at that level, this would correspond to roughly 8-10 TB of outbound traffic per day. For a hobby relay operator using a cloud provider, that is enough to exhaust a monthly traffic allowance very quickly or potentially generate unexpected charges. If other cloud-hosted relays are experiencing the same behaviour, it could realistically cause operators to throttle or shut down otherwise healthy relays. I still cannot say whether this is an intentional attack, so I would prefer to describe it as abnormal or pathological traffic rather than claim a cause that I cannot prove. What is especially interesting to me is that KhanKrumCloud2, which was healthy in my previous comparison and is hosted by a different provider from KhanKrumCloud3, has now developed the same type of BEGIN_DIR/circuit-creation workload. Also, upgrading KhanKrumCloud2 to Tor 0.4.9.14 and rebooting did not eliminate it. I would really appreciate guidance from anyone familiar with Tor's relay and circuit handling on what additional data would be useful. I have the full two-hour CSV monitoring capture, Tor logs and metrics available, and I can run specific diagnostics if that would help identify what is happening. Regards, KhanKrum
Hi forest, and everyone, I would like to follow up on my earlier post about the Tor BEGIN_DIR Storm Sentinel with observations from another severe BEGIN_DIR storm affecting two of my non-exit ARM relays on October 12–13. *First, forest, you were right to raise the issue of MetricsPort responsiveness.* Our Sentinel polls Tor’s MetricsPort with a four-second HTTP timeout. During both storms, many requests timed out, leaving substantial gaps in our BEGIN_DIR measurements. In other words, our monitoring was partially blinded precisely when we needed it most. *In the eight-hour observation windows examined shortly after the storms:* - Cloud2 (Hetzner ARM): 868 invalid samples out of 1,608 (54.0%). - Cloud3 (Oracle Ampere ARM): 958 invalid samples out of 1,770 (54.1%). These percentages refer to collected samples over the entire eight-hour windows, not just the active storms. Many failures were caused by MetricsPort timeouts. Consequently, our recorded maximum BEGIN_DIR rates should be regarded as the highest successfully observed values, not necessarily the actual peaks. *Incident timeline (all times UTC):* Event Cloud2 / Hetzner Cloud3 / Oracle HSDir first observed (0→1) Oct 12, 21:23:20 Oct 12, 23:12:45 First WATCH event Oct 12, 23:09:20 Oct 12, 23:20:45 STORM_START Oct 13, 00:01:24 Oct 12, 23:20:55 HSDir loss observed (1→0) Oct 13, 01:22:30 Oct 13, 01:22:25 STORM_END Oct 13, 02:40:50 Oct 13, 02:43:15 Guard recovery observed Oct 13, 03:05:20 Oct 13, 03:13:15 These timestamps are based on Sentinel observations. Flag transitions indicate the first successful sample showing the new value, rather than the exact moment of a directory consensus change. Cloud3 entered STORM just 8 minutes and 10 seconds after HSDir was first observed. Cloud2 entered STORM approximately 2 hours and 38 minutes after acquiring HSDir, with several smaller WATCH episodes beforehand. Particularly striking is the near-simultaneous observed loss of HSDir on both relays, only five seconds apart, despite their different HSDir acquisition and storm onset times. *Measurements during the storms* Cloud2 — Hetzner ARM: - At STORM_START: 576.69 BEGIN_DIR/s. - Traffic: 3.515 MB/s received versus 30.019 MB/s transmitted (TX/RX ratio 8.54). - Highest successfully observed BEGIN_DIR rate: 2,300.38/s. - Tor subsequently reported 868,494 dropped ntor handshake requests, corresponding to a reported drop fraction of 12.60%. Cloud3 — Oracle Ampere ARM: - At STORM_START: 1,026.09 BEGIN_DIR/s. - Traffic: 2.952 MB/s received versus 12.246 MB/s transmitted (TX/RX ratio 4.15). - Highest successfully observed BEGIN_DIR rate: 2,801.88/s. - Tor reported 687,758 dropped ntor handshake requests, corresponding to a reported drop fraction of 11.00%. These dropped-handshake figures come from Tor’s own overload notices, rather than from Sentinel. They should not be interpreted as exact totals for the entire storm. The substantial outbound traffic asymmetry, together with Tor’s circuit-creation overload warnings and dropped handshake requests, indicates severe resource pressure on both relays. *Relay status and recovery* Both affected relays were running Tor 0.4.9.14. Neither Tor process restarted during the incident, as confirmed by unchanged PIDs in our Sentinel records and the uptime reported in Tor’s notice logs. Cloud2 received a manually initiated configuration reload at 01:22:13 UTC, following a reduction of its bandwidth limits. Cloud3 had no corresponding reload at that time in its notice logs. Interestingly, both relays were first observed losing their HSDir and Guard flags within seconds of each other, despite their Tor processes remaining alive. Both relays subsequently recovered their Guard flags — Cloud2 at 03:05:20 UTC and Cloud3 at 03:13:15 UTC — while HSDir remained absent. BEGIN_DIR activity eventually returned to normal on both machines. Our third relay, a smaller Oracle AMD instance without the HSDir flag, remained in NORMAL state throughout the same observation period, with no comparable BEGIN_DIR storm. *An additional observation: StaleDesc* Later on October 13, I noticed that both affected relays had acquired the StaleDesc flag while still retaining Fast, Guard, Running, Stable, V2Dir, and Valid. Interestingly, the archived 13:00 UTC network consensus listed server descriptors with publication timestamps of 01:22:19 UTC for Cloud2 and 01:22:27 UTC for Cloud3 — only eight seconds apart, and very close to the observed loss of HSDir on both relays. As I understand it, StaleDesc indicates that the directory authorities consider a relay’s published server descriptor too old (normally more than 18 hours). If no newer descriptors were accepted, those timestamps would explain the appearance of StaleDesc later that evening. We have not yet established why newer descriptors were not reflected in the consensus, whether the same thing happened during previous storms, or whether this is related to the overload. Has anyone else observed StaleDesc appearing after a severe BEGIN_DIR storm, particularly without a Tor process restart? *Monitoring limitations and possible improvements* Our current Sentinel relies on successful Tor MetricsPort responses to measure BEGIN_DIR activity and detect storms. Although it also collects Tor process information directly from Linux /proc, these measurements are tied to the same sampling loop. During prolonged MetricsPort timeouts, we lose the ability to observe the actual BEGIN_DIR rate. When MetricsPort eventually responds again, the calculated rate represents an average over the elapsed interval, potentially hiding much higher short-lived peaks. We have also identified a CPU calculation issue following failed MetricsPort scrapes, which may underestimate CPU utilization during recovery. We are therefore considering adding an independent, lightweight Linux monitoring component that would continue collecting: - Network interface RX/TX rates and traffic asymmetry. - Tor process and main-thread CPU utilization. - Resident memory, available RAM, and swap usage. - System CPU and memory pressure. - MetricsPort response latency and timeout frequency. The goal would be to preserve useful host-level measurements even when Tor’s MetricsPort becomes temporarily unresponsive, without interfering with Tor or collecting user-identifying traffic information. This is currently only a design proposal; no changes have been deployed. *A few questions for forest and the Tor developers* 1. *MetricsPort responsiveness:* Have other relay operators experienced similar prolonged MetricsPort timeouts during BEGIN_DIR storms? Is this a known consequence of Tor’s main event loop becoming overloaded? 2. *HSDir and consensus behavior:* Could the near-simultaneous loss of HSDir and Guard on two separately hosted relays be explained by the way directory authorities evaluate overloaded relays? Have others observed similar timing? 3. *StaleDesc:* Has anyone observed the StaleDesc flag appearing after severe BEGIN_DIR storms, particularly when the Tor process itself has not restarted? 4. *forest’s experimental patch:* I noticed that both of my non-exit relays report [DoSStreamCreationEnabled disabled] in their heartbeat logs. Forest, when you tested your patch moving BEGIN_DIR handling behind the existing stream-creation rate limiter, was DoSStreamCreationEnabled actually active on the test relay? Could this explain why the patch appeared to have no effect, or is there another issue with the per-circuit limiting mechanism? I would appreciate any technical insights, particularly regarding ways to observe these events more reliably without interfering with Tor’s normal operation. I can share additional Sentinel measurements and relevant Tor notice-log excerpts if they would be useful. Regards, Khan Krum Tor BEGIN_DIR Storm Sentinel: https://github.com/YankoNach/tor-begindir-sentinel On Sun, 4 Oct 2026 23:57:11 +0300, Khan Krum khankrum359@gmail.com wrote: Hi, I run three hobby relays and I seem to be seeing the same behavior on one of them. I compared two similar ARM, 2-core, non-exit Guard+HSDir relays, both running Tor 0.4.9.13. On the affected relay I measured 41,240 BEGIN_DIR streams in 61 seconds (~676/s). The Tor main thread reaches about 100% CPU, ntor onionskins are being dropped, and traffic is strongly asymmetric — roughly 2.2 MB/s read vs 7.1 MB/s written. On my other Guard+HSDir relay, measured at roughly the same time, I saw only 148 BEGIN_DIR streams in 60 seconds (~2.5/s). That relay carries much more traffic (~15 MB/s in both directions) and appears healthy. So in my case, simply having the HSDir flag does not seem sufficient to trigger the issue. One relay is affected, while another Guard+HSDir relay is not. I’m only a hobby relay operator, so I’m posting these numbers mainly in case they are useful for comparison. Regards, KhanKrum
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 2026-10-13 21:41, Khan Krum via tor-relays wrote:
*forest’s experimental patch:* I noticed that both of my non-exit relays report [DoSStreamCreationEnabled disabled] in their heartbeat logs. Forest, when you tested your patch moving BEGIN_DIR handling behind the existing stream-creation rate limiter, was DoSStreamCreationEnabled actually active on the test relay? Could this explain why the patch appeared to have no effect, or is there another issue with the per-circuit limiting mechanism?
That's strange. I did set "DoSStreamCreationEnabled 1". Could there be another bug elsewhere that's causing that option to be a no-op, even without my patch? On 2026-10-13 15:04, Mzungu via tor-relays wrote:
Was it determined that the attack you describe is general and not specifically aimed at you and all the relays you operate?
Yes, it is part of a wider attack. Only the relays of mine that obtain the HSDir flag are attacked, and I'm not the only person being impacted. On 2026-10-13 10:46, kullikovana via tor-relays wrote:
Im new to this and thinking about giving up. All the stats are f*cked again etc.
A temporary solution is to add "DirCache 0" to your torrc. It will stop further incidents of this particular attack, but your relay will not be a guard, just a middle. Regards, forest -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQQtr8ZXhq/o01Qf/pow+TRLM+X4xgUCas68dQAKCRAw+TRLM+X4 xuVcAQD5dcCA7luOSMtxUYt7IvZAoXfUbkIZryVdFkPQk3G1cAEAvmF7YYLgrYnF 2R/wtHOz89zWsAU1toTub8ZvzT1R/wo= =AeIV -----END PGP SIGNATURE-----
participants (4)
-
forest -
Khan Krum -
khankrum359@gmail.com -
kullikovana