Hi all,
Following hiro's "Security incident affecting some relay operators" post on
Oct 5, here is what we found on our own infrastructure, in case it helps
others check theirs. Two of our servers were rooted with the Ebury OpenSSH
backdoor (self-reported version 1.8.3), and we use these two names for them
throughout:
- the management server: our Contabo VPS, which was our WireGuard VPN
gateway and SSH jump host and also ran an obfs4 bridge;
- Switzerland2: a FranTech VPS running two of our Quetzalcoatl exit relays
(29FEFE36A5F66A6C93D24775B9D2364A3831B597 and
8427937D5A39E15699C850F26FED3CD59C379C48).
When we examined them, neither server showed the NAT redirect, new users or
SSH keys, cron jobs, systemd units or modified tor binary from the
advisory's checklist. Both were examined read-only through several
independent AI-assisted review passes, cross-checked against logs, process
memory and public Tor Metrics data. All times are UTC.
One caveat up front: the attacker truncated some logs on the management
server, timestomped the files they planted, and turned off sshd's login
logging on Switzerland2, and Ebury's backdoor logins leave no login records
or shell history at all. Some details are therefore missing, and parts of
this timeline are inferred from what was left; we say so where it matters.
1. TIMELINE
- Background (management server): SSH password authentication was enabled,
and an encrypted backup of our admin workstation was stored on the
management server. That backup held our operator SSH private key, our
ansible inventory and copies of our relays' offline Ed25519 master keys.
- Background (both servers): both trusted the same operator SSH key, and
the operator account's password was also its sudo password, sat in
cleartext in a helper script on both servers, and was the backup's
passphrase. We don't know which of our other relays had that script.
- Aug 28, 03:19: the attacker logged in to the management server from
185.225.226.62 (VikHost, AS207560) with the operator account's password on
the first try, with no failed attempt from that IP ever recorded, so they
almost certainly knew the password in advance. By 03:24 they had root via
sudo su.
- Aug 28, 03:25-03:43: the attacker installed Ebury on the management
server, truncated some logs and installed gcc. They then tried apt inside
our WireGuard dashboard (WGDashboard) container, which runs on Alpine Linux
and only has Alpine's apk package manager; when apt failed, an SSH client
and nmap appeared in the container within minutes (dated by file change
times).
- Aug 28, 04:33-04:37: the same IP opened two SSH connections to
Switzerland2 as the operator account and disconnected before
authenticating; Switzerland2 only accepted SSH keys.
- Aug 28, 04:38: Switzerland2 accepted a login for the operator account
from the management server's IP address (also our VPN exit), using an RSA
key that was in no authorized_keys file we found on Switzerland2 (how it
was accepted is still open; see section 3). One second later, sudo bash -c
'echo <base64> | base64 -d | perl' installed the same Ebury build on
Switzerland2.
- The management server's logs show no SSH login or sudo at 04:38, and the
connection's source port fits the management server's own port range rather
than our VPN clients', so the attacker most likely made this hop through
the management server's backdoor.
- Aug 31 - Sep 6 (management server): root actions with no sudo record
behind them installed nmap on the management server itself on Aug 31. On
Sep 6, starting while no one was logged in, more such root actions pulled
an alpine image, loaded NFS client kernel modules, installed duplicity and
decrypted the workstation backup's metadata (whether the backup itself was
copied out is open).
- Sep 30 (Switzerland2): the attacker restarted sshd from inside the
backdoor, after which sshd stopped writing its "Accepted ..." lines, the
ones that record the user, key and source address of each login, because
its LogLevel was ERROR. The PAM "session opened" lines kept appearing, but
they record neither key nor source address.
- Sep 30 - Oct 1 (Switzerland2): the attacker installed curl and
wireguard-tools outside sudo, loaded the WireGuard kernel module, and
started hidden processes named guard, then failover and snf, none of which
left binaries on disk (what they did is open).
- Oct 1, 15:35-15:43 (Switzerland2): with no SSH login, sudo command or
cron job in the logs, the attacker prepended these lines to /etc/tor/torrc,
sent Tor SIGHUPs and then restarted it:
# ==== OUR_POLICY_BEGIN 2026-10-01 ====
ExitPolicy reject *:25
ExitPolicy reject *:465
ExitPolicy reject *:587
ExitPolicy reject *:110
ExitPolicy reject *:143
ExitPolicy reject *:993
ExitPolicy reject *:995
ExitPolicy reject *:3389
ExitPolicy reject *:135
ExitPolicy reject *:137-139
ExitPolicy reject *:445
ExitPolicy accept *:*
# ==== OUR_POLICY_END ====
That file is unused on Switzerland2: the two relays run as Debian tor@
instances that read /etc/tor/instances/<instance>/torrc, and the only unit
that reads /etc/tor/torrc (tor@default) was masked. The published policy
did not change.
- Oct 2, 09:34 (Switzerland2): again with no SSH login, sudo command or
cron job in the logs, both instance torrcs were rewritten. Our
ansible-generated accept-list (60 accepted port ranges and a final reject
*:*) was removed and five lines were appended after "# end of torrc":
ExitPolicy reject *:25, *:465, *:587, *:993 and *:995.
- Oct 2, 09:34:04: both Switzerland2 relays published the result, Tor's
default rejects plus those five ports and then accept *:* (confirmed in the
public CollecTor archive). At 09:42 the attacker tidied up: they emptied
the OUR_POLICY block in /etc/tor/torrc, appended the same five reject lines
to that file, and saved copies of all three files as *.bak20261002.
- Our reading of Oct 1-2, a hypothesis the logs can't prove: the attacker
assumed Tor reads the default /etc/tor/torrc, saw that reloads and a full
restart didn't change the published policy, and came back the next day to
edit the instance files. The policy they tried first also differed: it
blocked plaintext POP3/IMAP and RDP, while the one that went live opened
those and blocked the TLS mail ports instead.
- Oct 4, 05:46 (Switzerland2): a routine kernel upgrade and reboot killed
the hidden tools, but Ebury came straight back.
Oct 4, 13:56 (Switzerland2): after the Tor Project's network health team
spotted the exit-policy change and contacted us, we confirmed we hadn't
made it, asked for the relays to be marked bad and stopped Tor on
Switzerland2. The open policy had been live for about 52 hours on two
Guard+Exit relays moving roughly 3.4 TiB a day.
- Oct 5, 08:56-10:26: relays on 55 of our family's hosts were restarted in
one scripted sweep, in text-sorted IP order (who ran it is open). On
Switzerland2 that restart came from inside the backdoor, the directory
authorities rejected both relays' descriptor uploads (HTTP 400), and we
stopped Tor again.
- Oct 6: Onionoo's family query listed 168 of our relays on 57 IPs, all
down and all advertising that same near-open policy. Both servers are now
offline and deleted, and all of our relays and our bridge are retired and
blocklisted.
2. WHAT WE LEARNED
- The root cause was ours: one password reused for SSH login, sudo and the
backup passphrase, stored in plaintext on both servers, plus SSH password
authentication left enabled on the internet-facing management server. The
attacker already had the password when they first connected.
- The management server was a single point of failure. From it the attacker
reached Switzerland2 from an IP address that blended in with our
automation, every device on our WireGuard VPN was in reach, and Ebury also
hooks the SSH client on the management server.
- This Ebury is one unowned, setuid file,
/usr/lib/x86_64-linux-gnu/libkeyutils.so.1.10.2 (63,368 bytes,
timestomped), with the libkeyutils.so.1 symlink re-pointed at it. The
packaged library is untouched, so package verification (dpkg -V, debsums)
won't catch it, and sshd, sudo and PAM matched their package checksums on
both servers.
- On Debian 13 (Switzerland2) it is loaded by sshd-session through libkrb5,
so ldd /usr/sbin/sshd and the sshd listener process look clean; from there
it spreads into every process in an SSH session, sudo included.
- Infected processes are lied to (a shell can't see the library in its own
/proc/self/maps). The credential collector runs as a fake /sbin/rsyslogd
(earlier as systemd-udevd or arpd) on abstract socket @event-0YsOG9WSOE,
inside ssh.service's cgroup on Debian (Switzerland2) and, after a reboot,
inside init.scope on Ubuntu 24.04 (the management server), where
sshd-socket-generator starts it at boot.
- Backdoor logins bypass PAM, logging and shell history; the implant's
config includes UsePAM no, LogLevel QUIET, HISTFILE=/dev/null,
AuthorizedKeysFile /proc/self/environ and PermitTunnel yes. The main traces
were root actions no logged-in user could have made: apt history entries
without "Requested-By:", su sessions opened by uid 0, systemctl calls from
the ssh.service cgroup, Tor reloads nobody had asked for, and systemd's
"remains running after unit stopped" when sshd restarted.
- SSH encrypts passwords in transit, but sshd has to decrypt them to check
them, and Ebury runs inside sshd. So it captures every login password
exactly as typed, failed attempts included, notes which key logged in from
where for key logins, and it captured the password our investigators used
from every address they logged in from.
- Version 1.8.3 is newer than ESET's public IOCs (last updated May 2024, up
to 1.8.2), its socket name isn't on their list, and ESET's detect_ebury.sh
would not flag it, although their YARA rule libkeyutils_with_ctor still
matches. Each install stamps an 8-byte victim ID into the file, so the
implant's hash differs on every server.
- The policy flip opened plaintext POP3/IMAP, telnet, IRC, SSH and RDP
while closing IMAPS/POP3S, so mail retrieval through the Switzerland2
relays could only happen in cleartext. A hidden process named snf was
running on Switzerland2 for most of that window.
- It looks like the attacker went well beyond these two servers: every
relay Onionoo listed in our family on Oct 6 advertised the same near-open
policy, and the Oct 5 restart sweep covered 55 of our hosts. The management
server and Switzerland2 shared an operator key and password; we didn't
examine our other hosts, so we can't say whether they did too.
- Some of our review passes got it wrong. One concluded there was no
intrusion (it dismissed the setuid library as normal, trusted a clean dpkg
-V and grouped logins by IP instead of key), and others concluded the
policy change never reached the published descriptors; the public CollecTor
descriptor archive and reads of process memory settled it.
- Our "offline" master keys were not offline: copies sat in the workstation
backup on the management server, encrypted with the password the attacker
already had.
- systemctl disable tor is not enough on Debian multi-instance relays,
because instance units can be enabled on their own (on Switzerland2 they
were, and the attacker added another boot link); mask the tor@ units, and
since a live backdoor can undo that, take the affected server offline too.
- On Switzerland2, sudo logged the entire base64 installer across 67
"(command continued)" journal records, so we could decode it. On the
management server the attacker ran sudo su first, and nothing of the
install command was logged.
3. OPEN QUESTIONS
- Where did the attacker get the operator password before Aug 28? Not by
brute force, and not through the WGDashboard on the management server (its
web UI had been failing to start since April); the candidates are another
of our relays, the admin workstation, or reuse somewhere we haven't found.
- How did Switzerland2's sshd accept a key that was in no authorized_keys
file, one second before Ebury was installed on Switzerland2? ~/.ssh and
sshd_config.d on Switzerland2 never changed, which leaves a line in the
main sshd_config (replaced on Sep 30, so we can't check it) or an earlier
backdoor.
- What did guard, failover and snf do on Switzerland2? Their binaries died
with the Oct 4 reboot, so we don't know whether the newly opened port 22
was used to intercept SSH or whether a WireGuard tunnel was built (udev
logged two network-interface events on Sep 30, but no interface or config
survived).
- Where do the stolen credentials go? ESET's research on earlier versions
describes operators pulling them through the backdoor, with an automatic
encrypted send over UDP port 53 if nobody collects them for two weeks. On
our servers, memory holds an https://%s/xpost.php template but no host, and
a sandboxed run made no outbound connection, but the implant checks for
packet sniffers and strace, so that test doesn't settle it.
- Which of our other relays had their exit policy changed, when, and were
any non-exits turned into exits? We are pulling the public CollecTor
consensus and descriptor history for the whole family, back to May, and
will follow up.
- How many of the other 56 hosts in our family carry the implant, and was
any infected before Aug 28 03:25 (an earlier implant birth time would mean
an earlier victim)?
- Who ran the Oct 5 restart sweep, and from where: the management server's
backdoor, the admin workstation, or relay to relay? The logs on the
management server and Switzerland2 can't tell these apart, and we haven't
examined the workstation yet.
- Did the 5.3 GB workstation backup leave the management server? Its
metadata was decrypted on the management server on Sep 6 and NFS client
modules were loaded that night, but nothing in the logs shows a transfer.
- Was there a NAT redirect on Switzerland2 before the Oct 4 reboot?
iptables was installed outside sudo on Sep 2 (we don't know by whom), and
unsaved rules would not survive a reboot.
- Is this the same actor as the other cases behind hiro's notice (NAT
redirects, a fake uppercase CRON)? If you have seen libkeyutils.so.1.10.2,
@event-0YsOG9WSOE, 185.225.226.62 or the SSH keys listed below, please tell
us or bad-relays@.
4. BEST PRACTICES
- Never put passwords in scripts, and never reuse one across SSH login,
sudo and backups. Use key-only automation with narrowly scoped NOPASSWD
sudo rules.
- Use key-only SSH everywhere, especially on jump and management servers,
and reach them only through a VPN or an IP allow-list. Check the effective
config with sshd -T, because sshd takes the first value it finds (on
Switzerland2, a PermitRootLogin yes left near the top by the VPS image
silently beat the PermitRootLogin no our setup script appended).
- Treat your management/VPN server as the crown jewels: nothing web-facing
on it, no key backups on it, and no admin tooling in its containers. The
WGDashboard 4.3.0.1 on our management server was exposed to the internet
(its web UI happened to be broken since April), has an unauthenticated
file-read bug (CVE-2026-44343, fixed in 4.3.2) and logs every sign-in
password in cleartext.
- Keep OfflineMasterKey material genuinely offline. If a machine that ever
held it is compromised, retire those identities and roll a new family key
and FamilyId.
- Check for this Ebury from a rescue system or an offline disk image if you
can. On a live server, log in with a key (never a password) and use ESET's
clean-shell trick, H=1 LD_PRELOAD=":" LD_DEBUG=":" bash; a setuid
libkeyutils file, a libkeyutils.so.1 symlink to a file no package owns, an
@event- abstract socket, or a non-sshd process lingering in the ssh.service
cgroup is a strong sign of infection:
find / -xdev -name 'libkeyutils*' -perm -4000 2>/dev/null
dpkg -S "$(readlink -f /usr/lib/x86_64-linux-gnu/libkeyutils.so.1)"
(rpm -qf on RPM systems)
grep -E '@event-|@/dev/(event|stats)-' /proc/net/unix
systemctl status ssh | sed -n '/CGroup/,$p' (on Ubuntu 24.04, check
init.scope too)
- Search your journal for apt transactions with no "Requested-By:"
(unattended-upgrades also lacks it), su sessions opened by uid 0,
ssh.service "remains running after unit stopped" or unusually large
"Consumed ... CPU time" lines (Switzerland2's showed 41 minutes and 281
MB), "Reload requested from client ... (unit ssh.service)", and Tor reloads
nobody on your team caused. On Ubuntu, the install also leaves
apparmor="DENIED" operation="bind" profile="rsyslogd" addr="@event-..." in
the kernel log.
- Look for sudo running "base64 -d | perl" (long commands are split across
"(command continued)" records) or "echo 3 > /proc/sys/vm/drop_caches", a
missing /etc/falco/rules.d/ebury.yaml if you run Falco, and logins from
185.225.226.62, and review accepted logins by key fingerprint, not just
source IP. Keys: RSA SHA256:dvACokMYcXJ2om4wuS+TvdtjrAgPK7rC7ZQHDgvBHzk
(used to log in to Switzerland2) and the two embedded in the implant, RSA
SHA256:KlICz/B59619CpuQdGzgmmvzWBfTMUJ77EwihTYobuo and ED25519
SHA256:OfLY0uF9SwOW9xVjiMXZGDfk6Atmz6CbHy4epknLm8Y.
- Monitor your relays from outside, as the Tor Project's network health
team did for us. Alert when the exit policy, family or contact info in your
published descriptors (Onionoo or CollecTor) differs from what you
deployed, since files on a compromised server can be rewritten.
- Ship logs off your servers, keep NTP on, alert if sshd's LogLevel drops
below INFO, and watch for unowned or setuid files in library directories
(AIDE or similar).
- Never type a password into a server you suspect, because Ebury will
capture it. Investigate with keys from a clean machine or from an offline
image, and capture memory before rebooting or wiping.
- If you find it, take the relays down and mask the tor units, snapshot the
disk, and rebuild from scratch rather than cleaning in place. Then rotate
every SSH key, password and WireGuard key, generate new relay identities,
and report to bad-relays(a)lists.torproject.org.
Thanks to the Tor Project's network health team for catching the change and
acting quickly. We're happy to share more detail, including the de-fanged
installer, with anyone investigating a similar case.
Zimmer
Quetzalcoatl relays
