[tor-relays] My VPS relay has just been hacked

[Nick Sheppard]

For the last month I've been running a middle relay (no guard flag yet) 
on a 512 MB VPS provided by Edis.at in Switzerland (4.99 euro per 
month).  The software is Tor 0.2.5.8-rc with obfsproxy 2 and 3 running 
on up-to-date Debian Wheezy 7.6.  I left the iptables at their defaults.

Last night Edis suspended my VPS because of a suspected outgoing DDoS 
attack, and emailed me.  I rang their (very helpful) phone support in 
London and they de-suspended the VPS so I could ssh into it and 
investigate. ( I told them it was running a tor non-exit relay - that 
wasn't a problem.)

Sure enough the Solus control panel traffic graphs show the tor relay 
traffic stopping abruptly last night, followed about 40 minutes later by 
an exponentially increasing spike of outgoing traffic, soon cut off when 
the VPS was suspended.

I ssh'd into the VPS ("last login" showed my own last login, so no 
problem there) and looked at logs and top.  notices.log simply cut off 
when the tor traffic stopped, but showed nothing unusual before that.

The Solus control panel traffic graph started showing (a very small 
amount of) outgoing traffic as soon as the VPS was de-suspended, so I 
assumed the malware was still active, and used top.

This is typical of what I found.

1    root 20 0 10604  832  700 S ... 0:00.10 init
2    root 20 0     0    0    0 S ... 0:00.00 kthreadd/3277
3    root 20 0     0    0    0 S ... 0:00.00 khelper/3277
1370 root 20 0 36976  660  492 S ... 0:00.38 hxyqbutesc
1400 root 20 0  109m 1616 1188 S ... 0:00.00 rsyslogd
1446 root 20 0 18836  884  680 S ... 0:00.00 cron
1473 root 20 0 49888 1212  608 S ... 0:00.00 sshd
3187 root 20 0 27556  744  504 S ... 0:00.00 vzctl
3188 root 20 0 17808 1968 1500 S ... 0:00.00 bash
5374 root 20 0 21584 1416 1072 R ... 0:00.00 top
5440 root 20 0  6244  536  296 S ... 0:00.00 akcviaxtbl
5443 root 20 0  6244  532  296 S ... 0:00.00 akcviaxtbl
5446 root 20 0  6244  532  296 S ... 0:00.00 akcviaxtbl
5448 root 20 0  6244  536  300 S ... 0:00.00 akcviaxtbl
5449 root 20 0  6244  532  296 S ... 0:00.00 akcviaxtbl

There is a strange line near the top for process "hxyqbutesc", which 
didn't change; and a strange block of lines at the bottom, which changed 
every second or two. Sometimes it was five similar lines, as above; 
sometimes it was several block of lines, eg this:

5276 root 20 0 6244 528 292 S  0.0  0.1 0:00.00 qscntoweqb
5277 root 20 0 6244 536 300 S  0.0  0.1 0:00.00 qscntoweqb
5280 root 20 0 6244 532 296 S  0.0  0.1 0:00.00 qscntoweqb
5282 root 20 0 6244 536 300 S  0.0  0.1 0:00.00 qscntoweqb
5283 root 20 0 6244 528 292 S  0.0  0.1 0:00.00 qscntoweqb
5290 root 20 0 6244 532 296 S  0.0  0.1 0:00.00 saqizxaihz
5294 root 20 0 6244 528 300 S  0.0  0.1 0:00.00 saqizxaihz
5295 root 20 0 6244 532 296 S  0.0  0.1 0:00.00 saqizxaihz
5296 root 20 0 6244 524 296 S  0.0  0.1 0:00.00 saqizxaihz
5298 root 20 0 6244 528 296 S  0.0  0.1 0:00.00 saqizxaihz

Each block is always 5 lines, and the names (always 10 lower-case 
letters) seem to be different every time.  The blocks change fairly 
regularly every second or two.

I shut the VPS down to stop it doing any more harm, but I didn't delete 
anything; I can restart it and ssh in again for further investigation if 
necessary.

Eventually I'll have to reinstall everything from scratch, 
straightforward enough, but what can I do to make sure it doesn't happen 
again?  Would hardening my iptables work?  Has anyone else seen this?

Nick Sheppard