First, many thanks for all the advice. I will do some serious homework.
To the question What kind of virtualisation? It's OpenVZ. In fact it's the OVZ Starter option from here:
http://www.edis.at/en/server/linux-vps-solusvm-openvz/zurich
and the OS option I chose is what Edis describe as "Debian 7 amd64 minimal (64bit)"
I've restarted the VPS for 5-10 minutes at a time, long enough to watch the malware at work but not long enough for it to ramp up its flood of output. It seems to live here, in /boot:
root@xxxxx:/boot# ls -al total 128 drwxr-xr-x 2 root root 4096 Oct 26 14:19 . drwxr-xr-x 23 root root 4096 Oct 26 13:58 .. -rwxr-xr-x 1 root root 122698 Oct 24 17:19 hxyqbutesc
and it's started in a straightforward way from /etc and /etc/init.d ; rc1.d, rc2.d and so on have links to /etc/init.d/hxyqbutesc :
root@xxxxx:/etc# ls -l rc1.d total 4 1 root root 18 Sep 13 20:21 K01quotarpc -> ../init.d/quotarpc 1 root root 13 Sep 14 22:41 K01tor -> ../init.d/tor 1 root root 17 Sep 13 20:21 K04rsyslog -> ../init.d/rsyslog 1 root root 369 Aug 5 07:47 README 1 root root 18 Sep 13 20:21 S01bootlogs -> ../init.d/bootlogs 1 root root 20 Oct 24 17:19 S01hxyqbutesc -> ../init.d/hxyqbutesc ...
root@xxxxx:/etc# ls -l rc2.d total 4 1 root root 677 Aug 5 07:47 README 1 root root 18 Sep 13 20:21 S01bootlogs -> ../init.d/bootlogs 1 root root 20 Oct 24 17:19 S01hxyqbutesc -> ../init.d/hxyqbutesc ...
and in /etc/init.d is a short script:
root@xxxxx:/etc/init.d# ls -l total 196 -rw-r--r-- 1 root root 2427 Aug 5 07:47 README -rwxr-xr-x 1 root root 1276 Aug 5 07:47 bootlogs -rwxr-xr-x 1 root root 1281 Aug 5 07:47 bootmisc.sh -rwxr-xr-x 1 root root 3816 Aug 5 07:47 checkfs.sh -rwxr-xr-x 1 root root 1099 Aug 5 07:47 checkroot-bootclean.sh -rwxr-xr-x 1 root root 9673 Aug 5 07:47 checkroot.sh -rwxr-xr-x 1 root root 3033 Aug 5 07:47 cron -rwxr-xr-x 1 root root 1329 Aug 5 07:47 halt -rwxr-xr-x 1 root root 1423 Aug 5 07:47 hostname.sh -rwxr-xr-x 1 root root 3880 Aug 5 07:47 hwclock.sh -rwxr-xr-x 1 root root 317 Oct 26 13:58 hxyqbutesc ...
which looks like this:
root@xxxxx:/etc/init.d# cat hxyqbutesc #!/bin/sh # chkconfig: 12345 90 90 # description: hxyqbutesc ### BEGIN INIT INFO # Provides: hxyqbutesc # Required-Start: # Required-Stop: # Default-Start: 1 2 3 4 5 # Default-Stop: # Short-Description: hxyqbutesc ### END INIT INFO case $1 in start) /boot/hxyqbutesc ;; stop) ;; *) /boot/hxyqbutesc ;; esac
and runs the main file /boot/hxyqbutesc , which is mainly hex numbers with many embedded filenames and paths. The timestamp Oct 24 17:19 is an hour or two before the VPS was suspended on Friday evening (can't be more precise yet because I'm not sure what timezone the suspension email was sent from).
Thank you to the people who suggested using ps as well as top. ps -x gives eg this when the malware is working:
PID TTY STAT TIME COMMAND 1 ? Ss 0:00 init [2] 2 ? S 0:00 [kthreadd/3277] 3 ? S 0:00 [khelper/3277] 1373 ? Ssl 0:00 who 1406 ? Sl 0:00 /usr/sbin/rsyslogd -c5 1451 ? Ss 0:00 /usr/sbin/cron 1478 ? Ss 0:00 /usr/sbin/sshd 1857 ? Ss 0:00 vzctl: pts/0 1858 pts/0 Ss 0:00 -bash 3474 ? Ss 0:00 ls -la 3477 ? Ss 0:00 cat resolv.conf 3478 ? Ss 0:00 route -n 3479 ? Ss 0:00 ifconfig eth0 3481 ? Ss 0:00 su 3489 ? Ss 0:00 sleep 1 3492 ? Ss 0:00 ifconfig 3493 ? Ss 0:00 sh 3494 ? Ss 0:00 grep "A" 3496 ? Ss 0:00 pwd 3497 pts/0 R+ 0:00 ps -x
For process 1373 top would give "hxyqbutesc" instead of "who" (this line doesn't change), and the ten lines for 3474 - 3496 would be the two changing blocks of ten-letter random names. Assuming the glimpses of command names here can be taken at face value, something very sinister is clearly going on ... other commands that flash into view are "who, whoami, uptime, bash, id, cd /etc, netstat -antop, top, echo "find", gnome-terminal".
As for how it got in, most people seem to suspect an attack through ssh. But when I ssh'd in yesterday to start investigating, I was careful to note the "Last Login" message, and it gave my own genuine last login, three weeks ago from my home IP. Can I take this at face value to mean that ssh has not been used between my own two logins? Can "Last Login" be falsified?
Some suggest an exploit using out-of-date software. Well, I last did an apt-get dist-upgrade 29 days ago, (after the bash fix), and tor and obfsproxy (included in the dist-upgrade) were the only things I installed over the "minimal" Debian 7.6 - so I was reasonably up-to-date.
And some suggested that the malware might have come through the Solus control panel (which of course has root access - I ssh in by requesting a temporary (time-limited) console from Solus, and it gives me a one-time strong password to use for that). Is this plausible? perhaps the next step is to send all this to Edis support and see if they noticed anything unusual around 17:19 last Friday?
Thanks again for everybody's help,
Nick Sheppard