@jj tor
The fact that your relay is refusing connections says that the port isn't open, which is a good thing.
I suspect that persons unknown have port scanned your VPS, realised that you have Tor running (on standard ports) and is speculatively using a bot to (hopefully) connect to the SOCKS interface.
I would
a) move the Tor relay to non-standard ports
b) use iptables to drop all incoming connections apart from the (new) Tor ports and shell access.
Best,
--
Parity
parity.boy@gmail.com