@jj tor
The fact that your relay is refusing connections says that the port isn't open, which is a good thing.
I suspect that persons unknown have port scanned your VPS, realised that you have Tor running (on standard ports) and is speculatively using a bot to (hopefully) connect to the SOCKS interface.
I would
a) move the Tor relay to non-standard ports b) use iptables to drop all incoming connections apart from the (new) Tor ports and shell access.
Best,