I've thought about constructing iptables rules to limit the number of SYN packets for the same host per second or such
Multiple flows to the same host don't really bother routers of any class. Old routers choke when looking up many hosts in the routing table. So your proposed rules against port-scanning single hosts wouldn't help. Unless each SYN to a host is generated from multiple Tor-based IP-scanner's, in which case your node or Tor would probably be underwater from the parallel scans anyways.
Is there a known proper way to protect yourself from being used as a network scan relay?
You can't really implement rules to block IP-scanning because you'll just take yourself offline. Which is exactly what ISP's do when their router falls over. The problem is fixed at the source, not the dest.
In the TCP only case of Tor, best you can easily do is 'reject *:port' the ports being scanned, thus denying service to the scanner's Tor client and thus emitting no such traffic yourself. If it's well-known ports, such is life for your relay.
I am hosting a 3-5MB/s tor exit relay
...
does not want to route network scanning traffic since it is a severe load to their routers.
If they can't deal with a single host doing IP-routing lookups, sounds like they need to replace their 10yr old Crisco routers or exit the biz.