Hi all, I am the other user running into trouble with Hetzner participating in the experiment using scamper. Conclusion first: For my server at Hetzner tor trace route using scamper at full rate was flagged as abuse but 1/5 (PPS=200) seems to be fine up to now. I started the script on Friday and shortly after got the first abuse report. However in my case they did not block the IP in the end, even though they sent me several machine generated abuse reports. Each time I stopped scamper, sent them the explanation what the script was doing together with the request to next time not flag it as abuse with no effect whatsoever. Then I resumed scamper. Stopping was necessary because their system only allows you to explain what was happening after you "fixed" it. I also tried to call them but didn't succeed to get the right people on the phone. The time frame they gave me for "fixing" the problem was usually only a few hours. Finally scamper was defunct, presumably due to being stopped two times, so I restarted the whole Trusted Tor Traceroutes script on monday with PPS=200 (reducing the traceroute rate to 1/5 of the default value). So far I did not receive any machine generated abuse reports. I assume the packet rate is now below the limit of what the monitoring thinks is a netscan. I will report back if I should receive another abuse report connected to the experiment. "Netscans" are listed as unacceptable in the Hetzner server policy [1]. And since one can apparently not or only with great pains communicate to a human in their support system there seems to be no chance of reliably informing them why this does not qualify as a netscan. For me a blocking of the IP would not be too painful since it looks like they will block by IP. The IP belongs to an "experimental" KVM I run as a sandbox to test things and which has nothing of importance running on it. Therefore I will just continue running the script with the current settings and see what happens. Best regards, Paul [1] http://www.hetzner.de/en/hosting/legal/system-policies-rs On 15.01.2014 06:00, Anupam Das wrote:

Hi Alex,

We are very sorry to hear about the problems our measurements caused. Up until yesterday, we had received no reports of them triggering these kinds of responses from providers. However, yesterday we heard a very similar story from another relay operator using Hetzner.

Thanks for sharing your experience with the tor-relays community. We have also updated our FAQ to inform contributors about this potential problem.

Also, we'd like to help others avoid this while still providing useful measurements, if possible. Have you gotten any feedback from Hetzner about what rule was triggered and maybe how to avoid it? Do you have any ideas about how one might stay below their radar? If it is something simple like reducing the measurement rate that would be a great option to prevent problems while still providing valuable data about the the Tor network.

We do still hope that most relay operators will be willing to give this project a shot. We have received data from over 90 separate IP addresses and have gotten 2 negative reports so far, although certainly the issues could be more widespread without us being aware. We don't want to add to the headaches that can result from running a Tor relay, but on the other hand Tor relay operators are probably pretty adept at handling this kind of stuff.

Thanks

Anupam

On Tue, Jan 14, 2014 at 5:02 PM, <irregulator@riseup.net <mailto:irregulator@riseup.net>> wrote:

Hello there,

We're running a Tor relay (not exit) on a virtual private server at Hetzner for about a year. On Wednesday January 8th, we decided to take part in the "Trying Trusted Tor Traceroutes" [1] research experiment. There have been various calls for participation on public mailing lists [2] [3].

The traceroutes were conducted using the scamper package, as suggested in README. We imposed no rate limiting to requests, just run the script with default values.

Some hours later, Thursday 9th, we received an email from Hetzner stating that our server was taking part in attacks and they would suspend our instance if we didn't react within 8 hours. As soon as we got the warning we killed scamper conducting the traceroutes, and followed the procedure so as not to get our instance suspended. Hetzner also asked for some explanations about why we think our server was not taking part in the attack.

We responded via email with a full explanation about the traceroutes from our server and the "Trying Trusted Tor Traceroutes" experiment from various researchers from University of Illinois [1]. We told Hetzner that our server was making harmless and legal traceroutes to various destinations on the Internet, thus they had no reason to suspend our instance.

Twenty four hours later, Friday 10th, Hetzner blocked network access to the IP address of our server, did send us an email about blocking, but ignored our exlanations submitted the previous day. After the blockage of our IP we insisted on trying to resolve the case by sending one more email exlaining the situation and asking to unblock us, and then opening a ticket. Hetner's response to the last email (5 hours later) was that we should open a ticket, which we already had done. Alas, our ticket was marked as duplicate and closed(?).

During this loophole support nightmare most responses from Hetzner's part actually seemed to be machine generated. At last Hetzner asked us via email to send them a signed document via fax(!) containing explanations about the incident. Now that was ridiculous, since we had submitted explanations already three times with the first submission only four hours after Hetzner's first warning on Thursday. Nevertheless, we did resend the explanation.

After about 7 hours of downtime, Hetzner unblocked network access to our server. More than 36 hours later they sent an email "Dear Mr. xxxx, your server is unlocked."

Concluding,

- Hetzner considers traceroutes to various internet destinations as attack. All relay operators with machines at Hetzner should be _careful_ when taking part in "Trying Trusted Tor Traceroutes" experiment.

- Hetzner has awful customer support.

Cheers, Alex

[1] https://web.engr.illinois.edu/~das17/tor-traceroute_v1.html

[2] https://lists.torproject.org/pipermail/tor-relays/2013-October/003113.html

[3] https://lists.torproject.org/pipermail/tor-news/2014-January/000027.html

_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org <mailto:tor-relays@lists.torproject.org> https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

-- Paul Görgen

Hi, Apparently even with the lowered rate from time to time the abuse system will complain. I just received an abuse message from Hetzner even though now running with the reduced rate. Just so you know. Next time this happens I will try to escalate the problem by not solving it in the framework of the automated abuse reports. Instead I will put the info about what I do into the trouble ticket of the abuse message and put a strong plea to contact me about if and how they can stop flagging it as abuse. Best regards Paul On 15.01.2014 16:41, irregulator@riseup.net wrote:

On 01/15/2014 07:00 AM, Anupam Das wrote:

...

Hi Alex,

We are very sorry to hear about the problems our measurements caused. Up until yesterday, we had received no reports of them triggering these kinds of responses from providers. However, yesterday we heard a very similar story from another relay operator using Hetzner.

Thanks for sharing your experience with the tor-relays community. We have also updated our FAQ to inform contributors about this potential problem.

Also, we'd like to help others avoid this while still providing useful measurements, if possible. Have you gotten any feedback from Hetzner about what rule was triggered and maybe how to avoid it? Do you have any ideas about how one might stay below their radar? If it is something simple like reducing the measurement rate that would be a great option to prevent problems while still providing valuable data about the the Tor network.

We do still hope that most relay operators will be willing to give this project a shot. We have received data from over 90 separate IP addresses and have gotten 2 negative reports so far, although certainly the issues could be more widespread without us being aware. We don't want to add to the headaches that can result from running a Tor relay, but on the other hand Tor relay operators are probably pretty adept at handling this kind of stuff.

Thanks

Anupam

Hi again,

Anupam I wish I knew how to run the script and avoid any complaints from Hetzner. Unfortunately Hetzner didn't give us any helpful info. We even asked them explicitly if rate limiting would be a solution, but there was no answer on that.

On 01/15/2014 02:20 PM, Paul Görgen wrote:

...

Finally scamper was defunct, presumably due to being stopped two times, so I restarted the whole Trusted Tor Traceroutes script on monday with PPS=200 (reducing the traceroute rate to 1/5 of the default value). So far I did not receive any machine generated abuse reports. I assume the packet rate is now below the limit of what the monitoring thinks is a netscan. I will report back if I should receive another abuse report connected to the experiment.

Paul's answer may indicate that imposing a rate limit to the script's requests might do the trick.

Greetings. Alex

_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

-- Paul Görgen Dieburger Straße 94a, Mobile +4917620181608 64287 Darmstadt http://www.pgoergen.de

Hi Paul, Great that you were able contact Hetzner. Now at least we know what is triggering their IDS. But I'm not sure we can work around this scenario because as you hinted- we don't know a head of time if certain IPs will be unrouted by intermediate routers. Moreover, their routing policies also change over time. We will have to think more elaborately on this. One simple solution might be to do traceroutes to only well known destination IPs but that would not give us the full picture. But again thanks for your cooperation and its fine if you can't run the script due to Hetzner's IDS policy. Thanks Anupam On Thu, Jan 16, 2014 at 10:32 AM, Paul Görgen <tor@pgoergen.de> wrote:

Hi again,

Hetzner now blocked my testing Server. This gave me chance to finally get into contact with a human person. They said their IDS triggered on "connects to unrouted IPs" (I assume that will be connects to unrouted IPs per time). I will try to convince them to set up an exception rule for my server but I am not too hopeful.

As a backup: I guess that there is no way to make sure that you don't try to connect to unrouted IPs, given that the IPs owner can decide to route or not route at will, right?

If both don't work I will not be able to run the script.

Best,

Paul Görgen

On 15.01.2014 23:23, Paul Görgen wrote:

...

Hi,

Apparently even with the lowered rate from time to time the abuse system will complain.

I just received an abuse message from Hetzner even though now running with the reduced rate. Just so you know. Next time this happens I will try to escalate the problem by not solving it in the framework of the automated abuse reports. Instead I will put the info about what I do into the trouble ticket of the abuse message and put a strong plea to contact me about if and how they can stop flagging it as abuse.

Best regards

Paul

On 15.01.2014 16:41, irregulator@riseup.net wrote:

...

On 01/15/2014 07:00 AM, Anupam Das wrote:

...

Hi Alex,

We are very sorry to hear about the problems our measurements caused. Up until yesterday, we had received no reports of them triggering these kinds of responses from providers. However, yesterday we heard a very similar story from another relay operator using Hetzner.

Thanks for sharing your experience with the tor-relays community. We have also updated our FAQ to inform contributors about this potential problem.

Also, we'd like to help others avoid this while still providing useful measurements, if possible. Have you gotten any feedback from Hetzner about what rule was triggered and maybe how to avoid it? Do you have any ideas about how one might stay below their radar? If it is something simple like reducing the measurement rate that would be a great option to prevent problems while still providing valuable data about the the Tor network.

We do still hope that most relay operators will be willing to give this project a shot. We have received data from over 90 separate IP addresses and have gotten 2 negative reports so far, although certainly the issues could be more widespread without us being aware. We don't want to add to the headaches that can result from running a Tor relay, but on the other hand Tor relay operators are probably pretty adept at handling this kind of stuff.

Thanks

Anupam

Hi again,

Anupam I wish I knew how to run the script and avoid any complaints from Hetzner. Unfortunately Hetzner didn't give us any helpful info. We even asked them explicitly if rate limiting would be a solution, but there was no answer on that.

On 01/15/2014 02:20 PM, Paul Görgen wrote:

...

Finally scamper was defunct, presumably due to being stopped two times, so I restarted the whole Trusted Tor Traceroutes script on monday with PPS=200 (reducing the traceroute rate to 1/5 of the default value). So far I did not receive any machine generated abuse reports. I assume the packet rate is now below the limit of what the monitoring thinks is a netscan. I will report back if I should receive another abuse report connected to the experiment.

Paul's answer may indicate that imposing a rate limit to the script's requests might do the trick.

Greetings. Alex

_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays

-- Paul Görgen Dieburger Straße 94a, Mobile +4917620181608 64287 Darmstadt http://www.pgoergen.de _______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays