Undiagnosable Crashes in Relays
Hello List, I am wondering if anyone has had their relay randomly crash in the past week or so. Three of mine (I run 6, nonexits) have fallen over. One of them ~5 days ago, one of them ~4 days ago, and one of them earlier today . The logs indicate everything was operating normally, and the only reason I noticed they had crashed was when I was checking what bandwidth globe.torproject was measuring for them it said some were down. The affected relays are as follows: AC9803701F9EE18194D40B38E47CE4C68CF2F567 73067CD4ADD8A294BDA913DF45B63190A52B5F9F D76252B1A6E9F01FC6772CFFB651056A2B54F92B I have no idea why they all fell over, the last thing in the logs was the usual "current status" output with some traffic measuring, seemingly immediately afterwards, the process killed itself for no reason. All of the affected relays are on DigitalOcean droplets. I intend to launch more on EC2 in the near future, however, would like to know if anyone else has experienced relay instability issues over the past few days incase its something nasty like a denial of service attempt or something. Nothing on the boxes indicates any kind of compromise, just the usual bruteforce activity from the usual actors. I have decided to start writing a "Shitty Monitoring System" to copy logs down to my local box every hour or so and HUP tor at the same time as a kind of countermeasure incase its simply a resource use issue or something, however, would love to know if anyone else has been having similar issues lately. Regards, .d
On Wed, Mar 25, 2015 at 4:26 PM, skyhighatrist <skyhighatrist@tfwno.gf> wrote:
I am wondering if anyone has had their relay randomly crash in the past week or so. Three of mine (I run 6, nonexits) have fallen over. One of them ~5 days ago, one of them ~4 days ago, and one of them earlier today .
This also happened to my two relays about three weeks ago. They are: https://globe.torproject.org/#/relay/C309A31AD772FFDD0805C9FECB6D4748A7CBF68... https://globe.torproject.org/#/relay/18BE989663CF3351F73D33C672BB1C985E0EA5D... They are both middle/guard relays (about 200 Mbps each) on the 0.2.6 branch and are on the same dedicated server. They went down at different times, and as far as I can remember, there was nothing notable in the Tor and system logs. No issues prior to this over the past 12 months. I haven't had time to investigate this. Vincent
One of my relays went down a few weeks ago, and I didn't notice until a few days ago. https://atlas.torproject.org/#details/4CA46581FB3C82102565B02C1ECB6DD38EF665... I did find what caused it, but thus far I cannot remember what it was. If I remember, I'll post again.
As promised, heres my utterly hideous monitoring script for checking/restarting relays. Its a work in progress, any advice/comments would be greatly appreciated. https://github.com/0x27/relaycheck Hopefully we can get to the bottom of this wierd relay-collapse problem, its a bit of a nuisance! On 25/03/15 12:53, Speak Freely wrote:
One of my relays went down a few weeks ago, and I didn't notice until a few days ago.
https://atlas.torproject.org/#details/4CA46581FB3C82102565B02C1ECB6DD38EF665...
I did find what caused it, but thus far I cannot remember what it was. If I remember, I'll post again.
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
On Wed, Mar 25, 2015 at 4:26 AM, skyhighatrist <skyhighatrist@tfwno.gf> wrote:
Hello List, I am wondering if anyone has had their relay randomly crash in the past week or so. Three of mine (I run 6, nonexits) have fallen over. One of them ~5 days ago, one of them ~4 days ago, and one of them earlier today .
The logs indicate everything was operating normally, and the only reason I noticed they had crashed was when I was checking what bandwidth globe.torproject was measuring for them it said some were down.
The affected relays are as follows: AC9803701F9EE18194D40B38E47CE4C68CF2F567 73067CD4ADD8A294BDA913DF45B63190A52B5F9F D76252B1A6E9F01FC6772CFFB651056A2B54F92B
I have no idea why they all fell over, the last thing in the logs was the usual "current status" output with some traffic measuring, seemingly immediately afterwards, the process killed itself for no reason.
What version of Tor did these relays run? Is it possible that one of the crash bugs fixed in 0.2.5.11 is to blame? -- Nick
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I'll do an upgrade across all my relays tonight, hopefully that resolves the issue in the future. Interesting that SpeakFreely's relay was affected given its on the patched version... On 25/03/15 13:53, Speak Freely wrote:
Vincent Yu is Tor 0.2.6.5-rc on Linux skyhighartist is Tor 0.2.4.24 on Linux my affected relay Tor 0.2.5.11 on Linux
Cool.
Matt Speak Freely
Nick Mathewson:
What version of Tor did these relays run? Is it possible that one of the crash bugs fixed in 0.2.5.11 is to blame?
_______________________________________________ tor-relays mailing list tor-relays@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-relays
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBCAAGBQJVEs2ZAAoJEMRV9IW1w5addBwP+QFv2btGjYUE/E30+5w4aySn CiP9Xqep3xEHRKCfxE+zbAb+KIOLyJXfKQo5wF9UvOHr0WSD1UQHzaurwwnbgvyv ylvDkyRj7w2aYTfZiMi2aXlLuH5KKgaDPm72g9T8pL8rUwMfdMtK969wyayXOgJS Cm7klx+R4j4iK74bA5mselYeTgqfGBbHEvTk13HpII/WJfI3e87ilaqN/w8KyUnl T+0pEAJdLJML61/OIsarH6ydCePLNaE6Dbgy0BAI0vLWqeQILrIW8S0MpU1xq4zp 0GAFPa5qiuXDCYqOt5KSzqATUjJV9HwJqfbpLwNO58dVdpeDSDJaZNGOzkuuXM0V xDmKQ6ztolUkhJYhcwexhR+FjuiUXZW0i6kl9OMWdbYEyYg0X0J/kL7w6yx40BwP 8ikd9hcxXFYui8jURfqNw+iR8e7HrymvvgL7R5zny8qGmHGdwfxgjVBjrwVV+Zwv uWrElZpnQjx8ollpCjPaAumb/I2Wc6rcS0G7IysZKhWkLmlrrSGh3fJH2+L7aDIV EVDDCrrmz10aAJyNhVBlwGumLJv0qnWJiqJSremFIC3ZGEAMXb/zUKegz/lQyxE2 k3HnR+sklhchVq9CL9XKq/YbbwdSXAZhYQqpmFPwzz5rdBcaPCk8bMp/O/WmILCA Mjid9oC8CbRMV1Q/OzEE =go/q -----END PGP SIGNATURE-----
On Wed, Mar 25, 2015 at 08:26:20AM +0000, skyhighatrist wrote:
I have no idea why they all fell over, the last thing in the logs was the usual "current status" output with some traffic measuring, seemingly immediately afterwards, the process killed itself for no reason.
You might also enjoy https://www.torproject.org/docs/faq#TorCrash My first guess is that the out-of-memory killer killed it. --Roger
My apologies for my lack of memory... It's still fuzzy, but Roger helped jog the the gerbil into action. There was a line regarding "assert failure" in my logs. I could not get Tor to start again until I followed these instructions: https://trac.torproject.org/projects/tor/ticket/13111 My secret onion keys had 0 values, so I removed them and restarted Tor and it went on it's merry way. It's quite possible the cause of my crash was completely different than Vincent and skyhighatrist, as neither of them indicated they had to do that to get them back up and running. Matt Speak Freely
participants (5)
-
Nick Mathewson -
Roger Dingledine -
skyhighatrist -
Speak Freely -
Vincent Yu