[tor-bugs] #30912 [Internal Services/Tor Sysadmin Team]: Investigate stunnel outage on crm-ext-01

[Tor Bug Tracker & Wiki]

#30912: Investigate stunnel outage on crm-ext-01
-------------------------------------------------+---------------------
 Reporter:  peterh                               |          Owner:  tpa
     Type:  defect                               |         Status:  new
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+---------------------

Comment (by anarcat):

 so last time this happened, I *did* look at the (stunnel? or redis? both?
 i forgot) logs but didn't see anything fishy. one thing that *did* happen,
 it seems, is that:

 > it looked like the stunnel process on crm-int-01 was gone. I restarted
 both stunnel processes and things seem to be back in order now.

 So maybe the simplest fix would be to tweak the systemd .service file for
 stunnel to forcibly restart the process when it exits, regardless of the
 status. I think that by default, systemd doesn't always restart services
 on crash so this might fix the problem for us.

 What I suspect happened is we rebooted boxes for security upgrades
 recently. Maybe that threw the stunnels out of whack... I don't know. We
 have more security reboots to performed soon, so I'll hold off on
 deploying a fix here to see if the reboot causes the problem.

 spiped looks interesting (thanks! didn't know about that one before!) but
 I'm not sure it's the right solution now because it only introduces
 another "unknown" with similar properties as stunnel. if I would fix this
 another way, i would create an IPsec tunnel between the two machines,
 something we already have code to automatically deploy for (as opposed to
 spiped, which isn't used anywhere in TPA yet).

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/30912#comment:2>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online