[tor-bugs] #33406 [Internal Services/Tor Sysadmin Team]: automate reboots

[Tor Bug Tracker & Wiki]

#33406: automate reboots
-------------------------------------------------+---------------------
 Reporter:  anarcat                              |          Owner:  tpa
     Type:  project                              |         Status:  new
 Priority:  Low                                  |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Major                                |     Resolution:
 Keywords:  tpa-roadmap-march                    |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+---------------------

Comment (by anarcat):

 note that this may very well mean just removing tsa-misc/reboot-host and
 tsa-misc/reboot-guest, and documenting the multi-tool better. :)

 i just tried `./torproject-reboot-rotation` and ``./torproject-reboot-
 simple` and the unattended operation isn't great... it fires up all those
 reboots, and doesn't show clearly what it did. for example, it seems to
 have queued reboots on a bunch of hosts, but it doesn't say which.

 after further inspection (with `cumin '*' 'screen -ls | grep reboot-
 job'`), i have found it has scheduled reboots on

 * static-master-fsn.torproject.org
 * cdn-backend-sunet-01.torproject.org
 * web-fsn-01.torproject.org
 * onionoo-frontend-01.torproject.org
 * orestis.torproject.org
 * nutans.torproject.org
 * chives.torproject.org
 * onionbalance-01.torproject.org
 * listera.torproject.org
 * peninsulare.torproject.org

 Most of those are okay and should return unattended. But in some cases,
 those could have been covered by a libvirt reboot (i had performed those
 before, in this case, so non were). Eventually though, that point is moot
 because we'll all be running under ganeti and will separate host and guest
 reboot procedures.

 one host is problematic in there (chives) as it needs a specific warning
 to users. maybe chives should be taken out of "justdoit" rotation...

 i also wonder, in general, if we should warn users about those reboots, as
 part of the reboot script.

 then i don't know which hosts are left to do manually, but i guess that,
 with time, nagios will let us know. it would be nice to have a scenario
 for those as well.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33406#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online