[tor-bugs] #33406 [Internal Services/Tor Sysadmin Team]: automate reboots

[Tor Bug Tracker & Wiki]

#33406: automate reboots
-------------------------------------------------+---------------------
 Reporter:  anarcat                              |          Owner:  tpa
     Type:  project                              |         Status:  new
 Priority:  Low                                  |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Major                                |     Resolution:
 Keywords:  tpa-roadmap-march                    |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+---------------------

Comment (by anarcat):

 just for future reference, ganeti-reboot-cluster, as we have in our puppet
 repo, doesn't work in our cluster, because it relies on assumptions
 specific to the DSA clusters (namely that the last node is an empty
 spare). so it fails with:

 {{{
 fsn-node-03.torproject.org not empty.
 }}}

 apparently, the latest version of the script might fix that with the
 `crossmigratemany` function:

 https://salsa.debian.org/dsa-team/mirror/dsa-
 puppet/raw/master/modules/ganeti2/files/ganeti-reboot-cluster

 for now, i'll just do the reboot by hand.

 in theory, rebooting a ganeti node is to:

  1. migrate all the primaries off of the node: `ssh $master gnt-migrate
 $node`
  2. if it's a master, promote another master: `ssh $notmaster gnt-cluster
 master-failover` (optional, only if we can't afford having the master down
 during the reboot)
  3. reboot the node `ssh $node reboot`

 ... for each node.

 i'm testing that procedure on fsn-node-03 now.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33406#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online