[tor-bugs] #34185 [Internal Services/Tor Sysadmin Team]: ganeti clusters don't like automatic upgrades

[Tor Bug Tracker & Wiki]

#34185: ganeti clusters don't like automatic upgrades
-------------------------------------------------+-------------------------
     Reporter:  anarcat                          |      Owner:  hiro
         Type:  defect                           |     Status:  assigned
     Priority:  High                             |  Milestone:
    Component:  Internal Services/Tor Sysadmin   |    Version:
  Team                                           |   Keywords:  tpa-
     Severity:  Major                            |  roadmap-may
Actual Points:                                   |  Parent ID:
       Points:                                   |   Reviewer:
      Sponsor:                                   |
-------------------------------------------------+-------------------------
 this weekend the ganeti cluster had a partial outage: nodes were
 reachable, but networking was broken on all instances.

 this is, presumably, because of the Debian buster point release that
 occured on saturday (!). last time this happened, weasel identified
 openvswitch as the culprit, and hiro deployed a fix that would make it
 survive such situations. but either something else came up or the fix
 didn't work, because the problem happened again this weekend.

 i fixed it by rebooting all nodes forcibly (without migrating first).

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/34185>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online