[tor-bugs] #29817 [Internal Services/Tor Sysadmin Team]: dead disk on moly

[Tor Bug Tracker & Wiki]

#29817: dead disk on moly
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  defect                               |         Status:  closed
 Priority:  Medium                               |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Normal                               |     Resolution:  fixed
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------
Changes (by anarcat):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 the disk ended up being replaced on wednesday by cymru's remote hands, but
 unfortunately the machine didn't come back up after reboot.

 we weren't able to access the BIOS with the IPMI interface, neither
 through ipmi-console which i installed on peninsulare and should provide a
 SOL (Serial Over LAN) interface, nor through IPMIview which weasel was
 able to run in an older jessie (or wheezy?) VM. it seems there is
 configuration missing in the BIOS to redirect it to the serial console and
 a similar configuration is therefore likely missing from GRUB and the
 kernel as well.

 in the end our precious remote hands were able to reboot the machine by
 simply hitting the keyboard. it seems the BIOS was configured to hang on
 boot if a new drive was inserted in the machine, another BIOS
 configuration which might be nice to disable, to say the least.

 lessons learned:

  1. BIOS configuration should be standardized to a certain set of
 parameters to avoid those problems in the future (boot without
 interruption, console redirection, etc)
  2. GRUB also needs a similar configuration
  3. we should test serial consoles before rebooting. in this case, the
 machine might have been able to have the drive swapped without a reboot if
 we were worried about the reboot
  4. this machine could possibly be retired as it is getting to its 8th
 year anniversary

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29817#comment:5>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online