[tor-bugs] #29817 [- Select a component]: dead disk on moly

[Tor Bug Tracker & Wiki]

#29817: dead disk on moly
--------------------------------------+----------------------
     Reporter:  anarcat               |      Owner:  anarcat
         Type:  defect                |     Status:  assigned
     Priority:  Medium                |  Milestone:
    Component:  - Select a component  |    Version:
     Severity:  Normal                |   Keywords:
Actual Points:                        |  Parent ID:
       Points:                        |   Reviewer:
      Sponsor:                        |
--------------------------------------+----------------------
 one of the hard drives on moly has died. this was spotted by cymru's staff
 and confirmed when smartd was installed (#29709).

 i have done some research on the machine to figure out what's up, and
 wrote the following reply to Cymru's people:

 > [...] I can confirm that one of the hard drives in Moly has failed,
 according to SMART metrics we have available.
 >
 > According to smartd, that disk is:
 >
 > [SEAGATE  ST3600057SS      0008], lu id: 0x5000c5003b5bc36f, S/N:
 6SL1G7Q60000N1497K0E, 600 GB
 >
 > It's a 600GB SAS drive. It's part of a megaraid RAID-10 array that has
 marked the drive as "Firmware state: Failed". I'll go under the assertiont
 his means the drive is dead.
 >
 > Being new here, I'm not familiar with the machine either. From what I
 can tell, it's a Supermicro X8DTU motherboard, and possibly an iXsystems
 iX1204-R700UB case. Does it look like this this picture?
 >
 > https://static.ixsystems.co/uploads/2017/08/1204h-t_front.png
 >
 > If so, the only datasheet I could find is this limited PDF:
 >
 > https://www.ixsystems.com/wp-
 content/uploads/2017/09/Server_Line_2017_WEB.pdf
 >
 > It *does* say the hard drives are hot-swappable, so in theory, it should
 just be a matter of replacing the hard drive.
 >
 > It looks like each drive has its own LED, hopefully the one with the
 amber warning light should be the dead disk. I've issued a command to the
 RAID controller to make it "flash" the drive LED, so hopefully that will
 allow you to locate it better.
 >
 > I *think* the disk controller is new enough for you to simply hot swap
 the drive with a new one without any other intervention on our part. But
 it might be better if we are available during the operation. [...]

 I've created some documentation on the hardware RAID stuff here:

 https://help.torproject.org/tsa/howto/raid/

 we're at the waiting step now - we'll see if Cymru can do the replacement
 and when. i'm still not quite certain we can just hotswap the drive, but
 I'm hoping we can.

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29817>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online