[tor-bugs] #33098 [Internal Services/Tor Sysadmin Team]: fsn-node-03 disk problems

[Tor Bug Tracker & Wiki]

#33098: fsn-node-03 disk problems
-------------------------------------------------+-------------------------
 Reporter:  anarcat                              |          Owner:  anarcat
     Type:  defect                               |         Status:
                                                 |  assigned
 Priority:  High                                 |      Milestone:
Component:  Internal Services/Tor Sysadmin Team  |        Version:
 Severity:  Blocker                              |     Resolution:
 Keywords:                                       |  Actual Points:
Parent ID:                                       |         Points:
 Reviewer:                                       |        Sponsor:
-------------------------------------------------+-------------------------

Comment (by anarcat):

 they tested the drive (probably a smart short self-test) and didn't find
 anything. they swapped the cable and gave us back the box.

 after boot, the raid array *again* did not come up. i restarted it and re-
 added the extra drive. during the sync, we got a ATA error again, but this
 time it seems the drives didn't notice because smartd didn't send us
 email. i only noticed it through a dmesg tail:

 {{{
 [Jan29 20:00] microcode: microcode updated early to revision 0xca, date =
 2019-10-03
 [  +0.000000] Linux version 4.19.0-6-amd64 (debian-
 kernel at lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP
 Debian 4.19.67-2+deb10u2 (2019-11-11)
 [...]
 [Jan29 20:18] md/raid1:md2: active with 1 out of 2 mirrors
 [  +0.050102] md2: detected capacity change from 0 to 10000693985280
 [Jan29 20:19] md: recovery of RAID array md2
 [...]
 [Jan29 22:33] ata1.00: exception Emask 0x50 SAct 0xe000 SErr 0x480900
 action 0x6 frozen
 [  +0.000014] ata1.00: irq_stat 0x08000000, interface fatal error
 [  +0.000007] ata1: SError: { UnrecovData HostInt 10B8B Handshk }
 [  +0.000009] ata1.00: failed command: WRITE FPDMA QUEUED
 [  +0.000013] ata1.00: cmd 61/00:68:80:0c:c4/08:00:b8:00:00/40 tag 13 ncq
 dma 1048576 ou
                        res 40/00:68:80:0c:c4/00:00:b8:00:00/40 Emask 0x50
 (ATA bus error)
 [  +0.000011] ata1.00: status: { DRDY }
 [  +0.000005] ata1.00: failed command: WRITE FPDMA QUEUED
 [  +0.000012] ata1.00: cmd 61/00:70:80:14:c4/03:00:b8:00:00/40 tag 14 ncq
 dma 393216 out
                        res 40/00:68:80:0c:c4/00:00:b8:00:00/40 Emask 0x50
 (ATA bus error)
 [  +0.000011] ata1.00: status: { DRDY }
 [  +0.000006] ata1.00: failed command: WRITE FPDMA QUEUED
 [  +0.000011] ata1.00: cmd 61/80:78:80:17:c4/04:00:b8:00:00/40 tag 15 ncq
 dma 589824 out
                        res 40/00:68:80:0c:c4/00:00:b8:00:00/40 Emask 0x50
 (ATA bus error)
 [  +0.000011] ata1.00: status: { DRDY }
 [  +0.000008] ata1: hard resetting link
 [  +0.375942] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
 [  +0.001613] ata1.00: configured for UDMA/133
 [  +0.000026] ata1: EH complete
 }}}

--
Ticket URL: <https://trac.torproject.org/projects/tor/ticket/33098#comment:1>
Tor Bug Tracker & Wiki <https://trac.torproject.org/>
The Tor Project: anonymity online