I've been upgrading my zfs raidz2 setup from 3TB drives to 6TB drives on the cheap via ancient surplus drives on eBay and it's mostly gone ok in that only two of the ten drives I got were immediately bad. However after getting my new pool up and running I've run into another drive that starts throwing errors when it's under heavy load (in this case a zpool scrub!).
(da5:mps0:0:9:0): READ(10). CDB: 28 00 03 62 70 66 00 00 08 00
(da5:mps0:0:9:0): CAM status: SCSI Status Error
(da5:mps0:0:9:0): SCSI status: Check Condition
(da5:mps0:0:9:0): SCSI sense: HARDWARE FAILURE asc:44,0 (Internal target failure)
(da5:mps0:0:9:0): Info: 0x362706c
(da5:mps0:0:9:0): Command Specific Info: 0
(da5:mps0:0:9:0): Actual Retry Count: 1
(da5:mps0:0:9:0): Descriptor 0x80: f7 c3
(da5:mps0:0:9:0): Descriptor 0x81: 00 28 61 01 00 9e
(da5:mps0:0:9:0): Error 5, Unretryable error
The last time I mucked around with things at the SCSI protocol level was back with SCSI-2 and big fat 50 pin ribbon cables (which is to say that it was a while ago), but this looks like a pretty general `something went wrong' on the drive error. My reasoning is that it's got to be on the controller side of things since if there was a read or write error from the platters the controller would be sending back a specific error relating to that. The drive did make it through a full `reinitialize disk surface' pass from Hard Disk Sentinel, but now has failures under heavy read/write load I'm assuming that the platters are fine but that the drive controller board is flaky.
Obviously the simplest thing to do would just be to get a replacement drive and chuck this one, but since I've got some other drives where just the platters are bad, I thought that perhaps I could frankenstein a working drive using the good parts from both? At this point it's mostly out of curiosity
Anyway, I swapped the controllers and of course it turns out it's not so simple. =) The drive spun up and made a few seeking sounds, but shortly afterwards I was greeted with this message on the console:
(da5:mps0:0:9:0): Attempt to query device size failed: NOT READY, Logical unit not ready, initializing command required
This is where my knowledge of hard drives falls down a bit. I can only guess that there's some information stored on the platters that the controller is looking for and of course since I just swapped it out it can't find what it's expecting. But it would seem that there is a way to send an `initializing command' to the drive to fix that? Since the `bad' drive still works fine most of the time my hope is that I can dump the information that I need from it and then send it after the swap?
Or maybe it's simpler to just swap the data on the controller itself? I've found a couple `drive recovery' videos where they're trying to get data off a drive with a failed controller and they end up swapping a flash chip from the old one to the new one.
(213) How to Replace a Hard Drive PCB for Data Recovery - YouTube
(213) Hard Drive Bios Chip Swap - Replace HDD Motherboard For Data Recovery - YouTube
There is something that looks like a small SPI flash chip on the board, so that's probably the simplest way to go about it. It would seem that the simplest thing would be to just reprogram the flash chip, but maybe it's protected? I need to find my programmer and try and dump it to see, since while the soldering work doesn't seem too bad, it would be much easier to just flash the chip. =) I can only guess that it either can't be read or written since people are swapping the chips (which likely would mean that any 'initializing commands' would also be locked out), but maybe they just didn't try?
If anyone has dealt with this sort of thing before I'd love to know!
Actually, does anyone have a dump of the ADD5 firmware for the HGST HUS726060AL5210 laying around? That seems to be the latest version, but I haven't been able to find it anywhere since all of the old links I've found are expired. I was able to find AD05 and that's probaly fine -- it's not like I've even been able to find the release notes for the various firmwares and what they fix (or don't), but as long as I'm setting up the array I might as well get it as up to date as possible. =)
(da5:mps0:0:9:0): READ(10). CDB: 28 00 03 62 70 66 00 00 08 00
(da5:mps0:0:9:0): CAM status: SCSI Status Error
(da5:mps0:0:9:0): SCSI status: Check Condition
(da5:mps0:0:9:0): SCSI sense: HARDWARE FAILURE asc:44,0 (Internal target failure)
(da5:mps0:0:9:0): Info: 0x362706c
(da5:mps0:0:9:0): Command Specific Info: 0
(da5:mps0:0:9:0): Actual Retry Count: 1
(da5:mps0:0:9:0): Descriptor 0x80: f7 c3
(da5:mps0:0:9:0): Descriptor 0x81: 00 28 61 01 00 9e
(da5:mps0:0:9:0): Error 5, Unretryable error
The last time I mucked around with things at the SCSI protocol level was back with SCSI-2 and big fat 50 pin ribbon cables (which is to say that it was a while ago), but this looks like a pretty general `something went wrong' on the drive error. My reasoning is that it's got to be on the controller side of things since if there was a read or write error from the platters the controller would be sending back a specific error relating to that. The drive did make it through a full `reinitialize disk surface' pass from Hard Disk Sentinel, but now has failures under heavy read/write load I'm assuming that the platters are fine but that the drive controller board is flaky.
Obviously the simplest thing to do would just be to get a replacement drive and chuck this one, but since I've got some other drives where just the platters are bad, I thought that perhaps I could frankenstein a working drive using the good parts from both? At this point it's mostly out of curiosity
Anyway, I swapped the controllers and of course it turns out it's not so simple. =) The drive spun up and made a few seeking sounds, but shortly afterwards I was greeted with this message on the console:
(da5:mps0:0:9:0): Attempt to query device size failed: NOT READY, Logical unit not ready, initializing command required
This is where my knowledge of hard drives falls down a bit. I can only guess that there's some information stored on the platters that the controller is looking for and of course since I just swapped it out it can't find what it's expecting. But it would seem that there is a way to send an `initializing command' to the drive to fix that? Since the `bad' drive still works fine most of the time my hope is that I can dump the information that I need from it and then send it after the swap?
Or maybe it's simpler to just swap the data on the controller itself? I've found a couple `drive recovery' videos where they're trying to get data off a drive with a failed controller and they end up swapping a flash chip from the old one to the new one.
(213) How to Replace a Hard Drive PCB for Data Recovery - YouTube
(213) Hard Drive Bios Chip Swap - Replace HDD Motherboard For Data Recovery - YouTube
There is something that looks like a small SPI flash chip on the board, so that's probably the simplest way to go about it. It would seem that the simplest thing would be to just reprogram the flash chip, but maybe it's protected? I need to find my programmer and try and dump it to see, since while the soldering work doesn't seem too bad, it would be much easier to just flash the chip. =) I can only guess that it either can't be read or written since people are swapping the chips (which likely would mean that any 'initializing commands' would also be locked out), but maybe they just didn't try?
If anyone has dealt with this sort of thing before I'd love to know!
Actually, does anyone have a dump of the ADD5 firmware for the HGST HUS726060AL5210 laying around? That seems to be the latest version, but I haven't been able to find it anywhere since all of the old links I've found are expired. I was able to find AD05 and that's probaly fine -- it's not like I've even been able to find the release notes for the various firmwares and what they fix (or don't), but as long as I'm setting up the array I might as well get it as up to date as possible. =)