Hitachi 5K3000 dropped from array

[lifespeed]

Just had the unpleasant experience of finding a drive dropped from my RAID6 array with a Highpoint 3520 card. First time this has happened in almost 4 years of large-scale storage. My previous array of 5X Seagate ST31000340NS 1 TB never dropped a drive from the array, although a couple of them did accumulate a few bad sectors (I RMA'd these drives).

The 3 TB Hitachi 5K3000 (0S03230 HDS5C3030ALA630) had been in use for only 8 months, #4 in a RAID6 array with 6 drives. I was going to remove and re-seat the drive in the Norco 4220 chassis, so turned off the PC and pulled the plug on the power supply. Unfortunately I had forget the BIOS is set to power on after a power failure, so this resulted in the start of an automatic rebuild. At this point I intend to just let it finish, I'll report back with the results.

Anybody else ever have one of these drives drop out of an array? What card was used?

 

[mobilenvidia]

The 3520 is going to take sometime to rebuild the array.
Is it rebuilding the array on 5 drives or back to 6 (after the re-seat) ?

 

[lifespeed]

It will be done in a couple more hours, and it appears to be rebuilding it back to a 6-drive RAID6. No problems so far, and I have ripped a couple movies and continued recording TV during the rebuild.

I am definitely bothered the drive "failed" in the first place. And I did not ever get the chance to reseat it, although I may still do so when it is done.

 

[mobilenvidia]

TLER issues with these drives possibly ?

 

[lifespeed]

TLER issues with these drives possibly ?

That is the million dollar question! According to people on this board they are fine, but I've never seen documentation from the manufacturer stating that . . .

 

[mobilenvidia]

If a drives drops then happily rejoins after a power cycle, ofcourse the rebuild is not nice to have to wait for.
This is a symptom of TLER

It may have dropped a Sector on the HDD

Does the HP3520 do Logs ?
LSI keeps logs of these things in NVRAM as they happen

 

[nitrobass24]

Have a hard time believing its a TLER type of issue, just because i have been running these for a long time and have not had that happen...but i guess anything is possible.

 

[lifespeed]

There was another guy on this board whose name escapes me, but he claimed that occasional dropouts of HDD from the array were normal. And he was using the newer Hitachi drives.

Of course, I do not agree that a drive dropping from an array is normal, especially as I have had several years without that ever happening on another set of drives.

I think I will try to get the updated firmware (newer than 580) and keep an eye on this drive. If it drops again I'll RMA it.

No, there are no bad sectors on it now. And a bad sector should not cause it to drop. My Seagate enterprise drives silently accumulated a few bad sectors over the years, the controller is supposed to handle this seamlessly - just mark them and continue. And it does.

I did pull the drive and inspect the connector. Nothing visible, of course. It appeared to be seated in the connector well. The Seagate that used to be in the same tray never had an issue.

 

[Patrick]

Actually, bad sectors are a big cause of the whole TLER thing. Basically what is happening there is that the drive sees a bad sector, then puts the head over that sector over and over to try a "heroic recovery" of the data in the sector. That process if you have one drive is not a big deal, but you may notice the system stall and wait for a drive to do a read/ write while it is occupied with that process. With a RAID card you end up having a drive attempt the recovery for a few seconds, and the controller thinks that the drive has failed as it is unresponsive while doing the heroic recovery.

 

[lifespeed]

Actually, bad sectors are a big cause of the whole TLER thing.

Agreed. I meant that bad sectors are a non-event for a properly functioning RAID array, which would include disks that don't go into a deep recovery cycle.

 

[lifespeed]

10 days later and no indications of problems, no bad sectors. Temps are in the low 40C range. A disturbing event, but not much I can do other than keep an eye on it. Wouldn't mind trying new firmware if it ever become available.

 

[Patrick]

That is good to hear lifespeed. I would still watch the system, especially if you have not had it under considerable load in the past 10 days.

 

[jjwa]

I was curious to hear how the array has been behaving lifespeed?
I am considering buying the 5K3000 (3TB version, the 2TB version's price is now too high compared to other 2TB drives), because I thought it to be one of the very few 2TB or 3TB consumer drives without TLER issues.

 

[lifespeed]

The array has been performing flawlessly 24/7 (although it is configured to spin down when idle). I could not say why it dropped 1 out of the 6 drives 5 months ago, but it has been fine ever since. I have run a couple array verifications without incident as well.

These may well be our only drive option for RAID arrays at reasonable prices. I am down to 3 TB free space, so will probably buy another when prices come back down to earth some time in the distant future.

 

[Patrick]

Great update. Been very happy with my Hitachi drives (even the 1TB and 2TB ones.)