Hardware Failures in 2022 - Post yours!

[Patrick]

Another year, another thread. Links to the long-lived 2017, 2020, and 2021 threads. Feel free to post your biggest hardware failures of 2022 in this thread.

Kicking this one off, two Seagate Exos X12 12TB drives that failed at the same time out of eight in the NAS.


We will have another one soon, and will also be putting a few of these on the main site. Someone pointed out that we do not discuss hardware failures enough.

 

[i386]

Kicking this one off, two Seagate Exos X12 12TB drives that failed at the same time out of eight in the NAS.

Did you check if they had firmware updates available?
"Publicly available" firmware updates are one of the reasons I like the exos hdds

 

[Patrick]

Did you check if they had firmware updates available?
"Publicly available" firmware updates are one of the reasons I like the exos hdds

Sadly, not that kind of failure.

 

[T_Minus]

It's 2022, will we see the "you should use different brands to be safe" replies

Look forward to learning more about those two failures.

 

[Y0s]

Do enterprise printers count? Neither of our two HP DesignJet Z6200 42inch poster printers survived return-to-the office/restart of conferences, both exploding in literal red ink! I guess don't leave inkjets idle?

 

[Malvineous]

Do old servers count? I had a Dell R720 power supply fail last week. There was a 'pop', followed a minute later by the smell of burning plastic, and then another minute later by smoke coming out the back.

Unfortunately the iDRAC on the machine had long since crashed and I hadn't had a chance to power cycle it, so I wasn't able to check to see whether the PSU actually failed or not. But nothing switched off or showed warnings so I couldn't work out what had broken. I was about to go through the DRACs one by one in each server to check, but then smoke started wafting out the back of the rack so since I didn't want to take any chances, I cut the power (and the UPS power) to get everything switched off quickly. It's a small rack and against a wall under a desk so it's not the easiest to get access to, so I needed time to pull it out, which I didn't feel I had with smoke suddenly appearing!

Eventually I found the culprit PSU and replaced it and the server is working fine again. In the photo of the failed PSU, the bubbly grey part touching the green bit (in the middle of the photo) isn't meant to be all bubbly like that. I haven't had a chance to properly disassemble it yet, but it looks like it may be a filter capacitor that shorted internally, but I can't be certain until I take a proper look.

 

[CPowerMav]

Do enterprise printers count? Neither of our two HP DesignJet Z6200 42inch poster printers survived return-to-the office/restart of conferences, both exploding in literal red ink! I guess don't leave inkjets idle?

Ah. The old bleeding printer.

 

[Patrick]

100% old servers and printers count. @Y0s and @Malvineous

 

[i386]

It's 2022, will we see the "you should use different brands to be safe" replies

In 2022 this means you can look only for toshiba or wd hdds as an alternative to seagate :/

 

[Jason Antes]

Had an old 2TB SAS drive in my backup system die at power on (it's only on for a week once a month to do D2D2T backups) and just had a P420 cache battery fail on my ESXi server. HDD replaced from my spare inventory and a new battery ordered.

 

[NablaSquaredG]

IDK if that counts...

http://imgur.com/a/b1AiVtA

Supermicro 8048B-TRFT (Quad Socket Xeon E7)

Issue:
One of the memory riser has factory installed fan cables nearby.
It may happen that the fan cable wraps and thus the RAM slot gets caught in it.

Because you always need a lot of force to insert those memory anyway (Supermicro went with the reference design which is a bazillion pin edge connector), you don't realise that you just yanked the RAM Slot.

Really bad design, like the rest of the server too...

 

[Andrewpaulb]

Well we had 2x ST16000NM002G 16TB Exos Enterprise SAS fail after each other from 14 drives. At least we had a RAID 6 in place...

 

[Railgun]

Did you check if they had firmware updates available?
"Publicly available" firmware updates are one of the reasons I like the exos hdds

They shipped FW of et03 was replaced by et04. That simply masked the issue. There was a self-admitted manufacturing issue from Seagate around these drives. I replaced two sets of 36…twice. The second set, while more behaved in the boxes they were in, were failing under the hood. Easily the worst disks I’ve ever had the displeasure of dealing with.

The “new” replacements are now derated 14TB (to 12TB). I’m not holding my breath on these either.

 

[rune-san]

I just had my last OCZ Vertex 4 SSD fail, after spending the last 10 years as various Server's Read / Write Caches for batch jobs. That little 120GB SSD had over 800TB of writes to it in the end, way WAY more than it was rated for. Overall these OCZ SSDs weren't anything great, but while the previous 6 or so died around 3 years ago, these last 2, failing about 2 months apart, gave way more than what the specifications rated them for, and I can definitely say I got my money's worth.

 

[RageBone]

My 550W EVGA power-supply went out with a loud Bang and tripped the breaker.
Currently in the Process of RMAing it.

 

[NablaSquaredG]

My 550W EVGA power-supply went out with a loud Bang and tripped the breaker.
Currently in the Process of RMAing it.

Are you sure you didn't accidently buy a Gigabyte PSU?

 

[Terry Kennedy]

Do enterprise printers count? Neither of our two HP DesignJet Z6200 42inch poster printers survived return-to-the office/restart of conferences, both exploding in literal red ink! I guess don't leave inkjets idle?

Large format printers need regular exercise. My experience is with Epson - I have a SC-P10000 and a P6000 here at home. I run a printless nozzle check daily in the dry season and print a full nozzle check around once a week. The P10K has 8000 print nozzles on a $2500 printhead, so you really don't want it to develop clogs.

As far as leaks go, that ink is nearly impossible to get out once it gets on something. An older 9500 series leaked ink on me and my hands looked like a Smurf for several weeks. The P10K holds about 2 gallons of ink - 700 ml in each of 10 cartridges plus all of the ink in the lines and head (it takes about 100 ml of each color to do the initial charge of the ink system).

 

[Y0s]

Large format printers need regular exercise...As far as leaks go, that ink is nearly impossible to get out once it gets on something.

Yeah, lesson learnt, though it's looking like we're just going to write off both expensive printers and use a printing service instead. The staff are asking for cloth posters for ease of transport.

 

[Silly Valley Serf]

First-time poster, long-time lurker.

I had a NAS running XigmaNAS in an old PC tower case, running on a SuperMicro C3758 motherboard. One day this spring, the main CPU just quit working. The IPMI was still responsive, but nothing I tried could get the main CPU to work again. And of course the mobo was long out of warranty.

I had obtained a used SuperMicro SC836 to move everything into prior to the failure, so I bought a new SM Xeon D-2123 motherboard to go with it. The RAM and the SSD from the old NAS dropped right in, and it booted right away. I put the drives from the old NAS chassis in, and the ZFS pools came right up too! I ran scrubs on them and was relieved to find no errors.

The only problem with this new server is that it's too noisy for my office, and I don't have a rack, or an Ethernet drop, in the garage yet.

 

[Fritz]

Just had a Supermicro outer rail fail on me and had to catch the server on it's way to the floor. Seems the BB's all fell out leaving nothing to hold in the inner part that the inner rail slides into. I have no idea what caused this to happen. I used a HD magnet to pick up all the little balls from the carpet but couldn't find at least half of them. I'm wondering if maybe this rail was a ticking time bomb that decided to go boom today. I examined the other rail and all the BB's are intact.