Does no one update firmware?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Propaganda

Active Member
Dec 6, 2017
154
62
28
43
I have been getting various drives on EBay over the last couple months and found that nearly none of them came with up to date firmware. Does no one that runs these "enterprise" drives update them in the field? There are some pretty serious firmware bugs out in the wild and I don't understand how they would risk the server they spent 20k on just not working randomly because they couldn't spend 10 minutes downtime in 5 years to update firmware. I do understand the thought line of "if it ain't broke don't fix it" but the firmware issues are pretty well known and catastrophic in some cases so they would warrant running in my mind.
 

mattventura

Active Member
Nov 9, 2022
447
217
43
There's arguments to be made both ways. You have to weigh the risk of a botched update bricking a drive versus the risk of a nasty undiscovered firmware bug. If you get one of the really bad ones, like the 32768 hour SSD failure, it gets well-publicized and you have time to apply the update. There is also a risk that the new firmware introduces a new bug anyway, though I doubt that's terribly common. Point is, it's not objectively less of a risk to keep firmware up to date, especially if you're upgrade quite frequently.

Manufacturers tend to lean towards the "if it ain't broke, don't fix it" mentality. For example, from Seagate's site (I couldn't find an equivalent for enterprise drives):

Firmware updates
Performing a firmware update can be risky by itself even in the best of situations. For example, a power outage during a firmware update carries a very good possibility of ruining the drive. External, Serial ATA, and ATA drives are not designed for field firmware updates by end users.
Or from Microsoft:

Firmware updates are a potentially risky maintenance operation and you should only apply them after thorough testing of the new firmware image. It is possible that new firmware on unsupported hardware could negatively affect reliability and stability, or even cause data loss. Administrators should read the release notes a given update comes with to determine its impact and applicability.
However, some of them recommend updating, like Dell:

Having the latest firmware can improve the performance and or reliability of your product. If newer firmware is available for a drive, it will reflect the improvements engineered for the latest manufacturing.
So....pick your poison. Botched upgrade vs some unforseen firmware bug.
 

tinfoil3d

QSFP28
May 11, 2020
880
404
63
Japan
Most people in enterprise are propaganda-proof
On a serious note, i also share this view of "if it works, don't touch it". There's also a point in using SSDs from different manufacturing lots, not a single one.
What bugs are you referring to in particular, for example, 32k hours aside?
 
  • Haha
Reactions: zac1

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,142
594
113
New York City
www.glaver.org
Manufacturers tend to lean towards the "if it ain't broke, don't fix it" mentality. For example, from Seagate's site (I couldn't find an equivalent for enterprise drives):
End-user retail drives are a very small share of total drives shipped by a manufacturer. Most go into either OEM builds (Dell , HP, Lenovo, etc.) or directly to high-end integrator/user builds (AWS, Facebook, etc.)

Every OEM / integrator wants some mods to the firmware. "Clipping" the capacity so a 146GB drive from Seagate, Toshiba, ED, HGST, etc. all report the same number of blocks, making them interchangeable while the retail ones aren't. Next is support for various sector sizes / T10 protection. Then there's workaround for issues in vendor controllers. Lots of retail drives that could do 6Gbit SAS showed up surplus from Dell systems, and the Dell ones had the speed limited to 3Gbit/sec by the firmware. Lastly may be "labeling", both on the physical drive and as reported on an Inquiry command. The OEMs want you to believe that this other firmware is due to their "exhaustive testing" when in fact it is a branding exercise - after all, if some OEM finds a catastrophic firmware bug and reports it to the manufacturer, the manufacturer will usually want to fix it in the generic firmware although unless it is really bad, they won't make retail firmware downloads available.

Years ago I was getting drives by the palletload either direct from manufacturers or from top-tier distributors. Back in the days of parallel SCSI drives, the magic words to get retail firmware updates was "spindle sync" because that required totally identical drives all communicating with each other for top-tier storage array performance. For a couple years I had a dedicated Seagate support rep who would come around once a month, hand out new firmware and talk about possible future drive developments. With WD I had a support rep but he wasn't a field rep, so I'd have to ask for firmware. Granted, I opened that line of discussion with "Hi! I need to RMA five hundred-plus drives with advance replacement, please".

A couple other tidbits - several manufacturers have told me the profits on retail drives are so thin that if they get an RMA request for a drive, even if they get it back and it is a "no problem found", they overall lost money on that drive. That's why many manufacturers have utilities they want you to run to find out what kind of a failure (if any) your drive has before requesting an RMA. As far as OEM drives go, the purchase agreement usually specifies things like the warranty period (usually pretty short compared to retail drives) and any restrictions on RMA-ing (like must be in multiples of X drives, serial numbers bar-coded on the external shipping carton, and shipping at the customers cost to a specified site of the vendor's choosing). That site may be out of the US - believe me, palletizing drives to ship to Thailand is no fun. In exchange, the OEMs get very large discounts.
 
  • Like
Reactions: tinfoil3d

CyklonDX

Well-Known Member
Nov 8, 2022
847
279
63
so my take,

If there are no issues don't ever update firmware. Don't fix what isn't broken.

The case for some lsi raid controllers, the updated firmware actually reduces performance to keep the chips on the card last longer, overheat less. *(But if you can cool it down, there's no point to this)

Same goes to any other part. The only reason to upgrade firmware is to fix something.
 

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
I'm always updating firmware, bios etc to the newest version.
Firmware for me is not just "fixes" but also about improving performance and stability or adding (and sometimes removing) features/options.

(So far only the first spectre/meltdown bios updates in 2018 were problematic but were resolved with later updates :D)
 

CyklonDX

Well-Known Member
Nov 8, 2022
847
279
63
like decreased SQL performance? That was hit in a gut. (also pointless to apply systems not running inside VM in shared vmhosts like AWS or local shared enviroment where systems are exposed) (one day your haswell v4's began performing same as worse clocked sandy v1's)
 

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
like decreased SQL performance? That was hit in a gut. (also pointless to apply systems not running inside VM in shared vmhosts like AWS or local shared enviroment where systems are exposed) (one day your haswell v4's began performing same as worse clocked sandy v1's)
I didn't benchmark/measure databases or applications.
One thing were I noticed decreased performance on a e5 v4 was with my optane 900 ssd: ~450k iops vs ~540k iops in benchmarks :D
 
  • Love
Reactions: T_Minus

CyklonDX

Well-Known Member
Nov 8, 2022
847
279
63
same went with lucene indices 20-30% off in performance overnight. Worst updates ever, should be panel with option in system to disable them without tricks in msconfig. (it still affects cpu's even newest ones by few %)
 

Propaganda

Active Member
Dec 6, 2017
154
62
28
43

unwind-protect

Active Member
Mar 7, 2016
417
156
43
Boston
Performing a firmware update can be risky by itself even in the best of situations. For example, a power outage during a firmware update carries a very good possibility of ruining the drive. External, Serial ATA, and ATA drives are not designed for field firmware updates by end users.
Yeah, but I would do the update when the drive is not in use and hence not loaded with data I wouldn't like to recover elsewise.

My primary reason not to bother with drive firmware updates is the shitty release notes / changelogs. They should be more clear about what changed. If I can't trust the text they publish I don't trust the binary.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,641
2,058
113
I update then deploy, then forget about it unless it's catastrophic type issue. Seems to be common.
 
  • Like
Reactions: BackupProphet