Micron 5200 firmware bug

[EffrafaxOfWug]

I think that's basically going to be down to how different from the official firmware the HPE-specific models are; given that on the face of it this looks like it's only going wrong on a 16bit overflow on a certain SMART attr, and that HPE and other OEMs often have custom SMART attrs and counters, it's not out of the realms of possibility that the HPE drives are unaffected.

Certainly I'd have expected a fix or at least an advisory from HPE by now if this problem started rearing its head in mid-2025.

 

[wallacebrf]

Given the potential severity of the issue - mostly RAID arrays becoming toast - along with Micron's apparently inability to admit to regular users that there is a firmware fix available for this, I did wonder if it was worth pinging Patrick about this in terms of raising awareness...?

Micron might no longer care about individual users, but clearly cared enough about Hetzner to provide them with the files (albeit under NDA), and I suspect there's a large contingent of StHers and SMEs out there using 5200's that haven't had the good fortune to run in to this thread or the Reddit post (or the bad fortune to hit their 65k hours) yet.

I'm still clinging on to the forlorn hope that Micron's still just doing QA on this, but the apparent fact they're still telling users opening support tickets about this issue that the drives are obsolete and unsupported - despite the fixed firmware already existing for at least a month, and possibly since January - is leaving a very sour taste in the mouth, especially since their firmware support's generally been very good from a homelabber's perspective in the past.

I have debated about responding to my support tickets with something along the lines of

Hey, guess what, that firmware you said does not exist, well here it is,how about that?

But I do not want to make this too public to micron in case they either get the OP in trouble somehow or make the files disappear from the Internet....

 

[TrevorH]

if this problem started rearing its head in mid-2025.

I should clarify my mid-2025 date. I don't know if that's when it was first discovered but that is around the time that the first shipped drives would have experienced the problem if they'd been on 24/7.

 

[EffrafaxOfWug]

Mid-2025 sounds about right - my oldest is from october 2025, and it was usually a good few of months after release before they'd filter in to the UK retail channels (usually much longer these days), and I usually don't buy anything until it's been out for at least a couple of months.

The press release and the first review were both in January 2018.

65535 hours is just under 7.5 years so assuming it wasn't caught beforehand mid-2025 is the very earliest you could have expected to see this in the wild.

 

[wallacebrf]

Mid-2025 sounds about right - my oldest is from october 2025, and it was usually a good few of months after release before they'd filter in to the UK retail channels (usually much longer these days), and I usually don't buy anything until it's been out for at least a couple of months.

The press release and the first review were both in January 2018.

65535 hours is just under 7.5 years so assuming it wasn't caught beforehand mid-2025 is the very earliest you could have expected to see this in the wild.

I read this exact article and it was one of the reasons why I got the eco 1.92 TB drives to begin with

 

[EffrafaxOfWug]

Another post on the thread, this one using the Lenovo-branded FW (who it looks like haven't released an update yet either) said that normal performance was resumed with a re-flash of the current FW :shrugs:

Half-tempted to let one of my as-yet-unpatched drives roll over the 65k hours and see what happens, but there's probably ample time for more canaries in the coal mine to see if that works for anyone else.

 

[wallacebrf]

I have about 1 year left before my oldest drive hits the mark so I will not know until then

 

[Machhapuchhare]

I work for an educational institution, and we have over 20 Lenovo servers equipped with Micron 5200 SSDs.

We've noticed that reapplying the Lenovo MG41 firmware temporarily alleviates the issue, but only for a few days. After that, the problems return: long response times reported in Windows Resource Monitor, timeouts, and various errors flagged by the MegaRAID utility.

We used the MSECLI utility to collect debug data from the affected drives. The problem is that most of the output files are in BIN format and are not human-readable. We've tried hex editors and variety of converters without success, which leads me to suspect that proprietary software may be required to interpret these files.

I know some people have experimented with the Dell D1MU firmware family, but it appears to target a slightly different drive SKU than the ones we have. For example:

• 5200MMU02EHC0677 (D1MU040, D1MU440, D1MU540, D1MU840)
• 5200MLE0D4HC0668 (MG41 from Lenovo)

References to both of these SKUs can be found within the respective BIN files.

My questions are:

• Does anyone have access to a newer Lenovo firmware release, assuming one even exists?
• Does anyone know how to decode the debug files generated by MSECLI?

We've already contacted both Lenovo and Micron regarding this issue, but neither has been particularly helpful. At this point, the likelihood of Lenovo releasing updated firmware seems close to zero.

 

[EffrafaxOfWug]

My questions are:

• Does anyone have access to a newer Lenovo firmware release, assuming one even exists?

We've already contacted both Lenovo and Micron regarding this issue, but neither has been particularly helpful. At this point, the likelihood of Lenovo releasing updated firmware seems close to zero.

When I saw the post on the reddit thread, I did have a delve to see if there was any sign of a firmware update from Lenovo - not because I have any myself, I'm just curious how public Micron are making this.

In a curious coincidence, another post on the reddit thread today about HPE releasing an updated firmware for drives that sound like 5200s underneath, so these clearly have gone out to at least one OEM; specifically the following model numbers:

• VK000480GWSXF
• VK000960GWSXH
• VK001920GWSXK
• MK000480GWUGF
• MK000960GWUGH
• MK001920GWUGK

We're an HPE shop but from my inventory sheets at least we don't have any of these drives in the wild.

Addresses performance degradation issue for read/write workloads when drive exceeds 65535 hours (~7.5 years) of Power-On hours.

...so it might be worth prodding Lenovo to see if there's one in the pipeline.

Edit: In relation to your second bullet point, I've never tried using msecli's debug dump feature before but I've just given it a whirl on a random drive to see the sort of stuff you're looking at.

root@wug:~# msecli -P debug.zip -n /dev/sda
Collecting debug data........................................................................................................................................................
Zip file is created on the specified location
Device Name  : /dev/sda
Debug data has been retrieved successfully
CMD_STATUS   : Success
STATUS_CODE  : 0

Copyright (C) 2020 Micron Technology, Inc.

Juicy stuff looks to be sitting under <serial>/Controller in the resulting zip file, some of it's readable via strings...

root@wug:~/msecli_debug/<serial>/Controller# strings firmware_version.bin
Compiled:
Oct 24 2019  15:59:04
a654e79
User Capacity:
SVN Info:
$Rev: a654e79 $
$URL: ssh://git@bitbucket.micron.com/ssdfw/acadia.git {release/Dylan_AD0161_Release} $
Internal Ver:
5300MMU025HD0185
External Ver:
 D3MU801
Meta Info:
Rev: 0.0
BootLoader Ver:
$Rev: a654e79 $
$URL: ssh://git@bitbucket.micron.com/ssdfw/acadia.git {release/Dylan_AD0161_Release} $
NAND Info:
NAND Reticle:
0013
NAND Cell Rev:
NAND MPPR:
NAND MTPR:
NAND Prm CRC:
1B 6B 91 DD

...but I think things like the SMART data/logs (possible just in raw format? But don't think usual suspects like smartctl can read from files) and whatever the gplogs are definitely need some magic.

My gplog's appear to be mostly entirely empty in fact, mostly just long strings of 0's;

root@wug:~/msecli_debug/<serial>/Controller# xxd gplog_A6.bin |head
00000000: 2500 0000 75a0 0000 0000 0000 0000 0000  %...u...........
00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000  ................
root@wug:~/msecli_debug/<serial>/Controller# xxd gplog_A6.bin |tail
003e7f60: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7f70: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7f80: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7f90: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7fa0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7fb0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7fc0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7fd0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7fe0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
003e7ff0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

...so not a great deal to see here TBH. Suspect it's mostly a way of getting "honest" info out of drives from people like me who've been tempted to tell Micron my drive's definitely over 65k hours and has gone all teh slow.

Edit 2: Seems like those binary logs being mostly empty might be because of these lines in mseapiLogFile.txt. GPL logs = General Purpose Logs logs which are part of the ATA spec; potentially that didn't work because the system in question is lacking sg_sat_read_gplog.

Thu Jun 11 19:57:26 2026:WARNING: GetDebugData: Getting SMART Logs failed for Drive : /dev/sda
Thu Jun 11 19:59:56 2026:WARNING: GetDebugData: Getting GPL Logs for client drive failed for Drive : /dev/sda

 

[Machhapuchhare]

It appears that Lenovo released two firmware packages on June 4th and 5th for Windows and Linux, respectively. However, for some unknown reason, only the Linux package includes the MG42 firmware for Micron 5200 drives, which I only realised today.

Also, I haven't found anything in the release notes detailing what was fixed in this update. That said, given that it was released around the same time HP published its own update—and explicitly confirmed what many of us already suspected: "Addresses performance degradation issue for read/write workloads when drive exceeds 65,535 hours (~7.5 years) of power-on time."—there's a good chance this firmware addresses the same underlying issue.

Assuming that's the case, we should be in good shape.

 

[ketiljo]

HPE released a new version last week where they say " Addresses performance degradation issue for read/write workloads when drive exceeds 65535 hours (~7.5 years) of Power-On hours. "

https://support.hpe.com/connect/s/softwaredetails?language=en_US&collectionId=MTX-4cc24985bf9c4850&tab=revisionHistory&softwareId=MTX_ba6b193738c04fac9ce493b05d

Edit: Sorry, this is already posted above

 

[mjshark]

Apparently Micron 5100 is also affected by this issue, same changes are listed in the release notes for HPE OEM Micron 5100.

https://support.hpe.com/connect/s/softwaredetails?language=en_US&collectionId=MTX-8c0c47d5bf1446e8&tab=releaseNotes

Lenovo's package also includes updated firmware for their version of the Micron 5100.

https://datacentersupport.lenovo.com/us/en/products/servers/thinksystem/sr650/7x05/downloads/ds581601?category=Storage

 

[Stephan]

Cisco got hit too, there are updated firmwares behind paywall:

Micron 5100: D0MU077
Micron 5300: D3CN001

(There could be even newer versions but Micron homepage has nothing)

So basically, if you have a Micron 5x00 and firmware older than end of 2025 or 2026, this bug will affect you after 65535 hours.

 

[ca3y6]

The 5300 is still supported by Micron so I am watching their download page anxiously (though I have a few years before that's a problem)

 

[EffrafaxOfWug]

Cisco got hit too, there are updated firmwares behind paywall:

Micron 5100: D0MU077
Micron 5300: D3CN001

(There could be even newer versions but Micron homepage has nothing)

So basically, if you have a Micron 5x00 and firmware older than end of 2025 or 2026, this bug will affect you after 65535 hours.

Interesting; assuming you have access to the page in question do they provide any notes in the changelogs for the 5100 and 5300 calling out the 65k hour bug?

I don't have any 5100s myself but given they came out ~2016 I'd have expected a lot more people to have run in too bugs on those already. 5300s came out at the end of 2019 so failures for those probably won't start coming out of the woodwork until mid-2027.

I'd have to wonder if this also affects the 5400 series as well at this point... I have to admit I was rather confused as to why their download for the 5400 had one qualified as "for specific customers" with the older one flagged as "for general customers", is anyone aware of what their distinction is here?

 

[gregsachs]

Interesting; assuming you have access to the page in question do they provide any notes in the changelogs for the 5100 and 5300 calling out the 65k hour bug?

I don't have any 5100s myself but given they came out ~2016 I'd have expected a lot more people to have run in too bugs on those already. 5300s came out at the end of 2019 so failures for those probably won't start coming out of the woodwork until mid-2027.

I'd have to wonder if this also affects the 5400 series as well at this point... I have to admit I was rather confused as to why their download for the 5400 had one qualified as "for specific customers" with the older one flagged as "for general customers", is anyone aware of what their distinction is here?

Not sure if I'm sitting on a couple bombs here or...
Can someone share the 5100 series updates ala the 5200?

 

[EffrafaxOfWug]

Those are already well beyond the 65k hours threshold - is their performance still decent?

Canaries in the coal mine from the users on the reddit thread said when their 5200s exceeded the threshold they became nigh-unusable.

 

[ca3y6]

Also the original guy on reddit said they also had a bunch of 5100 which didn't experience this problem.

 

[gregsachs]

Those are already well beyond the 65k hours threshold - is their performance still decent?

Canaries in the coal mine from the users on the reddit thread said when their 5200s exceeded the threshold they became nigh-unusable.

Unknown-they are in a storage pool with low demands, most of it is used for backup space.

 

[luckylinux]

Nothing on Hetzner NFS right now about the 5100 .

D0MU071 / D0MU441 / D0MU841 from 2022 are latest.

 

[Stephan]

assuming you have access to the page in question do they provide any notes in the changelogs for the 5100 and 5300 calling out the 65k hour bug

Nothing that specific as far as I can read. Cisco Nexus 9000 Series switches reboot without dumping core because SSD no longer accepts writes.

Conditions: Switch must have a Micron SSD running firmware version as below :- Micron 5100 - D0MU075 and earlier Micron 5300 - D3CN000 and earlier Workaround: None Further Problem Description: Fix is available for with below SSD FW updates :- Micron 5100 - D0MU077 and later Micron 5300 - D3CN001 and later

Fixed in "Nexus 93108TC-EX Switch" Release 16.1(3f).