I have a couple of these, HPE branded, vk001920gwsxk. I assume the Micron FW doesn't work on these? HPE seems to a version HPE3(E) from sep 20. 2024, not sure it this is new enough?
Micron 5200 firmware bug
- Thread starter ca3y6
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I think that's basically going to be down to how different from the official firmware the HPE-specific models are; given that on the face of it this looks like it's only going wrong on a 16bit overflow on a certain SMART attr, and that HPE and other OEMs often have custom SMART attrs and counters, it's not out of the realms of possibility that the HPE drives are unaffected.
Certainly I'd have expected a fix or at least an advisory from HPE by now if this problem started rearing its head in mid-2025.
Certainly I'd have expected a fix or at least an advisory from HPE by now if this problem started rearing its head in mid-2025.
I have debated about responding to my support tickets with something along the lines of
Hey, guess what, that firmware you said does not exist, well here it is,how about that?
But I do not want to make this too public to micron in case they either get the OP in trouble somehow or make the files disappear from the Internet....
I should clarify my mid-2025 date. I don't know if that's when it was first discovered but that is around the time that the first shipped drives would have experienced the problem if they'd been on 24/7.
Mid-2025 sounds about right - my oldest is from october 2025, and it was usually a good few of months after release before they'd filter in to the UK retail channels (usually much longer these days), and I usually don't buy anything until it's been out for at least a couple of months.
The press release and the first review were both in January 2018.
65535 hours is just under 7.5 years so assuming it wasn't caught beforehand mid-2025 is the very earliest you could have expected to see this in the wild.
The press release and the first review were both in January 2018.
65535 hours is just under 7.5 years so assuming it wasn't caught beforehand mid-2025 is the very earliest you could have expected to see this in the wild.
I read this exact article and it was one of the reasons why I got the eco 1.92 TB drives to begin with
Another post on the thread, this one using the Lenovo-branded FW (who it looks like haven't released an update yet either) said that normal performance was resumed with a re-flash of the current FW :shrugs:
Half-tempted to let one of my as-yet-unpatched drives roll over the 65k hours and see what happens, but there's probably ample time for more canaries in the coal mine to see if that works for anyone else.
Half-tempted to let one of my as-yet-unpatched drives roll over the 65k hours and see what happens, but there's probably ample time for more canaries in the coal mine to see if that works for anyone else.
I work for an educational institution, and we have over 20 Lenovo servers equipped with Micron 5200 SSDs.
We've noticed that reapplying the Lenovo MG41 firmware temporarily alleviates the issue, but only for a few days. After that, the problems return: long response times reported in Windows Resource Monitor, timeouts, and various errors flagged by the MegaRAID utility.
We used the MSECLI utility to collect debug data from the affected drives. The problem is that most of the output files are in BIN format and are not human-readable. We've tried hex editors and variety of converters without success, which leads me to suspect that proprietary software may be required to interpret these files.
I know some people have experimented with the Dell D1MU firmware family, but it appears to target a slightly different drive SKU than the ones we have. For example:
My questions are:
We've noticed that reapplying the Lenovo MG41 firmware temporarily alleviates the issue, but only for a few days. After that, the problems return: long response times reported in Windows Resource Monitor, timeouts, and various errors flagged by the MegaRAID utility.
We used the MSECLI utility to collect debug data from the affected drives. The problem is that most of the output files are in BIN format and are not human-readable. We've tried hex editors and variety of converters without success, which leads me to suspect that proprietary software may be required to interpret these files.
I know some people have experimented with the Dell D1MU firmware family, but it appears to target a slightly different drive SKU than the ones we have. For example:
- 5200MMU02EHC0677 (D1MU040, D1MU440, D1MU540, D1MU840)
- 5200MLE0D4HC0668 (MG41 from Lenovo)
My questions are:
- Does anyone have access to a newer Lenovo firmware release, assuming one even exists?
- Does anyone know how to decode the debug files generated by MSECLI?
Last edited:
When I saw the post on the reddit thread, I did have a delve to see if there was any sign of a firmware update from Lenovo - not because I have any myself, I'm just curious how public Micron are making this.
In a curious coincidence, another post on the reddit thread today about HPE releasing an updated firmware for drives that sound like 5200s underneath, so these clearly have gone out to at least one OEM; specifically the following model numbers:
- VK000480GWSXF
- VK000960GWSXH
- VK001920GWSXK
- MK000480GWUGF
- MK000960GWUGH
- MK001920GWUGK
...so it might be worth prodding Lenovo to see if there's one in the pipeline.HPE said:
Edit: In relation to your second bullet point, I've never tried using msecli's debug dump feature before but I've just given it a whirl on a random drive to see the sort of stuff you're looking at.
Code:
root@wug:~# msecli -P debug.zip -n /dev/sda
Collecting debug data........................................................................................................................................................
Zip file is created on the specified location
Device Name : /dev/sda
Debug data has been retrieved successfully
CMD_STATUS : Success
STATUS_CODE : 0
Copyright (C) 2020 Micron Technology, Inc.
Code:
root@wug:~/msecli_debug/<serial>/Controller# strings firmware_version.bin
Compiled:
Oct 24 2019 15:59:04
a654e79
User Capacity:
SVN Info:
$Rev: a654e79 $
$URL: ssh://git@bitbucket.micron.com/ssdfw/acadia.git {release/Dylan_AD0161_Release} $
Internal Ver:
5300MMU025HD0185
External Ver:
D3MU801
Meta Info:
Rev: 0.0
BootLoader Ver:
$Rev: a654e79 $
$URL: ssh://git@bitbucket.micron.com/ssdfw/acadia.git {release/Dylan_AD0161_Release} $
NAND Info:
NAND Reticle:
0013
NAND Cell Rev:
NAND MPPR:
NAND MTPR:
NAND Prm CRC:
1B 6B 91 DDMy gplog's appear to be mostly entirely empty in fact, mostly just long strings of 0's;
Code:
root@wug:~/msecli_debug/<serial>/Controller# xxd gplog_A6.bin |head
00000000: 2500 0000 75a0 0000 0000 0000 0000 0000 %...u...........
00000010: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000030: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000050: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000060: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000070: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00000090: 0000 0000 0000 0000 0000 0000 0000 0000 ................
root@wug:~/msecli_debug/<serial>/Controller# xxd gplog_A6.bin |tail
003e7f60: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7f70: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7f80: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7f90: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7fa0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7fb0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7fc0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7fd0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7fe0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
003e7ff0: 0000 0000 0000 0000 0000 0000 0000 0000 ................Edit 2: Seems like those binary logs being mostly empty might be because of these lines in mseapiLogFile.txt. GPL logs = General Purpose Logs logs which are part of the ATA spec; potentially that didn't work because the system in question is lacking sg_sat_read_gplog.
Code:
Thu Jun 11 19:57:26 2026:WARNING: GetDebugData: Getting SMART Logs failed for Drive : /dev/sda
Thu Jun 11 19:59:56 2026:WARNING: GetDebugData: Getting GPL Logs for client drive failed for Drive : /dev/sda
Last edited:
It appears that Lenovo released two firmware packages on June 4th and 5th for Windows and Linux, respectively. However, for some unknown reason, only the Linux package includes the MG42 firmware for Micron 5200 drives, which I only realised today.
Also, I haven't found anything in the release notes detailing what was fixed in this update. That said, given that it was released around the same time HP published its own update—and explicitly confirmed what many of us already suspected: "Addresses performance degradation issue for read/write workloads when drive exceeds 65,535 hours (~7.5 years) of power-on time."—there's a good chance this firmware addresses the same underlying issue.
Assuming that's the case, we should be in good shape.
Also, I haven't found anything in the release notes detailing what was fixed in this update. That said, given that it was released around the same time HP published its own update—and explicitly confirmed what many of us already suspected: "Addresses performance degradation issue for read/write workloads when drive exceeds 65,535 hours (~7.5 years) of power-on time."—there's a good chance this firmware addresses the same underlying issue.
Assuming that's the case, we should be in good shape.
Last edited:
HPE released a new version last week where they say " Addresses performance degradation issue for read/write workloads when drive exceeds 65535 hours (~7.5 years) of Power-On hours. "
Edit: Sorry, this is already posted above
Edit: Sorry, this is already posted above
Last edited:
Apparently Micron 5100 is also affected by this issue, same changes are listed in the release notes for HPE OEM Micron 5100.
Lenovo's package also includes updated firmware for their version of the Micron 5100.
Lenovo's package also includes updated firmware for their version of the Micron 5100.
Cisco got hit too, there are updated firmwares behind paywall:
Micron 5100: D0MU077
Micron 5300: D3CN001
(There could be even newer versions but Micron homepage has nothing)
So basically, if you have a Micron 5x00 and firmware older than end of 2025 or 2026, this bug will affect you after 65535 hours.
Micron 5100: D0MU077
Micron 5300: D3CN001
(There could be even newer versions but Micron homepage has nothing)
So basically, if you have a Micron 5x00 and firmware older than end of 2025 or 2026, this bug will affect you after 65535 hours.
The 5300 is still supported by Micron so I am watching their download page anxiously (though I have a few years before that's a problem)
Interesting; assuming you have access to the page in question do they provide any notes in the changelogs for the 5100 and 5300 calling out the 65k hour bug?
I don't have any 5100s myself but given they came out ~2016 I'd have expected a lot more people to have run in too bugs on those already. 5300s came out at the end of 2019 so failures for those probably won't start coming out of the woodwork until mid-2027.
I'd have to wonder if this also affects the 5400 series as well at this point... I have to admit I was rather confused as to why their download for the 5400 had one qualified as "for specific customers" with the older one flagged as "for general customers", is anyone aware of what their distinction is here?
Not sure if I'm sitting on a couple bombs here or...
Can someone share the 5100 series updates ala the 5200?
Those are already well beyond the 65k hours threshold - is their performance still decent?
Canaries in the coal mine from the users on the reddit thread said when their 5200s exceeded the threshold they became nigh-unusable.
Canaries in the coal mine from the users on the reddit thread said when their 5200s exceeded the threshold they became nigh-unusable.