Confused about Micron 5100 performance with LSI2308 & ESXi passthru

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

hmw

Active Member
Apr 29, 2019
570
226
43
Got 4 x HPE branded Micron 5100 Eco SSDs and 4 x standard Micron 5100 Eco's - all 7.68 TB. I have been testing these and get different results on the LSI2308 vs on a normal Intel SATA port. I've been struggling to find benchmarks for the 5100 Eco's to compare against. What I've found is that there's weird variance in just the 4K performance

Here's Hot Hardware's CDM result of the 5100 Eco for 4K - notice the 4K QD32 speeds are 280/350 MB/sec for R/W ...

1603121352645.png 1603121383227.png


Here's the CDM 4K performance for my 5100, running on the latest D0MU841 firmware, on my LSI2308 HBA, with Windows write caching off and the HBA passed through to Windows Server 2019 - the R/W speeds for 4K QD32 are 228/100 MB/sec ...
1603132033861.png

Here's @Marsh with some CDM results with a 5100Eco 2TB (c.f. https://forums.servethehome.com/ind...0-eco-sata-ssd-250-obo-ebay.29219/post-271664)

Code:
[Read]
Sequential 1MiB (Q= 8, T= 1): 536.198 MB/s [ 511.4 IOPS] < 15611.69 us>
Sequential 1MiB (Q= 1, T= 1): 496.412 MB/s [ 473.4 IOPS] < 2111.00 us>
Random 4KiB (Q= 32, T=16): 356.945 MB/s [ 87144.8 IOPS] < 5869.51 us>
Random 4KiB (Q= 1, T= 1): 27.435 MB/s [ 6698.0 IOPS] < 148.69 us>

[Write]
Sequential 1MiB (Q= 8, T= 1): 481.979 MB/s [ 459.7 IOPS] < 17340.30 us>
Sequential 1MiB (Q= 1, T= 1): 431.159 MB/s [ 411.2 IOPS] < 2428.64 us>
Random 4KiB (Q= 32, T=16): 172.667 MB/s [ 42155.0 IOPS] < 12125.42 us>
Random 4KiB (Q= 1, T= 1): 108.610 MB/s [ 26516.1 IOPS] < 37.50 us>
His 4KQ1 & 4KQ32 speeds are way higher than mine and in line with HotHardware, but still a bit lower though. And the speeds I see are not just one Micron 5100, it's *both* the HPE and normal Micron drives I have, regardless of firmware.

Does anyone have any experience with Micron on LSI 2308 HBAs? Are there known incompatibilities? Micron support is pretty much useless. Could it be because I am passing the HBA to a Windows Server guest via ESXi passthru? Or is it the HBA itself?
 

Attachments

hmw

Active Member
Apr 29, 2019
570
226
43
@hmw

The drive was not fast before firmware update.
@Marsh - I updated the drive to D0MU841, which is the latest. I notice in the thread you indicated the fw update version was D0MU441 - that's the same fw version as mine's on (the first number indicates capacity ie 441 fw for 3.84TB, 841 fw for 7.68TB)

Still wondering why a 2308 HBA, which is a PCIe 3.0 8x HBA, would be worse off than a normal SATA controller. If the ESXi passthru caused speeds to drop, they would also cause the 1MB and 512K sequential speeds to drop, which would mean the CDM results should show a uniform drop in speed ... but it seems only the 4K performance is affected ...
 

Marsh

Moderator
May 12, 2013
2,644
1,496
113
Did you post the result from using normal SATA ports using bare metal window host?
 

hmw

Active Member
Apr 29, 2019
570
226
43
Did you post the result from using normal SATA ports using bare metal window host?
The result is from the 2308 via ESXi. I'll need to take the server down, install Windows on one SSD and attach a second SSD to one of the normal SATA ports - and then do the benchmark runs using that

It's quite an arduous task and hence avoiding it till the weekend :D
 
  • Like
Reactions: Marsh

hmw

Active Member
Apr 29, 2019
570
226
43
I was able to boot into Windows from USB, and take some CDM benchmarks on Windows bare metal

This is the HPE branded Micron :

1603231197173.png

And here's the stock Micron 5100

1603231170239.png

They're the same as the LSI 2308 on ESXi results.

I will try to open up the server and connect the backplane to either a Marvell SATA controller or then the AMD EYPC one - so that I can check whether the drives are this slow or whether it is the LSI2308 controller
 

hmw

Active Member
Apr 29, 2019
570
226
43
Now I am even MORE confused. Switched the drives to use the onboard S8030 AMD SATA controller, booted from Windows. Here's the stock Micron 5100 upgraded to the latest firmware (D0MU841)

1603235839101.png

Just to make sure I wasn't dreaming about those super low write numbers, here's another standard Micron 5100 with D0MU810 firmware

1603235971166.png

WTH ??? Is this something to do with AMD and EPYC and the Micron SSDs?
 

Iaroslav

Member
Aug 23, 2017
111
24
18
37
Kyiv
I'm not so deep into tests, but I've got some 5100 MAX 1.92Tb and was disappointed with its real performance, tests are fine, but they appeared to be way slower than even cheaper DC450R from Kingston. Not saying about DWPD, Microns claim to be way more durable
 

hmw

Active Member
Apr 29, 2019
570
226
43
I'm not so deep into tests, but I've got some 5100 MAX 1.92Tb and was disappointed with its real performance, tests are fine, but they appeared to be way slower than even cheaper DC450R from Kingston. Not saying about DWPD, Microns claim to be way more durable
Yeah the 5100MAX is claimed to do 1-3 DWPD, because it is actually the same as a 5100 3.84TB ECO, with half of the capacity over provisioned. In fact Micron encourages folks to do the same to the ECO via their "Flex Capacity" trick in Storage Assistant

Would you mind sharing any fio or Crystal Disk benchmarks for the 5100 MAX ? What HBA is it attached to?
 

3nodeproblem

Member
Jun 28, 2020
48
11
8
Might be relevant or not, but I'm super frustrated with my Micron 5210 ION 7.62 TB drives. They seem to perform as specified under synthetic benchmarks without any filesystem, at least from what I can surmise. fio targeting /dev/sd* and it does about 250 MiB/s for sequential write benchmark running for 15 min.

Real-world use is a completely different story.

Doing an rsync writing to a single-drive 5210 zpool/dataset (no compression, dedup or encryption, recordsize either 128k or 4k, atime=off) with a large number of 64MiB files and no parallelization: Disk I/O looks somewhat all right (~200 MiB/s) for the first 5min (consistently regardless of how I tweak various zfs filesystem or kernel module parameters or change the IO scheduler). After 5 min it drops to ~50-55MiB/s and ~400 IOPS and there doesn't seem to be any way I can get it above that. Haven't gone over 65% allocation.

Directly to mobos SATA ports, CPU is around 50-70% iowait but otherwise mostly idle.


This is making me very sad. I'm aware of the caveats with QLC and that these shouldn't be compared to TLC drives but at this point even my old 7200 rpm drives are spinning circles around it. And yes latest firmware.

I'm considering filing a warranty claim - is that warranted?
 

DDD

New Member
Oct 8, 2013
6
1
3
Germany
WTH ??? Is this something to do with AMD and EPYC and the Micron SSDs?
I don't think so. The specs I found for 7.68 TB 5100 Eco's indicate low 4K write ops in steady state, only ~9000. Your result still seems to be above that number. Running benchmarks on a new or secure erased drive with enough idle time for the drive to clear its nand will give you better results until you reach the steady state.
The smaller capacity drives have higher write 4K ops so you cannot compare your result with them.
 

Bjorn Smith

Well-Known Member
Sep 3, 2019
876
481
63
49
r00t.dk
If it is QLC then it is basically a consumer drive - QLC as far as I know uses a DDR(4) Cache that is of varying size and when that fills up, the performance TANKs.

read this.


So I would say your QLC drives performs as "expected" - that is why I don't use them - not even in my normal workstations.

QLC are good for capacity + reads
 

hmw

Active Member
Apr 29, 2019
570
226
43
Running benchmarks on a new or secure erased drive with enough idle time for the drive to clear its nand will give you better results until you reach the steady state.
Micron suggested that I secure erase and leave the machine in BIOS. It's in a VM so can't really do that, but from what I can tell - it takes DAYS for this drive to clear internal NAND, after making sure Windows Server is set to not let the internal disks go to sleep ...

Here's the latest result from stock Micron 5100 (D0MU841) and also a HP branded 5100. I've noticed the HPE branded has consistently higher write performance ...

1604069632050.png 1604070269242.png
It is better than before ... but a Samsung 2TB 860 blows this out of the water ...

1604069934891.png

Notice the HUGE difference in 4KQD32 speeds ?

The smaller capacity drives have higher write 4K ops so you cannot compare your result with them.
So perhaps the 8TB drives don't have the same amount of parallel channels as a normal 2/4TB drive ? That sounds bizzare. I read the 5100 8TB uses 12 x Micron NW856 NAND flash chips, these are 768Gb (96G x 8) 333MHz chips.

1604071412819.png1604071442005.png


That's a decent amount of overprovisioning (20%) and also speed - both in terms of clock speed and in channels.

And it still doesn't explain the weird behavior when connected to the onboard SATA controllers. The comment about EPYC was because the write performance drops by 50% when connected to onboard SATA instead of the LSI2308 - for the same drive. It has nothing to do with the NAND state or garbage collection ...
 

DDD

New Member
Oct 8, 2013
6
1
3
Germany
And it still doesn't explain the weird behavior when connected to the onboard SATA controllers. The comment about EPYC was because the write performance drops by 50% when connected to onboard SATA instead of the LSI2308 - for the same drive. It has nothing to do with the NAND state or garbage collection ...
You mean the sequential results? I missed them since in your initial post you were taking only about 4K iops - yes they look wierd.
Micron engineers may know why the specs for the highest capacity drives drop off on 4K write. This seems to be the same for the successors 5200 and 5300.