Broadcom 9560-16i and ToughArmor MB699VP-B

[akeen]

micron 9300 max model: mtfdhal3t2tdr-1at1zabyy

 

[dbTH]

so my advice to anyone out there who wants to use Broadcom 9560-16i with some nvme drives,
here is a list of things you should know
there is no nvme expander card from broadcom for the 9560-16i so you can only use 4 nvme drives max on each card
kioxia sucks dont use it
i have success with micron 9300 max and I suggest stick with that
speed of the raid 5 with 4 drives are insane
here is comparison single drive vs raid5 with 9560-16i

AS2124us-TNRP server with kioxia 12.8 tb nvme pcie4 direct attached to motherboard
windows 2019 server
kloxia kcd6xvul12t8 12.8tb u.3

------------------------------------------------------------------------------
CrystalDiskMark 8.0.4 x64 (C) 2007-2021 hiyohiyo
Crystal Dew World: Crystal Dew World
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
SEQ 1MiB (Q= 8, T= 1): 4003.678 MB/s [ 3818.2 IOPS] < 2094.30 us>
SEQ 1MiB (Q= 1, T= 1): 3077.569 MB/s [ 2935.0 IOPS] < 340.48 us>
RND 4KiB (Q= 32, T= 1): 526.520 MB/s [ 128544.9 IOPS] < 240.93 us>
RND 4KiB (Q= 1, T= 1): 39.477 MB/s [ 9637.9 IOPS] < 103.59 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 4029.832 MB/s [ 3843.1 IOPS] < 2076.96 us>
SEQ 1MiB (Q= 1, T= 1): 4032.671 MB/s [ 3845.9 IOPS] < 259.69 us>
RND 4KiB (Q= 32, T= 1): 380.773 MB/s [ 92962.2 IOPS] < 333.15 us>
RND 4KiB (Q= 1, T= 1): 220.150 MB/s [ 53747.6 IOPS] < 18.46 us>

Profile: Default
Test: 1 GiB (x5) [C: 1% (61/11920GiB)]
Mode: [Admin]
Time: Measure 5 sec / Interval 5 sec
Date: 2022/01/27 18:58:17
OS: Windows Server 2019 [10.0 Build 17763] (x64)

*******************************************************************************************

AS2124us-TNRP server with 4 x kioxia 12.8 tb nvme 2 x 9560-16i
windows 2022 server
raid 5 with 4 x kloxia kcd6xvul12t8 12.8tb u.3


------------------------------------------------------------------------------
CrystalDiskMark 8.0.2 x64 (C) 2007-2021 hiyohiyo
Crystal Dew World: Crystal Dew World
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
SEQ 1MiB (Q= 8, T= 1): 14770.785 MB/s [ 14086.5 IOPS] < 567.58 us>
SEQ 1MiB (Q= 1, T= 1): 7353.094 MB/s [ 7012.5 IOPS] < 142.43 us>
RND 4KiB (Q= 32, T= 1): 490.657 MB/s [ 119789.3 IOPS] < 266.90 us>
RND 4KiB (Q= 1, T= 1): 113.208 MB/s [ 27638.7 IOPS] < 36.05 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 4292.017 MB/s [ 4093.2 IOPS] < 1950.82 us>
SEQ 1MiB (Q= 1, T= 1): 4353.206 MB/s [ 4151.5 IOPS] < 240.60 us>
RND 4KiB (Q= 32, T= 1): 448.459 MB/s [ 109487.1 IOPS] < 283.70 us>
RND 4KiB (Q= 1, T= 1): 114.401 MB/s [ 27929.9 IOPS] < 35.66 us>

Profile: Default
Test: 1 GiB (x5) [C: 1% (13/1023GiB)]
Mode: [Admin]
Time: Measure 5 sec / Interval 5 sec
Date: 2022/03/08 8:42:58
OS: Windows Server 2019 Server Standard (full installation) [10.0 Build 20348] (x64)

Regarding this: "AS2124us-TNRP server with kioxia 12.8 tb nvme pcie4 direct attached to motherboard", your max read throughput of a single drive is much lower the speed of 5500 MB/s as published by koxia. IOPS is also lower What's wrong here? Is your load profile mixed, Read+Write simultaneously ?

 

[akeen]

i have never reached what they advertise and what actually is , what is load profile?
i have just realized crystaldiskmark has an option for nvme now, i will run again see if it makes any difference

 

[dbTH]

i have never reached what they advertise and what actually is , what is load profile?
i have just realized crystaldiskmark has an option for nvme now, i will run again see if it makes any difference

I don't own any Koxia ssd. But the Koxia kcd6xvul12t8 PCIe 4.0 disk you have appears to be weak. Seems it has no match against Intel (Solidigm) PCIe 4.0 P5506 AIC that I have. I use Linux base fio tool, and here's what I got with iodepth=64 bs=64K read only I/O on a single disk:

read: IOPS=114k, BW=7127MiB/s (7473MB/s)(835GiB/120003msec)

The bandwidth is inline with the published one .

We use Micron 9300 Max U.2 ssd at my work and I also have few such disks personally. Using software RAID10, and very impressive performance