Poor write speeds to Sata SSDs using LSI Megaraid 9361-16i

[zerosix]

Over the last few months I've been having continuous performance problems with my SSDs. I originally had them configured in a 4 disk, raid 5 array, however have tested them in almost any other configuration you can think of in order to try and get them to work. Even if I put the controller in JBOD mode and pass them through as dummy disks, don't install a hypervisor and just run vanilla windows 11, I still get performance problems.

So what performance problems? The disks will write at normal speeds until the cache is saturated (I'm guessing, this is how it looks to me). If I disable the cache on the raid volume, I see the performance issues immediately. The problem is, I cannot seem to push more than 50 - 100MB/s consistently, and quite often, the speed drops to ZERO for several seconds at a time, dead stop. The same thing happens in JBOD mode, write speeds go to zero for many seconds at a time. I'm doing testing with crystal disk mark and also just normal file copy operations using a large flat file of zeroes.

I have tried testing my disks individually over Sata directly to the motherboard bypassing the controller and they perform at the advertised W/R speeds over extended periods with large files.

If you can name it, I've probably tried it, but I'm open to suggestions. I'm desperate at this stage as I have no idea where the issue is, disks? controller? some other bottleneck? drivers? Does the controller just not like the disks? I don't know.

System specs:
Asus X99 Deluxe
Intel Xeon E5-2697 v3
4x 32GB DDR4 2400MHz
Megaraid 9361-16i (I also have a 9460-16i which experienced the same problem).
This drive cage: MB038SP-B_8 Bay 2.5" SAS/SATA HDD/SSD Mobile Rack for 5.25" Bay
4x 4TB Teamgroup Vulcan Z QLC 2.5in SATA SSD 4TB (T253TY004T0C101)

• I'm aware the above disks are not going to be considered good SSDs, so know that I have also tried 2x Samsung 870 EVO SSDs in RAID 0 and in JBOD, I get the exact same performance issues.

I have also tried placing the controller and drives in other entirely different computers just for testing and still get the same issues. I dont think it's the controller's fault as I have two controllers with the same problem, but perhaps the consumer sata SSDs are just not compatible with the controller in some way.

I'm happy to provide any other information or run specific tests.

Thanks in advance.

 

[shpitz461]

How much data are you writing at a time when this slowdown happens?
The DRAM cache is a mechanism for drive manufactures to hide the terrible TLC/QLC nand under the hood. Terrible speeds like that are common in el-cheapo drives.
Check out this review, he also mentions the 10MB/s write speed...

 

[zerosix]

How much data are you writing at a time when this slowdown happens?
The DRAM cache is a mechanism for drive manufactures to hide the terrible TLC/QLC nand under the hood. Terrible speeds like that are common in el-cheapo drives.
Check out this review, he also mentions the 10MB/s write speed...

Anything more than a few GB - A 10GB file for example will cause it.

I will retest the Samsung SSDs I have with a few tests to see if they perform any better, since the samsungs I believe are MLC and should perform well under sustained writes.

 

[CyklonDX]

try disabling cache, and see the results.

 

[name stolen]

I'm sorry to say but this is expected QLC behavior. It shouldn't be this way with products actually on the market, yet it is. I've returned every QLC drive I've ever tried.

 

[i386]

Vanilla broadcom Controllers or OEM Versions? I remember sas2 hab/Controllers from oems Had a much lower queue depth

 

[acquacow]

I couldn't get any sas3 cards to play nice with zfs and proxmox with my enterprise SSDs. Any writes resulted in channel resets and high I/O wait numbers. I had to drop down to a sas 2 card (9200 8i) and now I'm hitting rated speeds (up to slot/card limit).

 

[shpitz461]

Sounds to me like a heating issue, did you check the temps of the sas3 cards?

you can run the following command:

./storcli64 /c0 show all |grep -i temperature