RAID upgrade advice (from W2016 LSI 9750-8i to W2019 and 9460-16i with NVME + SATA)

[KLMR]

Hi,
I've been using a 3ware (LSI) 9750* since W2008R2 (+intel expander) +BBU and has been working rock solid almost 24/7 for more than 10 years.
*9750-8i seems to be 9260-8i (SAS 2108).

Usage scenario:
- I use it to store: run some services, raw data (SMB access), data HDD (vhdx) from 4-5 VM and backups (from machines to this server, and then from this server to a backup server).
- New usage: Storage for FEM simulations (60-100GB files and 40-60GB filesets (small files)**).

Server is an workstation HP420v2 E5-2697v2 64GB RAM, W2016. (May be upgraded to 5950x, but I want ECC RAM)

In my storage-update path I would like s a new RAID controller for:
RAID 6 (8drive) 18 or 20TB drives + 1 cache drive (1-2TB).
or 2 RAID5 (4drive) 18-20TB drives + 1 cache drive (1TB) for each raid.
+my current raids (4-5 4 disk sets)


I cannot afford a "last gen" raid controller from broadcom. My candidates are:
9460-8i
or
9361-8i
+ BBU + Cable adapters (hints appreciated to connect SATA drives).

I would like to use my current SAS expander/s*** (RES2SV240) to:
RAID CONTROLLER --> EXPANDER0 --> DISK0..DISKn
--> CacheNVME0..CacheNVMEp (or SSD)
--> EXPANDER1 --> DISKn+1..DISKm
--> CacheNVMEp+1..CacheNVMEq (or SSD)

My goals:
- Good transfer from multiple access points (VMs, SMB, etc.)
- Good response (IOPs) (take advantage of a RAID with cache drive)
- Good rebuild/check times (I think the 9750 my suffer from such a big disks)
- Robustness I've experienced so far
- 1000MB/s if I update to 10GB part of the network.
- Easy handle of permissions, VMS, and storage from the Wserver
- Power footprint as low as possible.

Any thoughts, comments, alternatives would be appreciated. (Using an HBA + Storage spaces ??). I've been thinking to move to Truenas but I think only adds more power consumption, more money (10GB network), and more complexity to the mix. 9750 in W2016 is "unsupported", I presume I wont be able to run it in W2019.

***I think I'll have to directly attatch NVME drives If I want to use them as cache. (?)

 

[i386]

RAID CONTROLLER --> EXPANDER0 --> DISK0..DISKn
--> CacheNVME0..CacheNVMEp (or SSD)
--> EXPANDER1 --> DISKn+1..DISKm
--> CacheNVMEp+1..CacheNVMEq (or SSD)

Only the 8460-8i supports nvme ssds and has 2 "nvme ports" (1nvme pcie 3.0 port = 4x sas 12gbit/s ports = 1 multi lane sas 12gbit/s port).
If you want to use hdds/sas + nvme with this controller the nvme cache will be read only (for write cache you need redundancy like raid 1 or 5).

***I think I'll have to directly attatch NVME drives If I want to use them as cache. (?)

I'm not sure what you mean with this.
If the nvme ssds is not connected to the 9460-8i, it can't be used as cache for the raid.

Using an HBA + Storage spaces ??

I tried storage spaces the last time with server 2019 and gave up on storage spaces. Microsofts software "raid" implementations are pretty bad performance wise.
If you need windows stick to hardware raid ._.

 

[KLMR]

Only the 8460-8i supports nvme ssds and has 2 "nvme ports" (1nvme pcie 3.0 port = 4x sas 12gbit/s ports = 1 multi lane sas 12gbit/s port).
If you want to use hdds/sas + nvme with this controller the nvme cache will be read only (for write cache you need redundancy like raid 1 or 5).

I'm not sure what you mean with this.
If the nvme ssds is not connected to the 9460-8i, it can't be used as cache for the raid.

I tried storage spaces the last time with server 2019 and gave up on storage spaces. Microsofts software "raid" implementations are pretty bad performance wise.
If you need windows stick to hardware raid ._.

In my scheme I set the NVME after the expanders. For the 9460-8i,
Can I have the NVME direclty attached to the controller and the HDD to the expander right?

I assumed that the cache storage on the 9460 worked as a L2ARC. Right know "write catching" is performed by windows, then the 9750 and I presume the own HDD's RAM.


I forgot to mention, if you can help me figure out the required cables. From SFF-8643 to 8087 (for the expander) and to NVME drives.

 

[KLMR]

I know this topic is kind of zombie. But I finally fot a 9460-16i.
Two of the SFF-8643 will go to regular SATA drivers
The third is intended for NVME so I'm looking for the marked or compatible variants of these cables, especially the 8643 to two 8639.

the alternative would be to use 8643 -> 8639 -> backplane NVME U2 - M2 adapter, is that possible?

I've found "variants" of

single-8643 to one-8639



two-8643 to two-8639



But not those single "8x" 8643 to two 8639

In the official cable listing here:

Broadcom Inc. | Connecting Everything

docs.broadcom.com

I'm looking for variants of these cables (1) and (2) but I've been unable to find them.