4x NVMe HBA with backplane for ESXi

[bekax]

Hello everyone.

I'm trying to upgrade my HPE ML30 Gen10 with a couple NVMe drives.
I really liked the IcyDock MB699VP-B and MB720M2K-B so that is the plan for the backplane.

The issue is that this server will only support 8x8 bifurcation and not 4x4x4x4 (I believe that's also a limitation of Xeon E 2000 processors according to manual).
Thus, every Retimer or Redriver Supermicro HBAs are not an option since they require 4x4x4x4 bifurcation.

Now I also saw a couple of HBA/RAID cards but none of them seem to support ESXi (7.0 would be best), support 32Gbps bandwidth for each SSD (4x PCIe), r even have x16 PCIe 3.0 host bus ..
Supermicro AOC-SLG3-8E2 or Highpoint SSD7120, won't support ESXi so that's a shame.
Broadcom HBA 9400-16i only supports SAS3 speeds (12Gbps) and I wanted to try and get the most performance out of the setup.

Are there any other good HBA or RAID cards for this use case?

 

[NablaSquaredG]

Thus, every Retimer or Redriver Supermicro HBAs are not an option since they require 4x4x4x4 bifurcation.

Are you sure about that?

The manual says:


To me it looks like this is a proper PCIe switch that does not require motherboard bifurcation

 

[bekax]

Are you sure about that?

The manual says:
View attachment 17404

To me it looks like this is a proper PCIe switch that does not require motherboard bifurcation

You raise a good point.
But isn't that just for allowing multiple NVMe with bifurcation ?
Imagine I am using 4 drives with 8x8 bifurcation how will the server know there are 2 nvme drives in each x8 lane ?

I may be wrong, but I think Retimer/Redriver don't have any PCIe switch..

According to Supermicro Storage RAID / HBA Cards perhaps the most suitable model would be Supermicro AOC-SLG3-8E2 (with PCIe Switch!), but again this specific one doesn't support ESXi.

 

[Rand__]

YI may be wrong, but I think Retimer/Redriver don't have any PCIe switch..

Correct.

According to Supermicro Storage RAID / HBA Cards perhaps the most suitable model would be Supermicro AOC-SLG3-8E2 (with PCIe Switch!), but again this specific one doesn't support ESXi.

I'd ask SM why not - the most likely answer is they have not tried/validated which means you're on your own with this, but I dont see a real need for OS support on pcie level cards; the OS should be agnostic to the physical layout.

Only thing that might happen is that ESXi is too picky re an extra pcie switch (why ever, added complexity, levels of devices, latency or whatever)

 

[bekax]

Well it's very strange because the retimer/redriver versions do in fact specify VMWare support..
Switched HBAs don't..

 

[Rand__]

Yes I've seen that, no idea why.
I'd ask

 

[andrewbedia]

Maybe try this? https://forums.servethehome.com/ind...apters-that-do-not-require-bifurcation.31172/

 

[bekax]

Maybe try this? https://forums.servethehome.com/ind...apters-that-do-not-require-bifurcation.31172/

That is super interesting.
I will for sure order one of the 8port cards for my server and test with 4 nvme.
I guess soon (1 year?) we will start to see the large companies offering these..

 

[bekax]

Maybe try this? https://forums.servethehome.com/ind...apters-that-do-not-require-bifurcation.31172/

Some of the sales I've seen report the 8-port Ceacent card to have 12Gbps connection (for each SSD).
However LinkReal has more details regarding the specs, and appears to be fully compatible with 32Gbps SSDs..
But both of them use the same chips..
Are these chips really PCIe x4 for each port ?
Anyway after the 4th SSD the Host Bus (PCIe 3.0 x16) will be maxed out!

And I'm not yet sure what's the difference between the PLX PEX 8748 and 8749 chips?