Mellanox ConnectX-3 (RDMA) RoCE v1 or v2?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

koifish59

Member
Sep 30, 2020
66
19
8
I plan on initially having a cluster of 2 hosts running ESXi, with a ICX6610 switch as the backbone. On each host, I will be using a single 40GbE port for vSAN and vMotion traffic on separate vLANs and trunk it to a port on the switch. The LAN port will be connected to the SFP+ ports.

I’m looking to pick up a Mellanox ConnectX-3 NIC for the 40GbE port. I can’t seem to find the PRO model (to support RoCE v2) for a decent price, so will a ConnectX-3 non-pro be sufficient for this? Will I benefit from RoCE v2 in any way or will RoCE v1 be good enough?

pfsense, freenas, and about a dozen other VMs will be on this cluster. Just wanted to confirm this setup and hardware will all work correctly before I purchase the switch and NICs.
 

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
Will I benefit from RoCE v2 in any way or will RoCE v1 be good enough?
Like always: "it depends"
RoCE v1 is a layer 2 protocol and mellanox is the only vendor with it.
RoCE v2 supports layer 2 & 3 and is more documented ("open") and available from different vendors (For instance Broadcomm supports RoCE v2 in their newest nics)
If all the software you want to use supports RoCE v1 you don't lose many features compared to RoCE v2.

More information about RoCE: https://community.mellanox.com/s/article/roce-v2-considerations
 
  • Like
Reactions: koifish59