ConnectX-3 vs ConnectX-3 Pro?
- Thread starter frogtech
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
While we're on the topic: Does anyone know if the hardware is identical between the C-X3 and the C-X3 Pro? I haven't been able to find high res pictures of both cards without heatsinks anywhere but the product shots for both ICs seem to indicate that the IC part is identical.
http://www.mellanox.com/uploads/product_families/cat_81/gfx_00725.jpg
http://www.mellanox.com/uploads/product_families/cat_83/gfx_01819.jpg
http://www.mellanox.com/uploads/product_families/cat_81/gfx_00725.jpg
http://www.mellanox.com/uploads/product_families/cat_83/gfx_01819.jpg
These pictures are pretty good. Heat sink on but the model number on the sticker is the same as my c-x3 and it looks identical. I think there is a good chance it could be the same board. Even the PCB number is the same in the bottom right. So would be nice to see what is under the heatsink. All the ones I see is just a generic card.
Mellanox MCX354A-FCCT ConnectX-3 Pro VPI Dua Port QSFP Adapt Card Low Pro Brckt 641676226150 | eBay
Mellanox MCX354A-FCCT ConnectX-3 Pro VPI Dua Port QSFP Adapt Card Low Pro Brckt 641676226150 | eBay
While we're still on the topic...I'm getting a little confused by the whole RDMA, RoCE v1, v2, iWARP thing. It seems like some of these run on different protocols. In common scenarios you would want RDMA for a Storage Spaces Direct or other hyper-converged environments. But how do you know which protocol to go with and do you really need data center bridging on a switch for it to work? If you do routing at a switch with static routes do you need RoCE v2 which has routable RoCE packets? Does VMWare VSAN benefit from RDMA?
I think the CX-3 Pro cards have a slightly newer asic to support roce v2.
On their switches they have an ethernet switch that's basically a sx6036 that can only do roce v1, a newer revision with a new asic can do roce v2!
RDMA means Remote Direct Memory Access > reading and writing directly to another hosts memory without involving cpu or the tcp/ip stack.
RoCE = RDMA over Converged Ethernet, it's mellanox implementation of RDMA for ethernet.
There are currently two versions of that protocol, v1 and v2. v1 is using ethernet frames for RoCE and is limited to the "local" network (not routable), v2 is using ip packets for RoCE making it a routable protocoll.
RoCE v1 is supported by CX-2 and CX-3 cards.
RoCE v2 is supported by CX-3 Pro and newer CX cards (they are backward compatible with roce v1)
iWARP is the standard protocol for RDMA over ethernet networks, but besides Chelsio nobody uses it (not even Intel who helped establishing that standard!)
So would local mean devices/subnets that are directly connected on the same switch?
If RoCE is mellanox's implementation, was there ever "just RDMA", or was iWARP the first implementation? I guess to me it sounds like RDMA is a standard, and these consortiums implement their own value adds or product features based on them.
RDMA functions over Infiniband networks. RoCE and iWARP are the implementations that allow RDMA to occur over Ethernet.
Update link to Mellanox NICs comparison table
Mellanox Adapters - Comparison Table
Mellanox Adapters - Comparison Table
Mellanox Adapters - Comparison Table
Mellanox Adapters - Comparison Table
Many common RDMA protocols assume a lossless network, and will lock up or degrade dramatically if there is any RDMA packet loss. One way to detect the issue is to run a bandwitdh test such as ib_send_bw -a, that ramps up from low to high bandwidth. If it ramps up to %95 and stays there, great. If it ramps up to %50 and then gets stuck at 0%, it can be a flow control issue.
Two ways for an ethernet switch to support lossless RDMA include: global pause or priority flow control (PFC). In simple terms, Priority flow control can apply to an individual vlan, while global pause affects all switch ports and traffic.
Data center bridging includes PFC, but PFC and ECN (congestion notification) is the crucial requirement.
Some software (eg.Windows SMB Direct, and Vsphere NVMeoF, vmotion and ISER) require PFC, and what they are requiring is lossless RDMA.
The link below shows that ECN and PFC are required for a lossless network with Mellanox cards. That offers the widest support for various protocols. Even if you limit use to a Layer 2 isolated network with no other traffic, you still need flow control for a lossless network.
Mellanox has a table comparing lossy and lossles configurations.
A CX3 Pro can be used for a lossless ethernet network, whereas I don't think the CX3 can. But it really depends on the whole software and hardware recipe, because the two ends and the switch all need to all be configured for a compatible flow control method.
