40TB build, ZFS Raid-50 = 50% perf of Raid-0
- Thread starter gigatexal
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
You get multiple streams for free, but RDMA requires card support. I wonder if gigabit cards support it?
Update: Forget my terminology - Microsoft calls it "SMB Multichannel". See slide 17 and beyond in this presentation: https://www.eiseverywhere.com/file_...arreto_SMB3_Remote_File_Protocol_revision.pdf
Last edited:
As far as I can tell there are only three generally available Ethernet cards that support RDMA. All of them are 10Gbe cards, though they all also support 1Gbe speeds so I guess its possible over GigE.
Mellanox ConnectX-3 (EN or VPI running with Ethernet SFP+ adapter) which implements RoCE (RoCE = RDMA over Converged Ethernet). RoCE is a layer-2 encapsulation of RMDA (basically, stuffing an Infiniband link layer packet directly inside an Ethernet frame). It has the advantage of very low latency, like true IB RDMA, but it cannot be routed so it only works within a single Ethernet switching domain. Technically it also requires "Priority Flow Control" extensions to be implemented on the Ethernet switch, but this is not enforced so it works without it - subject to the fact that any congestion on your LAN will quickly put performance into the toilet. Probably OK for test networks, labs and limited scale work but definitely NOT ok for production without it.
Intel iWARP and Chelsio HPC NIC are both iWARP cards (iWARP = internet Wide Area RDMA Protocol). iWARP encapsulates RDMA into a layer-3 (IP) packet. The advantage is that it is routeable and does not really require specialized support from the Layer-2 switch (no PFC) or the layer-3 switch (although there are some optimizations you can do if the layer-3 switch is "aware" of iWARP). The downside is a slightly increased latency that really diminishes the value of doing RDMA in the first place.
I've not found any other cards supporting RDMA over Ethernet. Last week I picked up 4 ConnextX-3 EN cards for $150/each ($600 total). I'm just beginning to get them installed and have to set up Server 2012 R2 on a couple of machines to test them. I also noticed that MS has installed policies on Windows 8 that prevent activation of SMB Direct - the error message is "only supported on Server OSs". This is a real PITA because my main use case was for fast disk access at the workstation. I guess I could just run Server 2012 with GUI on the desktop but it would be disruptive to tear it down and re-install all the apps (and I'd have to test first to ensure that all the apps I need can even be installed on Server 2012).
Mellanox ConnectX-3 (EN or VPI running with Ethernet SFP+ adapter) which implements RoCE (RoCE = RDMA over Converged Ethernet). RoCE is a layer-2 encapsulation of RMDA (basically, stuffing an Infiniband link layer packet directly inside an Ethernet frame). It has the advantage of very low latency, like true IB RDMA, but it cannot be routed so it only works within a single Ethernet switching domain. Technically it also requires "Priority Flow Control" extensions to be implemented on the Ethernet switch, but this is not enforced so it works without it - subject to the fact that any congestion on your LAN will quickly put performance into the toilet. Probably OK for test networks, labs and limited scale work but definitely NOT ok for production without it.
Intel iWARP and Chelsio HPC NIC are both iWARP cards (iWARP = internet Wide Area RDMA Protocol). iWARP encapsulates RDMA into a layer-3 (IP) packet. The advantage is that it is routeable and does not really require specialized support from the Layer-2 switch (no PFC) or the layer-3 switch (although there are some optimizations you can do if the layer-3 switch is "aware" of iWARP). The downside is a slightly increased latency that really diminishes the value of doing RDMA in the first place.
I've not found any other cards supporting RDMA over Ethernet. Last week I picked up 4 ConnextX-3 EN cards for $150/each ($600 total). I'm just beginning to get them installed and have to set up Server 2012 R2 on a couple of machines to test them. I also noticed that MS has installed policies on Windows 8 that prevent activation of SMB Direct - the error message is "only supported on Server OSs". This is a real PITA because my main use case was for fast disk access at the workstation. I guess I could just run Server 2012 with GUI on the desktop but it would be disruptive to tear it down and re-install all the apps (and I'd have to test first to ensure that all the apps I need can even be installed on Server 2012).
Last edited:
I found that the Mellanox ConnectX-2 cards also support RDMA, although it takes a bit of work to get it in Windows.
yes instead of 2.5 gbps - with windows 8/2012 you end up pushing the max speed link regardless of card. If you carve up virtual functions ensure the OS sees 2.5gbps minimum - it does some automatic guessing.
Honestly if you can get 9gbps out of a $75 nic (per port) then you are doing well. Most folks don't even have the bandwidth to support 9/18/36gbps of disk IO unless they are running massive ramdisk? In which case I suggest using DAS