TOR Switch Needed for 3-Node Hyper-Converged S2D 2016 Cluster

[Dave Waasdorp]

Using Windows Server 2016DC, I'm building a 3-node Hyper-converged, Storage Spaces Direct failover cluster. It seems to work good and passes validation using an ordinary D-Link 10GB switch except the fact that I can't see any RDMA traffic over the switch. I confirmed that I can see RDMA activity when I bypass the D-Link and connect the 3 nodes directly.

The problem is, I'm having trouble finding a TOR switch that's RDMA (RoCE) compatible for under $5k. to replace that D-Link. Anyone have any suggested models?

I was also considering changing to iwarp or infiniband instead. What do you think?

 

[rune-san]

Is this your own home or you looking at new supported equipment? RoCE requires Lossless Ethernet, which therefore requires some sort of Priority Flow Control, which therefore likely requires Data Center Bridging support. Essentially, a switch with PFC or DCB should be able to get you what you need, but on the new market, they can still be quite expensive. An HPE 5700 series in 10Gb flavor should come a hair under $5K brand new, so that's what I'd probably vouch for.

 

[cesmith9999]

Technically you need switches with PFC and DCB to have RDMA (RoCE) work correctly. which usually means you need a fully managed switch (not smart or web managed).

DXS-3400-24TC - $3500-$4500

ebay may be a better choice for sourcing your switches...

Chris

 

[jmck]

I haven't received mine yet but I just picked up one of these to do the same thing you're looking at. They took $900 for it. It's 40Gb instead of 10Gb but you could use a breakout cable or upgrade your cards to 40Gb. It is EoS, so you can no longer add/renew a service contract and it's EoL'd the end of this year.

You can find 7050QX-32's and Dell Z9000's used/refurbished cheaper than this if you aren't looking for something new. I was only looking at 40Gb when shopping so I'm not sure of any 10Gb switch deals.

New Arista DCS-7050Q-16-R 16x 40Gb QSFP+ Switch w/ 8x 10Gb SFP+ R-to-F Air - JMW | eBay

 

[i386]

@Dave Waasdorp
You can use roce on a switch without pfc/dcb.
It will work, but it's not a supported configuration for a production environment.

@jmck
EOS (aristas networking os) is far from end of life. Certain switches are "end of sales" or "end of life".

The switch from your link for example is "end of sale" since june 2015, and december 2017 was the last chance to renew support contracts. The dcs-7050q-16-r can use the eos images that are realeased till december 2018.
Source: End of Sale of 40GbE Models of the Arista 7050 Series - Arista