qsfp+ switch seems too good to be true

[AllenAsm]

Thank you, that's really clear and makes a lot of sense. The DAC appears is all I need for this.

So the last piece i need then is the network cards. Would this card work with all of this equipment so far?

Pardon Our Interruption...

This is all for my personal lab and has hyperv and s2d everywhere. My hope is/was that we could use RDMA or ROCE working on these to speed things up even more.

 

[NablaSquaredG]

Too expensive ;-)

You should look for OEM CX354A (QCBT), they can be had for 20-25$ and easily be crossflashed to FCBT firmware that gives you 56G Infiniband and Ethernet (40G Ethernet works too, of course)

Re RoCE:
Keep in mind that ConnectX-3 NON-Pro only support RoCE v1. Only ConnectX-3 PRO and upwards support RoCE v2.

 

[AllenAsm]

so does the cx354a cross flashed support rdma or roce?

 

[AllenAsm]

does anyone know if the connectx-3 cards do qsfp+ or is it an infiniband protocol thats going to be flowing over everything?

 

[NablaSquaredG]

so does the cx354a cross flashed support rdma or roce?

Crossflashed cards are identical to genuine cards and have all features you'd also get from a genuine card. So yes, RDMA and RoCE would work on a crossflashed CX354A

does anyone know if the connectx-3 cards do qsfp+ or is it an infiniband protocol thats going to be flowing over everything?

Sorry, I can't follow...

QSFP+ is just the connector and supports different protocols (Infiniband, Ethernet, SAS, etc...) with the appropriate cables / optical transceivers.
Mellanox CX-3 support Ethernet and Infiniband.

 

[AllenAsm]

QSFP+ is just the connector and supports different protocols (Infiniband, Ethernet, SAS, etc...) with the appropriate cables / optical transceivers.
Mellanox CX-3 support Ethernet and Infiniband.

This is the part I didn't understand, thank you for clarifying! So with that said, will these connectx-3 cards be using ethernet or inifiband protocol and/or does it matter?

Also, it just occurred to me that an 8x pci only has 8gb/s bandwidth so how is it possible to even use qsfp+ 40gbe with that?

 

[i386]

Also, it just occurred to me that an 8x pci only has 8gb/s bandwidth so how is it possible to even use qsfp+ 40gbe with that?

Thesese are different units...
40GBE = 40 Gigabit/second Ethernet (~3.9 Gigabyte/second)
x8 PCI ~8Gigabyte/second

 

[NablaSquaredG]

Yes, it does matter ;-)
RDMA is easier to set up with Infiniband and Infiniband also supports IPoIB. But IPoIB lacks the hardware acceleration, so you won't get full throughput if your application uses Ethernet, but you set up an Infiniband link.

CX3 has three modes per Port: Auto (IB or Eth), IB or Eth. Auto tries to negotiate IB and falls back to IB.

PCIe 3.0 x8 has ~7.887GByte (without overhead) which is more than 40GBit (40Gbe = 40Gbit, not 40GByte)

 

[AllenAsm]

I think I'm going to have to use ETH even with losing some of the throughput. I read up on hyper-v and it says "Hyper-V Virtual Switch only supports Ethernet, and does not support any other wired local area network (LAN) technologies, such as Infiniband and Fibre Channel".

I bought the cables and the switch so the cards are the last thing I need. I'd love to do the CX354A with cross flash but I'm not sure I will have time to really do the work on those and get them flashed up so I'm leaning towards the connectx-3 pro cards. RDMA is crucial for this build as the s2d/hyperv cluster docs say it will speed things up considerably.

 

[NablaSquaredG]

CX354A does support RoCE, but only v1.

So if you don't need to route RoCE, you're fine with CX3 non-pro.

 

[AllenAsm]

I bought 5 of the connectx-3 pro cards. Can't wait to put this all together. I'll do a write up with pictures when I do the build. Hoping it all just works but doubting it will.

Mellanox ConnectX-3 Pro EN Network Interface Card (MCX314A-BCCT) for sale online | eBay

Find many great new & used options and get the best deals for Mellanox ConnectX-3 Pro EN Network Interface Card (MCX314A-BCCT) at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[mmx01]

Could one overcome ROCE v1 limitations with VXLAN? While you are limited to the same segment, segment is no longer physically restricted.

 

[AllenAsm]

I purchased all of this equipment, including the connectx-3 PRO cards. I will do a writeup of how it goes. This is all going into my personal 42u colocation lab and I'm hopeful its fun. One extra note to this, I'm going to try and put one of the cards into a small business qnap and see if that actually works as well.

 

[alex.forencich]

That's not a stupid question. Transceiver are devices that can transmit and receive signals, be it optical, electrical or other forms and are the electronics in qsfp(+), sfp(+), cxp, xfp modules. So yes, these cables include the transceivers.

To be pedantic, they usually don't. DAC cables are generally copper all the way through, possibly with DC blocking caps on each end. Usually the only silicon in there is an ID EEPROM. No PHY/retimer/equalizer/CDR/etc. chips in most cables, as that would make it an active cable. But what they do have is connectors with the same form factor as an optical transceiver, so they fit in the same slot without any extra hardware.

 

[AllenAsm]

Stupid question time again. Whats the best way to convert sfp+ to 1gbps (short distance, less than 3 meters)? This switch has 4 sfp+ ports along with the 32 qsfp+ ports.

 

[NablaSquaredG]

Depends on what is on the other side. There are SFP+ to RJ45 transceivers, but if you have SFP+ on the other side too you can just use direct attach cables.

 

[AllenAsm]

other side is vanilla 1gbps sonicwall router. I don't see any sfp+ to 1gbps cables for direct attach. :/

 

[NablaSquaredG]

What is "1gbps"? SFP? RJ45 (1GBase-T)?

 

[AllenAsm]

sfp+ to rj45 1000baseT (1 gigabit per second)

 

[alex.forencich]

In general, you cannot convert an SFP+ to 1000BASE-T unless the switch supports running the nominally 10 Gbps SFP+ port at 1 Gbps. So, check the manual and/or try commands on the switch, if you can force the port to run at 1G, then you should be able to use an SFP to 1000BASE-T adapter. The spec sheet says the 7050QX supports up to 4 ports at 1 Gbps, presumably that's in reference to the four SFP+ ports, so I think there is a good chance that it will work.