qsfp+ switch seems too good to be true

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AllenAsm

Member
Jul 30, 2018
55
3
8
Thank you, that's really clear and makes a lot of sense. The DAC appears is all I need for this.

So the last piece i need then is the network cards. Would this card work with all of this equipment so far?


This is all for my personal lab and has hyperv and s2d everywhere. My hope is/was that we could use RDMA or ROCE working on these to speed things up even more.
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
Too expensive ;-)

You should look for OEM CX354A (QCBT), they can be had for 20-25$ and easily be crossflashed to FCBT firmware that gives you 56G Infiniband and Ethernet (40G Ethernet works too, of course)

Re RoCE:
Keep in mind that ConnectX-3 NON-Pro only support RoCE v1. Only ConnectX-3 PRO and upwards support RoCE v2.
 
  • Like
Reactions: AllenAsm

AllenAsm

Member
Jul 30, 2018
55
3
8
does anyone know if the connectx-3 cards do qsfp+ or is it an infiniband protocol thats going to be flowing over everything?
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
so does the cx354a cross flashed support rdma or roce?
Crossflashed cards are identical to genuine cards and have all features you'd also get from a genuine card. So yes, RDMA and RoCE would work on a crossflashed CX354A

does anyone know if the connectx-3 cards do qsfp+ or is it an infiniband protocol thats going to be flowing over everything?
Sorry, I can't follow...

QSFP+ is just the connector and supports different protocols (Infiniband, Ethernet, SAS, etc...) with the appropriate cables / optical transceivers.
Mellanox CX-3 support Ethernet and Infiniband.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
QSFP+ is just the connector and supports different protocols (Infiniband, Ethernet, SAS, etc...) with the appropriate cables / optical transceivers.
Mellanox CX-3 support Ethernet and Infiniband.
This is the part I didn't understand, thank you for clarifying! So with that said, will these connectx-3 cards be using ethernet or inifiband protocol and/or does it matter?

Also, it just occurred to me that an 8x pci only has 8gb/s bandwidth so how is it possible to even use qsfp+ 40gbe with that?
 

i386

Well-Known Member
Mar 18, 2016
4,221
1,540
113
34
Germany
Also, it just occurred to me that an 8x pci only has 8gb/s bandwidth so how is it possible to even use qsfp+ 40gbe with that?
Thesese are different units...
40GBE = 40 Gigabit/second Ethernet (~3.9 Gigabyte/second)
x8 PCI ~8Gigabyte/second
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
Yes, it does matter ;-)
RDMA is easier to set up with Infiniband and Infiniband also supports IPoIB. But IPoIB lacks the hardware acceleration, so you won't get full throughput if your application uses Ethernet, but you set up an Infiniband link.

CX3 has three modes per Port: Auto (IB or Eth), IB or Eth. Auto tries to negotiate IB and falls back to IB.

PCIe 3.0 x8 has ~7.887GByte (without overhead) which is more than 40GBit (40Gbe = 40Gbit, not 40GByte)
 

AllenAsm

Member
Jul 30, 2018
55
3
8
I think I'm going to have to use ETH even with losing some of the throughput. I read up on hyper-v and it says "Hyper-V Virtual Switch only supports Ethernet, and does not support any other wired local area network (LAN) technologies, such as Infiniband and Fibre Channel".

I bought the cables and the switch so the cards are the last thing I need. I'd love to do the CX354A with cross flash but I'm not sure I will have time to really do the work on those and get them flashed up so I'm leaning towards the connectx-3 pro cards. RDMA is crucial for this build as the s2d/hyperv cluster docs say it will speed things up considerably.
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
CX354A does support RoCE, but only v1.

So if you don't need to route RoCE, you're fine with CX3 non-pro.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
I bought 5 of the connectx-3 pro cards. Can't wait to put this all together. I'll do a write up with pictures when I do the build. Hoping it all just works but doubting it will. :)

 
Last edited:

mmx01

Member
Jan 17, 2020
38
20
18
Could one overcome ROCE v1 limitations with VXLAN? While you are limited to the same segment, segment is no longer physically restricted.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
I purchased all of this equipment, including the connectx-3 PRO cards. I will do a writeup of how it goes. This is all going into my personal 42u colocation lab and I'm hopeful its fun. One extra note to this, I'm going to try and put one of the cards into a small business qnap and see if that actually works as well.
 
Jul 19, 2020
51
19
8
That's not a stupid question. Transceiver are devices that can transmit and receive signals, be it optical, electrical or other forms and are the electronics in qsfp(+), sfp(+), cxp, xfp modules. So yes, these cables include the transceivers.
To be pedantic, they usually don't. DAC cables are generally copper all the way through, possibly with DC blocking caps on each end. Usually the only silicon in there is an ID EEPROM. No PHY/retimer/equalizer/CDR/etc. chips in most cables, as that would make it an active cable. But what they do have is connectors with the same form factor as an optical transceiver, so they fit in the same slot without any extra hardware.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
Stupid question time again. Whats the best way to convert sfp+ to 1gbps (short distance, less than 3 meters)? This switch has 4 sfp+ ports along with the 32 qsfp+ ports.
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,320
800
113
Depends on what is on the other side. There are SFP+ to RJ45 transceivers, but if you have SFP+ on the other side too you can just use direct attach cables.
 

AllenAsm

Member
Jul 30, 2018
55
3
8
other side is vanilla 1gbps sonicwall router. I don't see any sfp+ to 1gbps cables for direct attach. :/
 
Jul 19, 2020
51
19
8
In general, you cannot convert an SFP+ to 1000BASE-T unless the switch supports running the nominally 10 Gbps SFP+ port at 1 Gbps. So, check the manual and/or try commands on the switch, if you can force the port to run at 1G, then you should be able to use an SFP to 1000BASE-T adapter. The spec sheet says the 7050QX supports up to 4 ports at 1 Gbps, presumably that's in reference to the four SFP+ ports, so I think there is a good chance that it will work.
 
  • Like
Reactions: AllenAsm