Mellanox 40Gb Cards with QSFP+ Support

[fnc1]

Looking for some PCIe cards that can take QSFP+ modules
From what I've read QSFP+ transceivers will fit and work in QSFP28 ports
I have a Brocade 6610

Brand new to fiber, And have a couple of questions:

1) Did I buy the right thing here?
I have some XQX2502 KAIAM QSFP+40G-LR4 modules on the way.
https://kaiamcorp.com/media/XQX2502 QSFP+ 40G-LR4-Lite.pdf
Looking at the PDF, it says these will do 40GB, but i need duplex single mode fiber: "All of Kaiam’s products allow for superior performance at 40Gb/s and beyond with duplex single mode fiber."
I was planning to buy some MMF fiber cables off amazon. Some of the OM4 Cables (the teal ones) like this https://a.co/d/0kz9k4o
1a) Will these transceivers work in the 6610, and operate at 40G speeds
1b) Would the OM4 fiber cables work with the Above modules, and operate at 40G? Or do i to buy different Fiber Optic cables?

2) There are a lot of options out there for PCIe Cards... but it seems like the Mellanox ones (or variants are the most widely used) and I'd like the requirements to be:
a) Cards have dual ports.
b) Have support for using QASFP+ modules (specifically the ones above)
c) Will work out of the box, or could be cross flashed to support 40G
2a) Will the MCX354A-QCBT cards work for what i want to accomplish? Garland computers have a few available: Mellanox ConnectX-3 CX354A VPI Dual QSFP Port Adapter MCX354A-QCBT Half Height – Garland Computers
2b) Can these dual port cards utilize a QSFP+ module in both, i sometimes see one labeled Infiniband, and the other labeled something else.

I tried using this page, but there seem to be more options that im finding on ebay than listed here:

https://forums.servethehome.com/index.php?resources/mellanox-nics.47/

Some ref. links
CX354A Manual https://network.nvidia.com/pdf/user...d_Dual_QSFP_Port_Adapter_Card_User_Manual.pdf

 

[Scott Laird]

All 40 GbE is QSFP+[1]. OM4 is multimode fiber, according to the datasheet, this wants single-mode fiber. So look for OS2 or similar. It'll basically all be yellow, while OM4 is usually pinkish and OM3 is usually teal. There isn't really that much difference between OM3 and OM4.

I can't remember if Brocade switches care about which optics they use; a lot of switches only want to work with their own manufacturer's own optics. It's usually easy enough to buy new optics that are coded to be compatible with any given vendor (see 40G QSFP+ Transceiver Modules | Optical Transceivers - FS.com, for example).

If you're looking at less than ~5 meters or so, it'll almost always be cheaper to skip the optics entirely and get a DAC cable that connects the two QSFPs together without the expensive optics. For example, fs.com has 0.5m 40G DAC cables for $26, vs $39 for the cheapest 40G optics. Also, a lot of 40G optics actually want MPO/MPT fibers (with 4 pairs of fiber, instead of the 1 pair in duplex LC connectors); those end up being a lot more expensive than the LC equivalent.

On the card side, I don't think there's really a ton of difference between the ConnectX-3 cards, assuming that they have the same connectors. Generally, Mellanox made 3 major versions of their cards -- an Ethernet version, and Infiniband, and a VPI (either Ethernet or Infiniband) version. I've mostly ended up with VPI cards, but anything that'll support 40GbE is probably fine.

Be aware that ConnectX-3 is getting a bit long in the tooth, and driver support is getting a bit trickier. ConnectX-4 and above have better support and are a bit faster, but probably not enough to really matter for any normal use.

*Almost* everything that can do 100GbE via QSFP28 is also capable of 40GbE operation[2]. The only notable exception that I've seen is Intel's E8xx-series Ethernet cards. Weirdly, it supports 4x10G, 4x25G, and 1x100G but not 1x40G.


[1] *Technically* there's a spec for 40gbase-T; to the best of my knowledge, no hardware for it has ever been sold commercially.
[2] We're starting to see SFP112-based 100GbE NICs, with a single 100Gbps link, and there are probably SFP-DD56 devices out there *someplace* that can do 2x50 Gbps, but neither is common. Generally today, 100 GbE == QSFP28, although there are variants that are 10x10G, 4x25G, 2x50G, and 1x100G.

 

[klui]

The Kaiam transceivers will work on Brocades. These LR4-Ls are designed to be used with single mode fiber, not multi mode. But they will actually link using short (< 3 m) multi mode cables. How long they will work or under what tolerances/circumstances, I don't know. I just tried linking up to a CX3 but not passing any traffic. The specification of these modules only allow less than 10 nm variance. My guess is if you have a good length of cable, reflection could potentially cause a lambda (channel wavelength) to be received out of spec so your link will be lost--kinda like what happened to a recent member who had a channel on their transceiver die.

If you want to use modules spec'ed for multi mode, look for universal modules from Arista or Juniper. They allow the use of multi mode to reach 100-150 m or single mode to reach 500 m. Except for Cisco BiDi, these UNIV transceivers are compatible with LR4 modules on the other end, and I would presume LR4-L, too. Arista has great FAQs about their 40G and 40G UNIV transceivers.

Quite a few OM4s are also aqua in color. Just look at the part's datasheet. I would not buy any cables that don't have a datasheet.

EDIT: for some really interesting reading look up Arista's 100, 200, 400, and 800G FAQs

 

[fnc1]

Thanks for the replies - The XQX2502 KAIAM QSFP+40G-LR4's should be be here tomorrow, along with a couple
"10M OS2 LC to LC Fiber Patch Cable, Single Mode SFP Fiber Jumper, Duplex LC-LC 9/125um, LSZH, 33ft, for 1G/10G SMF SFP Transceiver, Router, Fiber Networks and More" fiber cables specifically these: https://a.co/d/bNr1oko

Received my Brocade 10gb SFP+ today, and will have cables for them tomorrow as well "10m(33ft) OM3 LC to LC Fiber Patch Cable,10GB/40GB Multimode Duplex Fiber Cable 50/125µm 2.0mm Fiber Optic Cable Length Option: 0.5M(1.6ft)-100M(328ft)" from here: https://a.co/d/7uTIzPq

And should have a couple Mellanox CX354A Dual-Port 10GbE SFP PCIe Network Adapter P/N: MCX354A-QCBT cards showing up early next week, and then should be all set to test things out.
Guess all ill need to do is flash these to support 40gb like such: https://forums.servethehome.com/ind...rnet-dual-port-qsfp-adapter.20525/post-198015

Anything to watch out for here?

 

[fnc1]

As of feb. 7 with Ubuntu 24.04 LTS iso

sudo su
#! /bin/sh
apt update && apt install gcc make dkms unzip linux-headers-$(uname -r)
# NVIDIA Firmware Tools (MFT) (use an archive version 4.30.1-8 LTS - 4.26.1 no longer supports cx3 NVIDIA Firmware Tools (MFT))
wget https://www.mellanox.com/downloads/MFT/mft-4.25.1-11-x86_64-deb.tgz

tar -xvf mft-4.25.1-11-x86_64-deb.tgz
./mft-4.25.1-11-x86_64-deb/install.sh

#Start MST service:
mst start
mst status
#copy the dev address with cr0 in it,
#/dev/mst/mt4099_pci_cr0
#use this in the flint / flash command

#download firmware (for MCX354A-FCBT else Firmware for ConnectX®-3 IB/VPI)
wget http://www.mellanox.com/downloads/f...00-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip
unzip fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin.zip

#backup current config
flint -d /dev/mst/mt4099_pci_cr0 dc orig_firmware.ini

#Flash it
flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX354A-FCB_A2-A5-FlexBoot-3.4.752.bin -allow_psid_change burn

#copy the new config just for fun
flint -d /dev/mst/mt4099_pci_cr0 dc new_firmware.ini

#set both ports to ethernet only
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

#turn off bootrom crap
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P1=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set BOOT_OPTION_ROM_EN_P2=false
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P1=0
mlxconfig -d /dev/mst/mt4099_pci_cr0 set LEGACY_BOOT_PROTOCOL_P2=0


##optional: delete bootrom off the card, so it doesn't slow down boot by popping up crap
##this is safe to do and supported by mellanox
flint -d /dev/mst/mt4099_pci_cr0 --allow_rom_change drom

 

[fnc1]

And to get it workiing on Proxmox8 (not entirely sure i needed to this, but its working)

apt-get install gcc make dkms pve-headers
wget https://www.mellanox.com/downloads/MFT/mft-4.22.1-417-x86_64-deb.tgz
tar -xzvf mft-4.22.1-417-x86_64-deb.tgz
./tar -xzvf mft-4.22.1-417-x86_64-deb/install.sh
mst start
mst status

 

[Scott Laird]

You *probably* don't need most of this. With Ubuntu (and likely Debian) you can `apt install mstflint` and get slightly differently named versions of most of Mellanox's tools. For example, mstconfig instead of mlxconfig. You don't need any of this to *use* the NICs, but they can be helpful when you need to change low-level configs in ways that `ethtool` doesn't support (like flipping between IB and Ethernet modes) or updating firmware.