SFP+ to QSFP+ switch upgrade, sailing in new waters, need some navigation.

[Havor]

Hi there to all, new to the forum, as the local Dutch forum I go to normally, the Pro network section is housing only crickets nowadays.

I got an upgrade for my 8 years old SFP+ ''D-Link SmartPro DGS-1510-28X'', to a new to me, QSFP+ ''MikroTik CRS326-24S+2Q+RM'' switch, for a real nice price. (€200)
And now I could use some help picking the right NIC and cables.

How is it going be used.
I have a W11Pro 5800X3D machine I use for video editing, and have a file server with a 5950X, ware I also do my video encoding on.
Different tasks needs a different CPU, the 5950X runs with an under clock for a better IPW, and it also frees up my work PC from the heavy video encoding.

What NIC to Pick.
The Mellanox stuff seems to start about a 1/4 of the price of an Intel XL710-QDA, is there a reason for that?
It's not that I am not willing to pay for the right stuff, but saving 200 bucks is always nice. ;-)
And from Mellanox, there seem to be many different models, are their models to be avoided?

I have to run the NIC in a PCIe 3.0 x4 (x16 slot), I take it that would work, as both x8/16 slots are in use. (raid controller card and a dual NVMe card)

What cable to use.
I could use lose transceiver modules, that would run me close to 200 bucks with fiber, but it would be nice if I could use this much cheaper Passive Direct Attach Copper Cable, but that one does not (officially) support MikroTik, but would it work?

My network setup:
Windows 11 Pro work/editing PC.
Windows server 2019 file server en encoding machine.
Offline backup server for nightly automatic backups. (will get a 10Gbit upgrade ;-)
Other regular network stuff, like couple of Shields, laptops, tablets, and Home Assistant

 

[mattventura]

Mellanox is fine, but you won't be able to get the full bandwidth of that card in a Gen3x4 slot - only about 32gb/s theoretical (really lower due to overhead). To get the full bandwidth, you'd need to move the M.2s to onboard slots or chipset lanes so that the NIC can use a x8 slot.

 

[Stephan]

PCIe 3.0 x4 needs to have an open end slot, otherwise x8 card won't fit physically. A ConnectX3 in x4 will only deliver half of 63 GBit/s PCI bandwidth at x8, maybe 32 GBit/s. I see a card from one PCIe slot to another in loopback do ~48 GBit/s on a single machine with CX3 at x8 and using a certified Mellanox FDR cable. Firmware Release Notes has a list of approved cables.

There is a glut of cards because 40G QSFP+ is a dead end in the datacenter, standards call for 10/25/50/100/200/400/800. Personally I like HPE 649281-B21, Oracle 7046442 or Dell 6RKNM. Easy to crossflash. You should upgrade the firmware anyway to latest. Usually I get the Oracle card in Rev. A3 or A4 (there is a sticker on the card) and a large slot bezel from ebay. Cards need airflow. If you have airflow, just stuff CX3 in everything and for the price get a spare.

To connect everything, DAC cables at 40/56G are stiff and limited to a couple meters. 56G is Mellanox FDR (fourteen data rate) if you had a Mellanox switch or link two CX3's then that would make sense. I think 8m max for the 038-900-030-01 EMC cable. If everything is next to each other, DAC looks possible.

Otherwise try 40G XQX2502 KAIAM modules for 8 bucks and some LC/UPC cable. Check STH main page for info about plenum grade fiber cables should you need that. CX3 will eat everything with generic firmware, HPE firmware has some stupid transceiver whitelist. Not sure about your switch.

 

[bitbckt]

Don’t bother with the Intel tax. Buy Mellanox.

Mikrotik switches will work with just about any transceiver or DAC.

 

[Havor]

Mellanox is fine, but you won't be able to get the full bandwidth of that card in a Gen3x4 slot - only about 32gb/s theoretical (really lower due to overhead). To get the full bandwidth, you'd need to move the M.2s to onboard slots or chipset lanes so that the NIC can use a x8 slot.

PCIe 3.0 x4 tops out at 3500MB with NVMe, think it will be the same for a NIC, that's still 3.5x of SFP+, a compromise I can live with.

PCIe 3.0 x4 needs to have an open end slot,

Yeah it's going in to a x16 slot that's x4 wired. (maybe ..... one day, i can upgrade to a TR or EPYC system ;-)

To connect everything, DAC cables at 40/56G are stiff and limited to a couple meters.

The work PC and the server are standing next to one other, only separated by the closet wall, so a 1m and a 2m cable should be just fine.

CX3 will eat everything with generic firmware.

And this NIC would be a good pick, Mellanox MCX314A-BCCT 40Gb Ethernet 40GbE CX314A ConnectX-3 Pro QSFP+ PCIe Card?
But there are many different flavors for sale on eBay for cheep, don't know what's the best pick.

Mikrotik switches will work with just about any transceiver or DAC.

Great, so this cable, Cisco QSFP-H40G-CU1M Compatible 40G QSFP+ Passive Direct Attach Copper Cable should then work just fine.

Don’t bother with the Intel tax. Buy Mellanox.

Good, because I really hate to pay TAXES.

 

[Havor]

After all the great information I got till now, thanks for that.

I am thinking of getting the following parts.

The NIC
Don't need a dual port, so I was thinking of getting this card: MCX313A-BCCT Mellanox ConnectX-3 Pro 40GB Single QSFP+ RDMA Network Card

But I am left with the following questions:

• Is this the best card I should buy, as there are many other alternatives.
• Driver is for Win10, but i take it Win11 Pro should be no problem?

The Cable
I wane use this cable in 1m (3ft) and 2m (7ft) length: 40G QSFP+ Passive Direct Attach Cable

• That's cable should just work?
• Is there a practical difference between passive I picked and active cables, other than length?

Anyway, thanks for the help.

 

[twinkletoes]

Unless you really want to only spend $20 on eBay, I'd get Mellanox CX4 or newer. You can get CX455A for like $50 and flash to CX415A firmware (instead of paying the CX415A rate of around $100.) The CX515A is around $120. CX4 and up uses the mlx5 driver base, that includes CX4, CX5, CX6, CX7, and Bluefield NVIDIA cards. It's supported everywhere. Also CX5 or newer have slightly better features and you can still plug a QSFP+ transceiver into their QSFP28 or QSFP56 ports.