Mellanox ConnectX-3 Setup Advice

[Tubz]

Seen a few of these Mellanox ConnectX-3 MCX354A-QCBT 40Gbps InfiniBand Dual Port QSFP High Profile going dirt cheap on ebay UK and was wondering what switch and cabling is recommended to connect 3 servers together and a switch that has a dual RJ45 10G port and provides 1G connections, price is the important bit and looking for simple solution, your talking to a noob on infiniband and I this could be my network solution for a long time.

 

[i386]

First: forget infiniband. You WILL know when you need it

For cabling you could use anything you want (brand wise) as long as it's a qsfp+ or qsfp14 (for 40GBit/s) or sfp+ cable (for 10GBit/s), the connectx-3 cardss are not picky.

For switches: for "pure" 40GBE look for mellanox 6036 it has the lowest power (~35Watt @ 230V) and can be made quiet via commands (quiet doesn't mean silent or no noise, it's a datacenter switch after all...)
If you want to mix 1GBE, 10GBE and 40GBE take a look into the brocade/ruckus icx mega thread, there are a few interesting mdoels listed

 

[BackupProphet]

I agree with i386. But I would like to add. RoCE and SR-IOV on these cards can be painful. Some seems to work fine, others not. For superfast, NVMe-oF or NFSoRDMA I have actually used Infiniband with much better results.

Also driver support is questionable these days, youre best of using Linux inbox drivers which are included in Almalinux or Debian

 

[kapone]

40Gbps InfiniBand

and a switch that has a dual RJ45 10G port and provides 1G connections

??

Those two statements are at odds with each other.

 

[i386]

Those two statements are at odds with each other.

I had to read the op multiple times, they ask for two switches
One for the servers with the cx-3s and one for 1/10gbe via rj45

 

[Tubz]

sorry let me clarify, I already have 2nd switch, a QSW-2104-2T-R2 that provides 1G connections and has 2x10G rj45 for linking to main switch.

Yeh, looks like things are out of my price range for switches.
Starting to think maybe a daisy chain peer-2-peer between the 3 servers and a x550-T2 10G to the QNAP and to my ISPs.
I won't be adding anymore servers, the servers will all be running Proxmox, with 1 server having OpenWRT.

If a cut price feasible solution, recommendations on cards and dacs ?

 

[i386]

Yeh, looks like things are out of my price range for switches.

What is your price range?
I have a few alarms on ebay set up and and a few weeks ago there were many listings for ruckus icx switches with sfp+ and/or qsfp+ ports for fair prices (series 7 switches, < £200)

 

[Blinky 42]

On the Ethernet side, If you want cheap 10G and only have 3 hosts right now one of the small guys like the CRS305 might work for you.
CRS305-1G-4S+IN | MikroTik

Runs warm but silent - I have a few to provide 10g between desktops and at the end of fiber runs between buildings.
Covered by STH in the past https://www.servethehome.com/mikrotik-crs305-1g-4sin-review-4-port-must-have-10gbe-switch/ and the copper version https://www.servethehome.com/mikrotik-crs304-4xg-in-4-port-10gbase-t-switch-launched/

 

[nexox]

If you have 3 hosts and don't expect to expand you can get three dual port 40G NICs and 3 cables and set up a simple ring network with no switch.