[solved] Quanta LB6M connection to Mellanox ConnectX-4 /5

[fbcadmin]

We have ordered two Quanta LB6M switches.
These will be used in an existing 7 node proxmox cluster using 65 400GB SSD's .

  data:
    pools:   2 pools, 1088 pgs
    objects: 137.66k objects, 498GiB
    usage:   1.46TiB used, 24.1TiB / 25.5TiB avail

Disk i/o is very low.
Our goal is to set up an excellent ceph network with very low latency. [ not sure if latency is the right word]. we want no lags / delays / and 0 slow requests at ceph.

We are thinking of getting refurbished ConnectX-5 or ConnectX-4 cards from Mellanox.

I assume we'll need SFP28 cabling . [ or is it OK to use SFP+ 10G cable sonce the switch max is 10G?]

Questions:

1- Is there such a thing as standard SFP28 cabled to connect those?

2- Or do we need to get different transceivers for switch and card and use a sfp28 cable? [ or something like that , in learning mode on this subject]

3- suggestions on a different NIC to get ?

thanks for reading and best regards, Roberto.

 

[fbcadmin]

I spoke to Mellanox pre sales support - and standard sfp+ cables will work. So will purchase ConnnectX-5 cards .

 

[Terry Wallace]

Why are you buy x-5’s which support upto 100g networking and connecting then to an lb6m which does 10g ?
Just curious why not something cheaper like the connectx-3 which does 40 and 10 and you can pickup a 40gig switch for not much more than lb6m.
If it’s just future proofing I get that. My 2 cents.

 

[fbcadmin]

Why are you buy x-5’s which support upto 100g networking and connecting then to an lb6m which does 10g ?
Just curious why not something cheaper like the connectx-3 which does 40 and 10 and you can pickup a 40gig switch for not much more than lb6m.
If it’s just future proofing I get that. My 2 cents.

Hello Terry Wallace.

It is for future proofing. I based my decision on the following:

the refurbished X5's are reasonably priced .

the latest Mellanox IB linux driver does not support ConnectX-2 . See release notes at https://www.mellanox.com/page/products_dyn?product_family=27&ssn=evjop174k1jlkgdbjg3midk001
so who knows how long the -3's and -4's would be supported.

Now we are going to use ethernet not IB to start. However I am not sure what we'll do a few years from now.

It was pointed out to me in Proxmox networking forum - that driver is for IB not ethernet. So probably ethernet drivers will always be in the kernel for the entire X series....

We still might go with refurbished -X4's as their store does not have the 8 -X5's we need.

It is so much fun trying to get an order in and get the cards delivered for the weekend . We do have network emergency...

thanks for your reply and have a great Thanksgiving, Rob