[solved] Quanta LB6M connection to Mellanox ConnectX-4 /5

Discussion in 'Networking' started by fbcadmin, Nov 19, 2018.

  1. fbcadmin

    fbcadmin New Member

    Joined:
    Nov 18, 2018
    Messages:
    19
    Likes Received:
    0
    We have ordered two Quanta LB6M switches.
    These will be used in an existing 7 node proxmox cluster using 65 400GB SSD's .
    Code:
      data:
        pools:   2 pools, 1088 pgs
        objects: 137.66k objects, 498GiB
        usage:   1.46TiB used, 24.1TiB / 25.5TiB avail
    
    Disk i/o is very low.
    Our goal is to set up an excellent ceph network with very low latency. [ not sure if latency is the right word]. we want no lags / delays / and 0 slow requests at ceph.

    We are thinking of getting refurbished ConnectX-5 or ConnectX-4 cards from Mellanox.

    I assume we'll need SFP28 cabling . [ or is it OK to use SFP+ 10G cable sonce the switch max is 10G?]

    Questions:

    1- Is there such a thing as standard SFP28 cabled to connect those?

    2- Or do we need to get different transceivers for switch and card and use a sfp28 cable? [ or something like that , in learning mode on this subject]

    3- suggestions on a different NIC to get ?

    thanks for reading and best regards, Roberto.
     
    #1
  2. fbcadmin

    fbcadmin New Member

    Joined:
    Nov 18, 2018
    Messages:
    19
    Likes Received:
    0
    I spoke to Mellanox pre sales support - and standard sfp+ cables will work. So will purchase ConnnectX-5 cards .
     
    #2
  3. Terry Wallace

    Terry Wallace PsyOps SysOp

    Joined:
    Aug 13, 2018
    Messages:
    66
    Likes Received:
    19
    Why are you buy x-5’s which support upto 100g networking and connecting then to an lb6m which does 10g ?
    Just curious why not something cheaper like the connectx-3 which does 40 and 10 and you can pickup a 40gig switch for not much more than lb6m.
    If it’s just future proofing I get that. My 2 cents. :)
     
    #3
  4. fbcadmin

    fbcadmin New Member

    Joined:
    Nov 18, 2018
    Messages:
    19
    Likes Received:
    0
    Hello Terry Wallace.

    It is for future proofing. I based my decision on the following:

    the refurbished X5's are reasonably priced .

    the latest Mellanox IB linux driver does not support ConnectX-2 . See release notes at https://www.mellanox.com/page/products_dyn?product_family=27&ssn=evjop174k1jlkgdbjg3midk001
    so who knows how long the -3's and -4's would be supported.

    Now we are going to use ethernet not IB to start. However I am not sure what we'll do a few years from now.

    It was pointed out to me in Proxmox networking forum - that driver is for IB not ethernet. So probably ethernet drivers will always be in the kernel for the entire X series....

    We still might go with refurbished -X4's as their store does not have the 8 -X5's we need.

    It is so much fun trying to get an order in and get the cards delivered for the weekend . We do have network emergency...

    thanks for your reply and have a great Thanksgiving, Rob
     
    #4
Similar Threads: [solved] Quanta
Forum Title Date
Networking [Solved] Brocade ICX6610-48P PoE Errors Apr 2, 2019
Networking [SOLVED]Mellanox ConnectX 3 can't get 40G only 10G Nov 16, 2018
Networking [Solved] BiDi fiber question Sep 7, 2016
Networking [SOLVED]Slow speeds between two Connectx-2 machines Mar 1, 2016
Networking Quanta LB6M / TurboIron 24x slow transfers Apr 4, 2019

Share This Page