1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Why oh why do I suck at the IB

Discussion in 'Networking' started by whitey, Nov 29, 2016.

  1. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    4,648
    Likes Received:
    822
    #21
  2. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    What can I say, I'm a glutton for punishment :-D

    Just cutting my IB newbie teeth, many thanks to those who have lived these nightmares and gave me a hint or two to get across the finish line.

    EDIT: Ya know what is freaking hilarious...vt-D of these devices work now...I have never believed in firmware so much in my life before...call me a convert. Simple 2.9.1000 to 2.9.1200 on the ConnectX-2 and they are indeed DAMNED working now through vSphere as well. Will go update my other thread in a bit as SOLVED!

    Down to 16Gbps from 21Gbps w/ vt-D/virt layer (25% tax...ouch). Set CPU shares to high, mem locked for dedicated rscs but that L5630 client IB node has only got soo much!

    DOUBLE EDIT: Guess only down to 18Gbps vt-D from 21Gbps previously phys

    'screen' session, sorry if it's hard to read :-(

    Code:
    [root@centos7-ib3 ~]# iperf -c 10.10.10.101
    ------------------------------------------------------------
    Client connecting to 10.10.10.101, TCP port 5001
    TCP window size: 2.50 MByte (default)
    ------------------------------------------------------------
    [  3] local 10.10.10.100 port 52781 connected with 10.10.10.101 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
    [root@centos7-ib3 ~]# iperf -c 10.10.10.101
    ------------------------------------------------------------
    Client connecting to 10.10.10.101, TCP port 5001
    TCP window size: 2.50 MByte (default)
    ------------------------------------------------------------
    [  3] local 10.10.10.100 port 52782 connected with 10.10.10.101 port 5001
    [ ID] Interval       Transfer     Bandwidth
    [  3]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
    [root@centos7-ib3 ~]#
       0 root@centos7-ib3:~
    [root@centos7-ib02 ~]# iperf -s
    ------------------------------------------------------------
    Server listening on TCP port 5001
    TCP window size: 85.3 KByte (default)
    ------------------------------------------------------------
    [  4] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 52781
    [ ID] Interval       Transfer     Bandwidth
    [  4]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
    [  5] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 52782
    [  5]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
    
     
    #22
    Last edited: Nov 30, 2016
  3. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    hm, reboot is always worth a try :D

    good to see it works now
     
    #23
  4. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    278
    Likes Received:
    6
    Hi @whitey, I just got my IS5022 and about to setup for 3 ESXI hosts. Which SM manager did you use and install? I have an ubuntu VM which is on all the time if that helps

    Without any software installation, plugged the cables in and no lights on yet. Is that right?
     
    #24
  5. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Yessir, w/out a SM they won't link. Direct-conn/PTP they will though, I'd start there first just to see them light up (warm fuzzy). I only tested on CentOS7/Linux and it was fairly easy there once I had my 'ah ha' moment, looks like when I want to implement in vSphere it's gonna be a bit trickier as I didn't get link lights on the cards running in vSphere w/ a OpenSM running on my CentOS7 boxes on same subnet as ESXi hosts (mgmt subnet). Assuming that the ESXi hosts want/need to run an instance of SM on each node. I'm gonna attempt/crack this nut soon though...very soon...maybe tonight if I get to feeling froggy!

    Hell, maybe I should reboot LOL
     
    #25
    Last edited: Jan 4, 2017
  6. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    You should be able to use opensm, can be installed by apt and the run the service on at least one node.
     
    #26
  7. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    278
    Likes Received:
    6
    OK. I connected 2 esxi servers to the IS5022, but both the LED port lights are lit orange, not green on the switch. Both What am I doing wrong?
    I have not installed an SM yet. Is that the reason why?
    Both Mellanox connectX cards (MT26428) LED are lt green, but ESXI says link down
     
    #27
  8. Rand__

    Rand__ Active Member

    Joined:
    Mar 6, 2014
    Messages:
    424
    Likes Received:
    46
    SM would be needed for IB at least.
    IPoIB might work without.
    Which ESX version are you on?
    Are the cards present in ESX or passed through?
     
    #28
  9. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
  10. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    no SM, no IB ... guess also IPoIB needs an SM somewhere in the farbic
     
    #30
  11. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    74
    Likes Received:
    34
    Absolutely, IPoIB literally means "IP over Infiniband", no SM->no Infiniband, no Infiniband->no IP...
     
    #31
  12. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    278
    Likes Received:
    6
    I am on esxi 6.0u2, and the mellanox card is present in esxi, not passed through.
    I cannot find in the manual what being lit orange on the LED port lights mean.
    Any suggestion to which SM should I install?
     
    #32
  13. Rand__

    Rand__ Active Member

    Joined:
    Mar 6, 2014
    Messages:
    424
    Likes Received:
    46
    if you are not passing through (ie only have esx) then u only can run the hypervisor.fr opensm variant
    Not tested on 6 but should work (does not work on 6.5)
     
    #33
  14. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    74
    Likes Received:
    34
    For starters, if you are on the stock ESXi image, it's using the built-in drivers that are intended for Ethernet-only mode. Depending on your card/switch and your intended usage of this infrastructure, you might need to do this:
    1. If you have an IB-only switch or want or utilise direct IB connection between ESXi hosts, you have to make sure your cards are in IB mode. Hope your cards are VPI ones, otherwise you're screwed. In the lucky situation your have ConnectX2/3/Pro VPI cards, then you need to decide whether you need SRP initiator (for RDMA access to iSCSI storage) on the ESXi hosts or not.
    If you need SRP, your best choice is to get the 1.8.2.5 Mellanox drivers. I'll post further instructions if this is indeed what you need.
    Otherwise (if you only need IPoIB), you can use the 2.4.0 drivers. Again, I'll post the instructions if this is your case.
    There is currently no solution at all for ConnectX4 and up cards in IB mode for ESXi.
    You'll need an SM in your fabric for IB mode. If you don't have a managed switch, you'll need to run OpenSM on one of the computers connected to your fabric. It can be ESXi host itself. There are materials describing how to do this.
    2. If you have an Ethernet switch or want to utilise direct connection in ETH mode and you have VPI/ETH cards, the built in drivers should work as is. You won't need an SM for this situation.
    The only case you'd need to change something is if you need iSER. Then you'd need to install 1.9.10.5 drivers. These would work only for ConnectX3/Pro, both VPI and EN. For everything else, all you can get is plain IP network.
     
    #34
    T_Minus and britinpdx like this.
  15. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    74
    Likes Received:
    34
    I'd strongly suggest everyone to read this thread on Mellanox communities to understand the current (miserable) situation with Mellanox and ESXi. If you have anything to say, please, do so there. I hope, with additional volume of concerned voices, Mellanox might actually reply and/or do something about this situation.
     
    #35
  16. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    278
    Likes Received:
    6
    Thank you @mpogr for your extensive reply. If you can indulge in me for a few more questions
    1. My mellanox cards come up as Mellanox MT26428, connectX VPI on esxi 6.0u2
    2. I would like to have RDMA access(not a must for now), but unfortunately I already installed the 2.40 drivers because I thought it was the latest and the easiest. I also removed the 3 conflicting drivers from vmware ( nmlx4-core, nmlx4-en and nmlx4-rdma) after installing the 2.40 drivers( or the led light won't even come on). Would I need to change drivers everytime I do an esxi update?
    3. I intend to install this SM in esxi ib-opensm-x64.3.3.15_offline_bundle.zip, is that ok?
    4. I can forget about iSER since I don't have the right cards?
     
    #36
  17. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    74
    Likes Received:
    34
    That's good, you've got ConnectX2 VPI, which is 40Gbps IB/10Gbps ETH.
    That's not an issue at all. Just remove all Mellanox drivers from your system:
    a) Get the list of the modules:
    esxcli software vib list | grep -i mel
    b) Remove them:
    esxcli software vib remove -n module1 -n module2 etc.
    Then reboot and install 1.8.2.5.
    Yes, you would. Just remove the same modules you did the first time and re-install 1.8.2.5 + reboot.
    Yes, but be conscious about having only one SM in your fabric. If your links are host-to-host, you'd need a copy of SM per link (on either side). If you have all your hosts connected to an unmanaged switch, you'd need only one copy running on a host. Make sure it's up and running all the time, a fabric can survive the SM going down for a short brief, but not for long time.
    Strictly speaking, you can install 1.8.3 beta drivers and use iSER with IPoIB, but it has no advantages over SRP. Do it only if you absolutely must.
     
    #37
  18. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    278
    Likes Received:
    6
    Thank you again.
    I am using an unmanaged IS5022 Mellanox switch, so i guess just 1 SM on a host will do.
    I do not see the 1.8.2.5 OFED package on the mellanox site, only the 1.8.2.4.
    Would you have a download link I can use?
     
    #38
  19. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    74
    Likes Received:
    34
    #39
    epicurean likes this.
  20. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    4,648
    Likes Received:
    822
    @mpogr that's some awesome information and instructions!! I have some VPI cards I really need to get figured out for my esxi hosts and a 4036 and 4036E (Ethernet bridge) that I'm not sure how it works for bridging and SM and/or if I need to bridge them or just run separate networks... I'll likely start a thread when I get the time.
     
    #40
Similar Threads: suck
Forum Title Date
Networking Mellanox ConnectX-3 and Ubuntu 16.04 - why does this suck? Jan 11, 2017

Share This Page