1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Why oh why do I suck at the IB

Discussion in 'Networking' started by whitey, Nov 29, 2016.

  1. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Morning fellow STHers, dying here w/ this IB switch issue buggin' me. Try to make this short and sweet. I have an IS5022 IB switch that I cannot for the life of me get to link up any lights on switch of HCA's. If I direct conn/crossover QSFP to each end of HCA's (ConnectX-2's) they link and opensm is happy.

    Got quite the novel going on over here if you are interested in backlog/how this all started. That was posted in Virt/vSphere forum due to me wanting to do this vt-D but ran into a brick wall there, this is a more generic IB networking issue at this point so I figured I'd post here to see if it get's any more intelligent eyes over it. TIA and sorry for sharting in two threads.

    Mellanox unhappy w/ vtD passthru in vSphere

    HELP! Am I being a tard monkey, is this switch bad (hoping not, no red lights but I cannot get any console output either on typical 9600,8,n,1 config)?

    Is there a IB noobie guide for next steps to bring some sanity to my madness? Sure would like a working IB fabric through the switch. I could flood us w/ screenshots but hoping you all know the trick.

    TIA, whitey

    EDIT: Do I need explicit opensm.conf GUID entries to simply make HCA's link light to switch, w/ direct-connect they quickly w/in 3-5 seconds go into master/standby state (running opensm on each node for sledgehammer sake)
     
    #1
    Last edited: Nov 30, 2016
  2. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Dang no one has a tid bit of GOLD info :-(

    Bummed
     
    #2
  3. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    hm, i had abslutely no issue with my 5022, connected two nodes with qsfp cables and had a running fabric. really just turn on, connect, wait a bit and go-on.

    didn't even see a need to access that switch on the console/mgmt-port as there is nothing/not much to manage. but could try this tomorrow if i have the right cable somewhere and see if i get something.
     
    #3
  4. Drewy

    Drewy Member

    Joined:
    Apr 23, 2016
    Messages:
    80
    Likes Received:
    7
    I'm no expert, I quickly grew tired of infiniband and dumped it for 10gb ethernet.
    Are you ensuring you are only running one SM per fabric?
     
    #4
  5. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    I have two opensm, one running on each host, sm should work in master/standby mode, I do see them coming up in this manner when direct connected QSFP cable between two phys hosts and two HCA's. Is a opensm.conf or opensm file needed in /etc/sysconfig, RH docs mention this, I did set PORTS="ALL" in there and bounced the svc...no luv
     
    #5
    Last edited: Nov 30, 2016
  6. Drewy

    Drewy Member

    Joined:
    Apr 23, 2016
    Messages:
    80
    Likes Received:
    7
    Don't you also have a subnet manager running on the switch?
     
    #6
  7. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    No the Mellanox IS5022 is a unmanaged IB switch w/out a SM on it. Must run one (opensm) in SW on the IB fabric/topology.

    What's funny is w/out ANY opensm.conf file or /etc/sysconfig/opensm configuration if i go direct-conn it 'JUST WORKS' and the SM managers come up in a master UP, standby UP state.
     
    #7
  8. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    That's how i figured it 'should' be...easy peasy, nothing fancy. Pissing me off right abt now, I dunno if i have a bad IB switch or what, no red/amber lights, all looks good, read the IS5022 User guide end to end several times. Exhausted my google-fu :-(
     
    #8
  9. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    This is the output from both hosts while connected to the switch w/ a tail -f /var/log/opensm.log

    BROKE
    centos7-ib01
    Code:
    Nov 30 22:09:08 398741 [FF6B1740] 0x80 -> Exiting SM
    Nov 30 22:09:09 430174 [31756740] 0x03 -> OpenSM 4.8.0.MLNX20160906.32a95b6
    OpenSM 4.8.0.MLNX20160906.32a95b6
    
    Nov 30 22:09:09 430244 [31756740] 0x80 -> OpenSM 4.8.0.MLNX20160906.32a95b6
    Using default GUID 0x2c903004af2b3
    Entering DISCOVERING state
    
    Nov 30 22:09:09 431491 [31756740] 0x02 -> osm_vendor_init: 1000 pending umads specified
    Nov 30 22:09:09 431571 [31756740] 0x02 -> osm_vendor_init: 1000 pending umads specified
    Nov 30 22:09:09 443791 [31756740] 0x80 -> Entering DISCOVERING state
    SM port is down
    
    Nov 30 22:09:09 443992 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x81 binding to port GUID 0x2c903004af2b3
    Nov 30 22:09:09 472209 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x03 binding to port GUID 0x2c903004af2b3
    Nov 30 22:09:09 499567 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x04 binding to port GUID 0x2c903004af2b3
    Nov 30 22:09:09 499648 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x21 binding to port GUID 0x2c903004af2b3
    Nov 30 22:09:09 499725 [31756740] 0x02 -> osm_opensm_bind: Setting IS_SM on port 0x0002c903004af2b3
    Nov 30 22:09:09 500766 [25AD3700] 0x80 -> SM port is down
    
    centos7-ib04
    Code:
    Nov 30 22:09:03 670059 [27D3740] 0x80 -> Exiting SM
    Nov 30 22:09:04 714856 [EB761740] 0x03 -> OpenSM 4.8.0.MLNX20160906.32a95b6
    OpenSM 4.8.0.MLNX20160906.32a95b6
    
    Nov 30 22:09:04 714948 [EB761740] 0x80 -> OpenSM 4.8.0.MLNX20160906.32a95b6
    Using default GUID 0x2c903004af2ab
    Entering DISCOVERING state
    
    Nov 30 22:09:04 717488 [EB761740] 0x02 -> osm_vendor_init: 1000 pending umads specified
    Nov 30 22:09:04 717587 [EB761740] 0x02 -> osm_vendor_init: 1000 pending umads specified
    Nov 30 22:09:04 730625 [EB761740] 0x80 -> Entering DISCOVERING state
    SM port is down
    
    Nov 30 22:09:04 730872 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x81 binding to port GUID 0x2c903004af2ab
    Nov 30 22:09:04 760951 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x03 binding to port GUID 0x2c903004af2ab
    Nov 30 22:09:04 790649 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x04 binding to port GUID 0x2c903004af2ab
    Nov 30 22:09:04 790737 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x21 binding to port GUID 0x2c903004af2ab
    Nov 30 22:09:04 790820 [EB761740] 0x02 -> osm_opensm_bind: Setting IS_SM on port 0x0002c903004af2ab
    Nov 30 22:09:04 792106 [D7ACE700] 0x80 -> SM port is down
    
    WORKING (via direct connect QSFP between HCA's, default opensm config on each node)
    centos7-ib01
    Code:
    SM port is up
    
    Nov 30 22:12:29 448906 [25AD3700] 0x80 -> SM port is up
    Entering STANDBY state
    
    Nov 30 22:12:29 454259 [25AD3700] 0x80 -> Entering STANDBY state
    
    centos7-ib04
    Code:
    SM port is up
    
    Nov 30 22:12:44 735228 [D7ACE700] 0x80 -> SM port is up
    Entering MASTER state
    
    Nov 30 22:12:44 738546 [D7ACE700] 0x80 -> Entering MASTER state
    Nov 30 22:12:44 743690 [D7ACE700] 0x02 -> SUBNET UP
    Nov 30 22:12:44 744914 [E7CEE700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:1 TID:0x0000000000000014
    Nov 30 22:12:44 745023 [E7CEE700] 0x02 -> trap_rcv_process_request: Trap 144 Node description update
    Nov 30 22:12:44 745068 [E7CEE700] 0x02 -> log_notice: Reporting Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) from LID:1 GID:fe80::2:c903:4a:f2b3
    Nov 30 22:12:44 745925 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:401b:ffff::ffff:ffff
    Nov 30 22:12:44 747333 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:401b:ffff::1
    Nov 30 22:12:44 748203 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::1
    Nov 30 22:12:44 748767 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::1:ff4a:f2b3
    Nov 30 22:12:44 749018 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:67 (Mcast group deleted) from LID:2 GID:ff12:601b:ffff::1:ff4a:f2b3
    Nov 30 22:12:44 753135 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::1:ff4a:f2b3
    Nov 30 22:12:44 754347 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::16
    Nov 30 22:12:44 781415 [E64EB700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:2 TID:0x000000000000000a
    
     
    #9
  10. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    hm, what does ibstatus / ibportstate say about the ports, anything in dmesg / /vat/log/messages ?
    how are the lights on the hca when connecting to the Switch?
    guess thes should blink for a while and then become green after the link is up, could check this tomorrow at work ...

    and, just to exclude bad cable/port: have you checked that both cables work on Host to Host Connection and tried differrent ports on the Switch ?
     
    #10
  11. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    I also just forced a SM master/standby switchover by issuing a 'systemctl stop opensm' on master SM while running a loop of iperf tests and the tests continued/interfaces stayed lit up and transmitting while the switchover from master/standby role swapped...took abt 30 secs so I have high confidence that my opensm config is working but HELL maybe there is something has HAS to be configured/set in a file when you connect them to an unmanaged IB switch.

    Remember this is ALL and ONLY working while direct connect from host-to-host HCA's, I cannot get ANY link at all on switchport/HCA when conned to switch although two top lights are on IS5022 are green (I believe good overall system status). I will hook back up to switch now that I have demo'ed a few observations here.

    Code:
    Nov 30 22:12:29 454259 [25AD3700] 0x80 -> Entering STANDBY state
    Nov 30 22:27:49 469277 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                            SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12c7
                            Initial path: 0,1 Return path: 0,1
    Nov 30 22:27:59 469446 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                            SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12c8
                            Initial path: 0,1 Return path: 0,1
    Nov 30 22:28:09 469631 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                            SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12c9
                            Initial path: 0,1 Return path: 0,1
    Nov 30 22:28:19 469757 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                            SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12ca
                            Initial path: 0,1 Return path: 0,1
    Nov 30 22:28:29 469836 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                            SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12cb
                            Initial path: 0,1 Return path: 0,1
    Entering DISCOVERING state
    
    Nov 30 22:28:39 471089 [2FCE7700] 0x80 -> Entering DISCOVERING state
    Entering MASTER state
    
    Nov 30 22:28:39 474687 [25AD3700] 0x80 -> Entering MASTER state
    Nov 30 22:28:39 479102 [25AD3700] 0x02 -> SUBNET UP
    Nov 30 22:28:39 975080 [2ECE5700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:2 TID:0x000000000000000b
    Nov 30 22:28:39 975143 [2ECE5700] 0x02 -> trap_rcv_process_request: Trap 144 Node description update
    Nov 30 22:28:39 975165 [2ECE5700] 0x02 -> log_notice: Reporting Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) from LID:2 GID:fe80::2:c903:4a:f2ab
    Nov 30 22:28:40 479662 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:401b:ffff::ffff:ffff
    Nov 30 22:28:40 481310 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:401b:ffff::1
    Nov 30 22:28:40 481983 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:601b:ffff::1
    Nov 30 22:28:40 482639 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:601b:ffff::1:ff4a:f2b3
    Nov 30 22:28:40 482991 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:601b:ffff::1:ff4a:f2ab
    Nov 30 22:28:49 464709 [25AD3700] 0x02 -> SUBNET UP
    Nov 30 22:28:59 464924 [25AD3700] 0x02 -> SUBNET UP
    Nov 30 22:29:09 465141 [25AD3700] 0x02 -> SUBNET UP
    Nov 30 22:29:12 021824 [2CCE1700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:2 TID:0x000000000000000c
    
     
    #11
  12. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    i wouldn't care about the sm at all until you get a link on the Switch. it runs fine without any config-file as long as there is at least one running on the fabric.

    i suspect a more physical problem as i guess there is no link.
     
    #12
    whitey likes this.
  13. Drewy

    Drewy Member

    Joined:
    Apr 23, 2016
    Messages:
    80
    Likes Received:
    7
    Does the switch log anything? Any indication that it likes your cables...
     
    #13
  14. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    drewy: this Switch does nothing, i would call it more a Hub than a Switch ;)
     
    #14
  15. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Nothing out of the ordinary in /var/log/messages, looks clean, typical mlx stuff, I'd like to think that since host to host direct works that the OS/OFED drivers/HCA's are good seeings how I can shoot 20+Gbps over them in that fashion.

    No lights at all (not even a quick blink) on switch of HCA that I can see, SM reports HCA interfaces as down always when conn'ed to IS5022.

    Next question is a resounding YES

    Tried all 8 ports on IS5022, nothing different :-( ebay seller (also a member here) says it all checked out and is in working order.
     
    #15
  16. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Both of my two QSFP copper cables work when direct-conn between phys hosts HCA's :-( Don't think it is a bad cable, smashing data @ 21Gbps host-to-host seems to eliminate bad HCA's. All signs seem to point back to switch but the top two indicators are green lit, RTFMing seems to lead me to believe it 'should' be fat and happy like this. WTF HAH

    This is the IS5022 guide I found, Status LED section (clickable) is what mine looks like w/ top two green ports (Status LED/Fan Indicators) lit.

    http://www.mellanox.com/related-docs/user_manuals/IS5022_User_Manual.pdf

    Can't seem to find a 'reset to factory default' reset button like I think there is on some of the other IS50xx series switches (bigger brothers) so I'll assume it doesn't exist. Sure would be nice to have that simple serial to rj45 mgmt cable work like it does on all my other gear to at least see the damn thing boot/initialize...kinda a black box at this point. I think I saw a wonky I2C kit to 'actually' get output from these...garbage...just what I need another mgmt cable.

    I did update the HCA FW to 2.9.1200 last night from 2.9.1000 successfully.
     
    #16
    Last edited: Nov 30, 2016
  17. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    350
    Likes Received:
    29
    ok, Sorry for the cables / ports question lol

    no lights on the adapters sounds wrong, i can't remember the exact colors, but you should see there if physical link comes up.

    also in ibstat you should be able to see status and physicsl state, where the First is related to the sm/fabrics status of the port ...
     
    #17
  18. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Here is the output of ibstat when connected to switch, look ugly/way different (not good) than when direct-conn....umm well it did before a reboot of both hosts/switch/restart of opensm...WTF

    Code:
    [root@centos7-ib01 ~]# ibstat
    CA 'mlx4_0'
            CA type: MT26428
            Number of ports: 1
            Firmware version: 2.9.1200
            Hardware version: b0
            Node GUID: 0x0002c903004af2b2
            System image GUID: 0x0002c903004af2b5
            Port 1:
                    State: Active
                    Physical state: LinkUp
                    Rate: 40
                    Base lid: 1
                    LMC: 0
                    SM lid: 2
                    Capability mask: 0x0251086a
                    Port GUID: 0x0002c903004af2b3
                    Link layer: InfiniBand
    [root@centos7-ib01 ~]#
    
    Code:
    [root@centos7-ib04 ~]# ibstat
    CA 'mlx4_0'
            CA type: MT26428
            Number of ports: 1
            Firmware version: 2.9.1200
            Hardware version: b0
            Node GUID: 0x0002c903004af2aa
            System image GUID: 0x0002c903004af2ad
            Port 1:
                    State: Active
                    Physical state: LinkUp
                    Rate: 40
                    Base lid: 2
                    LMC: 0
                    SM lid: 2
                    Capability mask: 0x0251086a
                    Port GUID: 0x0002c903004af2ab
                    Link layer: InfiniBand
    [root@centos7-ib04 ~]#
    
    I am LOLing here, that was insane, I just noticed it all looks happy now after a hail Mary reboot of everything. Well that doesn't exactly leave a warm fuzzy but I will take it I guess. Talk abt a WTF moment in the IT trenches. Sorry for the hassle guys, you all rock for jumping in!

    EDIT: Still iperfing between both phys hosts now @ 21Gbps (same as host-to-host direct QSFP connect)...a good sign (client CPU limited, ohh btw is iperf single threaded anyone know?)

    So I am CONNECTED AND WORKING through the IS5022 (WOOHOOO). Links look good on switch now, nice and blinky green...ohh the insanity, a freaking reboot of everything fixed this. I am thoroughly perplexed and semi amused...semi terrified!
     
    #18
    Last edited: Nov 30, 2016
  19. epicurean

    epicurean Member

    Joined:
    Sep 29, 2014
    Messages:
    279
    Likes Received:
    6
    I ordered an IS5022 which is on its way, and I must say I read this thread with a lot of trepidation. Hopefully, I have an easier time than you did @whitey .
     
    #19
  20. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,002
    Likes Received:
    605
    Worst case you know to pull a Windows move :-D

    I must admit I love that is only consumes 40-60 watts and it doesn't raise my home lab to jet engine noises. Hopefully it's a ROCK from here on out. I am just as surprised I had this many challenges to get a simple 2-node IB fabric/communications going but the pain is over now and many a lessons learned.
     
    #20
    _alex and T_Minus like this.
Similar Threads: suck
Forum Title Date
Networking Mellanox ConnectX-3 and Ubuntu 16.04 - why does this suck? Jan 11, 2017

Share This Page