Why oh why do I suck at the IB

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Morning fellow STHers, dying here w/ this IB switch issue buggin' me. Try to make this short and sweet. I have an IS5022 IB switch that I cannot for the life of me get to link up any lights on switch of HCA's. If I direct conn/crossover QSFP to each end of HCA's (ConnectX-2's) they link and opensm is happy.

Got quite the novel going on over here if you are interested in backlog/how this all started. That was posted in Virt/vSphere forum due to me wanting to do this vt-D but ran into a brick wall there, this is a more generic IB networking issue at this point so I figured I'd post here to see if it get's any more intelligent eyes over it. TIA and sorry for sharting in two threads.

Mellanox unhappy w/ vtD passthru in vSphere

HELP! Am I being a tard monkey, is this switch bad (hoping not, no red lights but I cannot get any console output either on typical 9600,8,n,1 config)?

Is there a IB noobie guide for next steps to bring some sanity to my madness? Sure would like a working IB fabric through the switch. I could flood us w/ screenshots but hoping you all know the trick.

TIA, whitey

EDIT: Do I need explicit opensm.conf GUID entries to simply make HCA's link light to switch, w/ direct-connect they quickly w/in 3-5 seconds go into master/standby state (running opensm on each node for sledgehammer sake)
 
Last edited:

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
hm, i had abslutely no issue with my 5022, connected two nodes with qsfp cables and had a running fabric. really just turn on, connect, wait a bit and go-on.

didn't even see a need to access that switch on the console/mgmt-port as there is nothing/not much to manage. but could try this tomorrow if i have the right cable somewhere and see if i get something.
 

Drewy

Active Member
Apr 23, 2016
208
56
28
54
I'm no expert, I quickly grew tired of infiniband and dumped it for 10gb ethernet.
Are you ensuring you are only running one SM per fabric?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I have two opensm, one running on each host, sm should work in master/standby mode, I do see them coming up in this manner when direct connected QSFP cable between two phys hosts and two HCA's. Is a opensm.conf or opensm file needed in /etc/sysconfig, RH docs mention this, I did set PORTS="ALL" in there and bounced the svc...no luv
 
Last edited:

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
No the Mellanox IS5022 is a unmanaged IB switch w/out a SM on it. Must run one (opensm) in SW on the IB fabric/topology.

What's funny is w/out ANY opensm.conf file or /etc/sysconfig/opensm configuration if i go direct-conn it 'JUST WORKS' and the SM managers come up in a master UP, standby UP state.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
hm, i had abslutely no issue with my 5022, connected two nodes with qsfp cables and had a running fabric. really just turn on, connect, wait a bit and go-on.

didn't even see a need to access that switch on the console/mgmt-port as there is nothing/not much to manage. but could try this tomorrow if i have the right cable somewhere and see if i get something.
That's how i figured it 'should' be...easy peasy, nothing fancy. Pissing me off right abt now, I dunno if i have a bad IB switch or what, no red/amber lights, all looks good, read the IS5022 User guide end to end several times. Exhausted my google-fu :-(
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
This is the output from both hosts while connected to the switch w/ a tail -f /var/log/opensm.log

BROKE
centos7-ib01
Code:
Nov 30 22:09:08 398741 [FF6B1740] 0x80 -> Exiting SM
Nov 30 22:09:09 430174 [31756740] 0x03 -> OpenSM 4.8.0.MLNX20160906.32a95b6
OpenSM 4.8.0.MLNX20160906.32a95b6

Nov 30 22:09:09 430244 [31756740] 0x80 -> OpenSM 4.8.0.MLNX20160906.32a95b6
Using default GUID 0x2c903004af2b3
Entering DISCOVERING state

Nov 30 22:09:09 431491 [31756740] 0x02 -> osm_vendor_init: 1000 pending umads specified
Nov 30 22:09:09 431571 [31756740] 0x02 -> osm_vendor_init: 1000 pending umads specified
Nov 30 22:09:09 443791 [31756740] 0x80 -> Entering DISCOVERING state
SM port is down

Nov 30 22:09:09 443992 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x81 binding to port GUID 0x2c903004af2b3
Nov 30 22:09:09 472209 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x03 binding to port GUID 0x2c903004af2b3
Nov 30 22:09:09 499567 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x04 binding to port GUID 0x2c903004af2b3
Nov 30 22:09:09 499648 [31756740] 0x02 -> osm_vendor_bind: Mgmt class 0x21 binding to port GUID 0x2c903004af2b3
Nov 30 22:09:09 499725 [31756740] 0x02 -> osm_opensm_bind: Setting IS_SM on port 0x0002c903004af2b3
Nov 30 22:09:09 500766 [25AD3700] 0x80 -> SM port is down
centos7-ib04
Code:
Nov 30 22:09:03 670059 [27D3740] 0x80 -> Exiting SM
Nov 30 22:09:04 714856 [EB761740] 0x03 -> OpenSM 4.8.0.MLNX20160906.32a95b6
OpenSM 4.8.0.MLNX20160906.32a95b6

Nov 30 22:09:04 714948 [EB761740] 0x80 -> OpenSM 4.8.0.MLNX20160906.32a95b6
Using default GUID 0x2c903004af2ab
Entering DISCOVERING state

Nov 30 22:09:04 717488 [EB761740] 0x02 -> osm_vendor_init: 1000 pending umads specified
Nov 30 22:09:04 717587 [EB761740] 0x02 -> osm_vendor_init: 1000 pending umads specified
Nov 30 22:09:04 730625 [EB761740] 0x80 -> Entering DISCOVERING state
SM port is down

Nov 30 22:09:04 730872 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x81 binding to port GUID 0x2c903004af2ab
Nov 30 22:09:04 760951 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x03 binding to port GUID 0x2c903004af2ab
Nov 30 22:09:04 790649 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x04 binding to port GUID 0x2c903004af2ab
Nov 30 22:09:04 790737 [EB761740] 0x02 -> osm_vendor_bind: Mgmt class 0x21 binding to port GUID 0x2c903004af2ab
Nov 30 22:09:04 790820 [EB761740] 0x02 -> osm_opensm_bind: Setting IS_SM on port 0x0002c903004af2ab
Nov 30 22:09:04 792106 [D7ACE700] 0x80 -> SM port is down
WORKING (via direct connect QSFP between HCA's, default opensm config on each node)
centos7-ib01
Code:
SM port is up

Nov 30 22:12:29 448906 [25AD3700] 0x80 -> SM port is up
Entering STANDBY state

Nov 30 22:12:29 454259 [25AD3700] 0x80 -> Entering STANDBY state
centos7-ib04
Code:
SM port is up

Nov 30 22:12:44 735228 [D7ACE700] 0x80 -> SM port is up
Entering MASTER state

Nov 30 22:12:44 738546 [D7ACE700] 0x80 -> Entering MASTER state
Nov 30 22:12:44 743690 [D7ACE700] 0x02 -> SUBNET UP
Nov 30 22:12:44 744914 [E7CEE700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:1 TID:0x0000000000000014
Nov 30 22:12:44 745023 [E7CEE700] 0x02 -> trap_rcv_process_request: Trap 144 Node description update
Nov 30 22:12:44 745068 [E7CEE700] 0x02 -> log_notice: Reporting Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) from LID:1 GID:fe80::2:c903:4a:f2b3
Nov 30 22:12:44 745925 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:401b:ffff::ffff:ffff
Nov 30 22:12:44 747333 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:401b:ffff::1
Nov 30 22:12:44 748203 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::1
Nov 30 22:12:44 748767 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::1:ff4a:f2b3
Nov 30 22:12:44 749018 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:67 (Mcast group deleted) from LID:2 GID:ff12:601b:ffff::1:ff4a:f2b3
Nov 30 22:12:44 753135 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::1:ff4a:f2b3
Nov 30 22:12:44 754347 [E14E1700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:2 GID:ff12:601b:ffff::16
Nov 30 22:12:44 781415 [E64EB700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:2 TID:0x000000000000000a
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
hm, what does ibstatus / ibportstate say about the ports, anything in dmesg / /vat/log/messages ?
how are the lights on the hca when connecting to the Switch?
guess thes should blink for a while and then become green after the link is up, could check this tomorrow at work ...

and, just to exclude bad cable/port: have you checked that both cables work on Host to Host Connection and tried differrent ports on the Switch ?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I also just forced a SM master/standby switchover by issuing a 'systemctl stop opensm' on master SM while running a loop of iperf tests and the tests continued/interfaces stayed lit up and transmitting while the switchover from master/standby role swapped...took abt 30 secs so I have high confidence that my opensm config is working but HELL maybe there is something has HAS to be configured/set in a file when you connect them to an unmanaged IB switch.

Remember this is ALL and ONLY working while direct connect from host-to-host HCA's, I cannot get ANY link at all on switchport/HCA when conned to switch although two top lights are on IS5022 are green (I believe good overall system status). I will hook back up to switch now that I have demo'ed a few observations here.

Code:
Nov 30 22:12:29 454259 [25AD3700] 0x80 -> Entering STANDBY state
Nov 30 22:27:49 469277 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                        SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12c7
                        Initial path: 0,1 Return path: 0,1
Nov 30 22:27:59 469446 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                        SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12c8
                        Initial path: 0,1 Return path: 0,1
Nov 30 22:28:09 469631 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                        SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12c9
                        Initial path: 0,1 Return path: 0,1
Nov 30 22:28:19 469757 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                        SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12ca
                        Initial path: 0,1 Return path: 0,1
Nov 30 22:28:29 469836 [242D0700] 0x01 -> log_rcv_cb_error: ERR 3111: Received MAD with error status = 0xC
                        SubnGetResp(SMInfo), attr_mod 0x0, TID 0x12cb
                        Initial path: 0,1 Return path: 0,1
Entering DISCOVERING state

Nov 30 22:28:39 471089 [2FCE7700] 0x80 -> Entering DISCOVERING state
Entering MASTER state

Nov 30 22:28:39 474687 [25AD3700] 0x80 -> Entering MASTER state
Nov 30 22:28:39 479102 [25AD3700] 0x02 -> SUBNET UP
Nov 30 22:28:39 975080 [2ECE5700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:2 TID:0x000000000000000b
Nov 30 22:28:39 975143 [2ECE5700] 0x02 -> trap_rcv_process_request: Trap 144 Node description update
Nov 30 22:28:39 975165 [2ECE5700] 0x02 -> log_notice: Reporting Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) from LID:2 GID:fe80::2:c903:4a:f2ab
Nov 30 22:28:40 479662 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:401b:ffff::ffff:ffff
Nov 30 22:28:40 481310 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:401b:ffff::1
Nov 30 22:28:40 481983 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:601b:ffff::1
Nov 30 22:28:40 482639 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:601b:ffff::1:ff4a:f2b3
Nov 30 22:28:40 482991 [2B4DE700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:66 (New mcast group created) from LID:1 GID:ff12:601b:ffff::1:ff4a:f2ab
Nov 30 22:28:49 464709 [25AD3700] 0x02 -> SUBNET UP
Nov 30 22:28:59 464924 [25AD3700] 0x02 -> SUBNET UP
Nov 30 22:29:09 465141 [25AD3700] 0x02 -> SUBNET UP
Nov 30 22:29:12 021824 [2CCE1700] 0x01 -> log_trap_info: Received Generic Notice type:4 num:144 (CapabilityMask, NodeDescription, Link [Width|Speed] Enabled, SM priority changed) Producer:1 (Channel Adapter) from LID:2 TID:0x000000000000000c
 

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
i wouldn't care about the sm at all until you get a link on the Switch. it runs fine without any config-file as long as there is at least one running on the fabric.

i suspect a more physical problem as i guess there is no link.
 
  • Like
Reactions: whitey

Drewy

Active Member
Apr 23, 2016
208
56
28
54
Does the switch log anything? Any indication that it likes your cables...
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
hm, what does ibstatus / ibportstate say about the ports, anything in dmesg / /vat/log/messages ?
how are the lights on the hca when connecting to the Switch?
guess thes should blink for a while and then become green after the link is up, could check this tomorrow at work ...

and, just to exclude bad cable/port: have you checked that both cables work on Host to Host Connection and tried differrent ports on the Switch ?
Nothing out of the ordinary in /var/log/messages, looks clean, typical mlx stuff, I'd like to think that since host to host direct works that the OS/OFED drivers/HCA's are good seeings how I can shoot 20+Gbps over them in that fashion.

No lights at all (not even a quick blink) on switch of HCA that I can see, SM reports HCA interfaces as down always when conn'ed to IS5022.

Next question is a resounding YES

Tried all 8 ports on IS5022, nothing different :-( ebay seller (also a member here) says it all checked out and is in working order.
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
i wouldn't care about the sm at all until you get a link on the Switch. it runs fine without any config-file as long as there is at least one running on the fabric.

i suspect a more physical problem as i guess there is no link.
Both of my two QSFP copper cables work when direct-conn between phys hosts HCA's :-( Don't think it is a bad cable, smashing data @ 21Gbps host-to-host seems to eliminate bad HCA's. All signs seem to point back to switch but the top two indicators are green lit, RTFMing seems to lead me to believe it 'should' be fat and happy like this. WTF HAH

This is the IS5022 guide I found, Status LED section (clickable) is what mine looks like w/ top two green ports (Status LED/Fan Indicators) lit.

http://www.mellanox.com/related-docs/user_manuals/IS5022_User_Manual.pdf

Can't seem to find a 'reset to factory default' reset button like I think there is on some of the other IS50xx series switches (bigger brothers) so I'll assume it doesn't exist. Sure would be nice to have that simple serial to rj45 mgmt cable work like it does on all my other gear to at least see the damn thing boot/initialize...kinda a black box at this point. I think I saw a wonky I2C kit to 'actually' get output from these...garbage...just what I need another mgmt cable.

I did update the HCA FW to 2.9.1200 last night from 2.9.1000 successfully.
 
Last edited:

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
ok, Sorry for the cables / ports question lol

no lights on the adapters sounds wrong, i can't remember the exact colors, but you should see there if physical link comes up.

also in ibstat you should be able to see status and physicsl state, where the First is related to the sm/fabrics status of the port ...
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Here is the output of ibstat when connected to switch, look ugly/way different (not good) than when direct-conn....umm well it did before a reboot of both hosts/switch/restart of opensm...WTF

Code:
[root@centos7-ib01 ~]# ibstat
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 1
        Firmware version: 2.9.1200
        Hardware version: b0
        Node GUID: 0x0002c903004af2b2
        System image GUID: 0x0002c903004af2b5
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 1
                LMC: 0
                SM lid: 2
                Capability mask: 0x0251086a
                Port GUID: 0x0002c903004af2b3
                Link layer: InfiniBand
[root@centos7-ib01 ~]#
Code:
[root@centos7-ib04 ~]# ibstat
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 1
        Firmware version: 2.9.1200
        Hardware version: b0
        Node GUID: 0x0002c903004af2aa
        System image GUID: 0x0002c903004af2ad
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 2
                LMC: 0
                SM lid: 2
                Capability mask: 0x0251086a
                Port GUID: 0x0002c903004af2ab
                Link layer: InfiniBand
[root@centos7-ib04 ~]#
I am LOLing here, that was insane, I just noticed it all looks happy now after a hail Mary reboot of everything. Well that doesn't exactly leave a warm fuzzy but I will take it I guess. Talk abt a WTF moment in the IT trenches. Sorry for the hassle guys, you all rock for jumping in!

EDIT: Still iperfing between both phys hosts now @ 21Gbps (same as host-to-host direct QSFP connect)...a good sign (client CPU limited, ohh btw is iperf single threaded anyone know?)

So I am CONNECTED AND WORKING through the IS5022 (WOOHOOO). Links look good on switch now, nice and blinky green...ohh the insanity, a freaking reboot of everything fixed this. I am thoroughly perplexed and semi amused...semi terrified!
 
Last edited:

epicurean

Active Member
Sep 29, 2014
785
80
28
I ordered an IS5022 which is on its way, and I must say I read this thread with a lot of trepidation. Hopefully, I have an easier time than you did @whitey .
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Worst case you know to pull a Windows move :-D

I must admit I love that is only consumes 40-60 watts and it doesn't raise my home lab to jet engine noises. Hopefully it's a ROCK from here on out. I am just as surprised I had this many challenges to get a simple 2-node IB fabric/communications going but the pain is over now and many a lessons learned.
 
  • Like
Reactions: _alex and T_Minus