Why oh why do I suck at the IB

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
What can I say, I'm a glutton for punishment :-D

Just cutting my IB newbie teeth, many thanks to those who have lived these nightmares and gave me a hint or two to get across the finish line.

EDIT: Ya know what is freaking hilarious...vt-D of these devices work now...I have never believed in firmware so much in my life before...call me a convert. Simple 2.9.1000 to 2.9.1200 on the ConnectX-2 and they are indeed DAMNED working now through vSphere as well. Will go update my other thread in a bit as SOLVED!

Down to 16Gbps from 21Gbps w/ vt-D/virt layer (25% tax...ouch). Set CPU shares to high, mem locked for dedicated rscs but that L5630 client IB node has only got soo much!

DOUBLE EDIT: Guess only down to 18Gbps vt-D from 21Gbps previously phys

'screen' session, sorry if it's hard to read :-(

Code:
[root@centos7-ib3 ~]# iperf -c 10.10.10.101
------------------------------------------------------------
Client connecting to 10.10.10.101, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 10.10.10.100 port 52781 connected with 10.10.10.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
[root@centos7-ib3 ~]# iperf -c 10.10.10.101
------------------------------------------------------------
Client connecting to 10.10.10.101, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 10.10.10.100 port 52782 connected with 10.10.10.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
[root@centos7-ib3 ~]#
   0 root@centos7-ib3:~
[root@centos7-ib02 ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 52781
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
[  5] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 52782
[  5]  0.0-10.0 sec  21.7 GBytes  18.6 Gbits/sec
 
Last edited:

epicurean

Active Member
Sep 29, 2014
785
80
28
Hi @whitey, I just got my IS5022 and about to setup for 3 ESXI hosts. Which SM manager did you use and install? I have an ubuntu VM which is on all the time if that helps

Without any software installation, plugged the cables in and no lights on yet. Is that right?
 

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Yessir, w/out a SM they won't link. Direct-conn/PTP they will though, I'd start there first just to see them light up (warm fuzzy). I only tested on CentOS7/Linux and it was fairly easy there once I had my 'ah ha' moment, looks like when I want to implement in vSphere it's gonna be a bit trickier as I didn't get link lights on the cards running in vSphere w/ a OpenSM running on my CentOS7 boxes on same subnet as ESXi hosts (mgmt subnet). Assuming that the ESXi hosts want/need to run an instance of SM on each node. I'm gonna attempt/crack this nut soon though...very soon...maybe tonight if I get to feeling froggy!

Hell, maybe I should reboot LOL
 
Last edited:

_alex

Active Member
Jan 28, 2016
866
97
28
Bavaria / Germany
Hi @whitey, I just got my IS5022 and about to setup for 3 ESXI hosts. Which SM manager did you use and install? I have an ubuntu VM which is on all the time if that helps

Without any software installation, plugged the cables in and no lights on yet. Is that right?
You should be able to use opensm, can be installed by apt and the run the service on at least one node.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
OK. I connected 2 esxi servers to the IS5022, but both the LED port lights are lit orange, not green on the switch. Both What am I doing wrong?
I have not installed an SM yet. Is that the reason why?
Both Mellanox connectX cards (MT26428) LED are lt green, but ESXI says link down
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
SM would be needed for IB at least.
IPoIB might work without.
Which ESX version are you on?
Are the cards present in ESX or passed through?
 

epicurean

Active Member
Sep 29, 2014
785
80
28
I am on esxi 6.0u2, and the mellanox card is present in esxi, not passed through.
I cannot find in the manual what being lit orange on the LED port lights mean.
Any suggestion to which SM should I install?
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
if you are not passing through (ie only have esx) then u only can run the hypervisor.fr opensm variant
Not tested on 6 but should work (does not work on 6.5)
 

mpogr

Active Member
Jul 14, 2016
115
95
28
53
I am on esxi 6.0u2, and the mellanox card is present in esxi, not passed through.
I cannot find in the manual what being lit orange on the LED port lights mean.
Any suggestion to which SM should I install?
For starters, if you are on the stock ESXi image, it's using the built-in drivers that are intended for Ethernet-only mode. Depending on your card/switch and your intended usage of this infrastructure, you might need to do this:
1. If you have an IB-only switch or want or utilise direct IB connection between ESXi hosts, you have to make sure your cards are in IB mode. Hope your cards are VPI ones, otherwise you're screwed. In the lucky situation your have ConnectX2/3/Pro VPI cards, then you need to decide whether you need SRP initiator (for RDMA access to iSCSI storage) on the ESXi hosts or not.
If you need SRP, your best choice is to get the 1.8.2.5 Mellanox drivers. I'll post further instructions if this is indeed what you need.
Otherwise (if you only need IPoIB), you can use the 2.4.0 drivers. Again, I'll post the instructions if this is your case.
There is currently no solution at all for ConnectX4 and up cards in IB mode for ESXi.
You'll need an SM in your fabric for IB mode. If you don't have a managed switch, you'll need to run OpenSM on one of the computers connected to your fabric. It can be ESXi host itself. There are materials describing how to do this.
2. If you have an Ethernet switch or want to utilise direct connection in ETH mode and you have VPI/ETH cards, the built in drivers should work as is. You won't need an SM for this situation.
The only case you'd need to change something is if you need iSER. Then you'd need to install 1.9.10.5 drivers. These would work only for ConnectX3/Pro, both VPI and EN. For everything else, all you can get is plain IP network.
 

mpogr

Active Member
Jul 14, 2016
115
95
28
53
I'd strongly suggest everyone to read this thread on Mellanox communities to understand the current (miserable) situation with Mellanox and ESXi. If you have anything to say, please, do so there. I hope, with additional volume of concerned voices, Mellanox might actually reply and/or do something about this situation.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
Thank you @mpogr for your extensive reply. If you can indulge in me for a few more questions
1. My mellanox cards come up as Mellanox MT26428, connectX VPI on esxi 6.0u2
2. I would like to have RDMA access(not a must for now), but unfortunately I already installed the 2.40 drivers because I thought it was the latest and the easiest. I also removed the 3 conflicting drivers from vmware ( nmlx4-core, nmlx4-en and nmlx4-rdma) after installing the 2.40 drivers( or the led light won't even come on). Would I need to change drivers everytime I do an esxi update?
3. I intend to install this SM in esxi ib-opensm-x64.3.3.15_offline_bundle.zip, is that ok?
4. I can forget about iSER since I don't have the right cards?
 

mpogr

Active Member
Jul 14, 2016
115
95
28
53
Thank you @mpogr for your extensive reply. If you can indulge in me for a few more questions
1. My mellanox cards come up as Mellanox MT26428, connectX VPI on esxi 6.0u2
That's good, you've got ConnectX2 VPI, which is 40Gbps IB/10Gbps ETH.
2. I would like to have RDMA access(not a must for now), but unfortunately I already installed the 2.40 drivers because I thought it was the latest and the easiest. I also removed the 3 conflicting drivers from vmware ( nmlx4-core, nmlx4-en and nmlx4-rdma) after installing the 2.40 drivers( or the led light won't even come on).
That's not an issue at all. Just remove all Mellanox drivers from your system:
a) Get the list of the modules:
esxcli software vib list | grep -i mel
b) Remove them:
esxcli software vib remove -n module1 -n module2 etc.
Then reboot and install 1.8.2.5.
Would I need to change drivers everytime I do an esxi update?
Yes, you would. Just remove the same modules you did the first time and re-install 1.8.2.5 + reboot.
3. I intend to install this SM in esxi ib-opensm-x64.3.3.15_offline_bundle.zip, is that ok?
Yes, but be conscious about having only one SM in your fabric. If your links are host-to-host, you'd need a copy of SM per link (on either side). If you have all your hosts connected to an unmanaged switch, you'd need only one copy running on a host. Make sure it's up and running all the time, a fabric can survive the SM going down for a short brief, but not for long time.
4. I can forget about iSER since I don't have the right cards?
Strictly speaking, you can install 1.8.3 beta drivers and use iSER with IPoIB, but it has no advantages over SRP. Do it only if you absolutely must.
 

epicurean

Active Member
Sep 29, 2014
785
80
28
Thank you again.
I am using an unmanaged IS5022 Mellanox switch, so i guess just 1 SM on a host will do.
I do not see the 1.8.2.5 OFED package on the mellanox site, only the 1.8.2.4.
Would you have a download link I can use?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
@mpogr that's some awesome information and instructions!! I have some VPI cards I really need to get figured out for my esxi hosts and a 4036 and 4036E (Ethernet bridge) that I'm not sure how it works for bridging and SM and/or if I need to bridge them or just run separate networks... I'll likely start a thread when I get the time.