[root@centos7-ib3 ~]# iperf -c 10.10.10.101
------------------------------------------------------------
Client connecting to 10.10.10.101, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.100 port 52781 connected with 10.10.10.101 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 21.7 GBytes 18.6 Gbits/sec
[root@centos7-ib3 ~]# iperf -c 10.10.10.101
------------------------------------------------------------
Client connecting to 10.10.10.101, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[ 3] local 10.10.10.100 port 52782 connected with 10.10.10.101 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 21.7 GBytes 18.6 Gbits/sec
[root@centos7-ib3 ~]#
0 root@centos7-ib3:~
[root@centos7-ib02 ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 52781
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 21.7 GBytes 18.6 Gbits/sec
[ 5] local 10.10.10.101 port 5001 connected with 10.10.10.100 port 52782
[ 5] 0.0-10.0 sec 21.7 GBytes 18.6 Gbits/sec
You should be able to use opensm, can be installed by apt and the run the service on at least one node.Hi @whitey, I just got my IS5022 and about to setup for 3 ESXI hosts. Which SM manager did you use and install? I have an ubuntu VM which is on all the time if that helps
Without any software installation, plugged the cables in and no lights on yet. Is that right?
Absolutely, IPoIB literally means "IP over Infiniband", no SM->no Infiniband, no Infiniband->no IP...no SM, no IB ... guess also IPoIB needs an SM somewhere in the farbic
For starters, if you are on the stock ESXi image, it's using the built-in drivers that are intended for Ethernet-only mode. Depending on your card/switch and your intended usage of this infrastructure, you might need to do this:I am on esxi 6.0u2, and the mellanox card is present in esxi, not passed through.
I cannot find in the manual what being lit orange on the LED port lights mean.
Any suggestion to which SM should I install?
That's good, you've got ConnectX2 VPI, which is 40Gbps IB/10Gbps ETH.Thank you @mpogr for your extensive reply. If you can indulge in me for a few more questions
1. My mellanox cards come up as Mellanox MT26428, connectX VPI on esxi 6.0u2
That's not an issue at all. Just remove all Mellanox drivers from your system:2. I would like to have RDMA access(not a must for now), but unfortunately I already installed the 2.40 drivers because I thought it was the latest and the easiest. I also removed the 3 conflicting drivers from vmware ( nmlx4-core, nmlx4-en and nmlx4-rdma) after installing the 2.40 drivers( or the led light won't even come on).
Yes, you would. Just remove the same modules you did the first time and re-install 1.8.2.5 + reboot.Would I need to change drivers everytime I do an esxi update?
Yes, but be conscious about having only one SM in your fabric. If your links are host-to-host, you'd need a copy of SM per link (on either side). If you have all your hosts connected to an unmanaged switch, you'd need only one copy running on a host. Make sure it's up and running all the time, a fabric can survive the SM going down for a short brief, but not for long time.3. I intend to install this SM in esxi ib-opensm-x64.3.3.15_offline_bundle.zip, is that ok?
Strictly speaking, you can install 1.8.3 beta drivers and use iSER with IPoIB, but it has no advantages over SRP. Do it only if you absolutely must.4. I can forget about iSER since I don't have the right cards?
Here you go: http://www.mellanox.com/downloads/Software/MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585.zipWould you have a download link I can use?