ESX Infiniband Config

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

CreoleLakerFan

Active Member
Oct 29, 2013
485
180
43
Running an IB switch with a subnet manager (SS 9024-D). I have partitions set-up, but cannot figure out how to map the IB port on the adapter on the ESX host to different partitions.

-> smSetPKey 1 0x0001 vmkernel
Successfully set PKey index: 1 to 0x0001
-> smSetPKey 2 0x0002 iSCSI
Successfully set PKey index: 2 to 0x0002
-> smShowPKeys

Index: 0 PKey: 0xffff Desc: Default PKey
Index: 1 PKey: 0x0001 Desc: vMotion
Index: 2 PKey: 0x0002 Desc: iSCSI

I created a vmkernel adapter and assigned the physical port. I can only communicate when I have the VLAN ID set to 0. Anything else breaks IP communication ...

The ESXi hosts can see the IB partitions ...

~ # grep -v "0000" /proc/infiniband/mlx4_0/ports/1/pkeys/*
/proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff
/proc/infiniband/mlx4_0/ports/1/pkeys/1:0x0001
/proc/infiniband/mlx4_0/ports/1/pkeys/2:0x0002

I just can't figure out how to map a port/adapter to a different partition.

Any thoughts?
 

CreoleLakerFan

Active Member
Oct 29, 2013
485
180
43
I combed over that link earlier ... It doesn't have what I'm looking for. I got the idea to set the VLAN tag from that posting, but you can only use decimal to set a tag on the adapters.

I don't have partitions.conf on my ESXi hosts, as I am running the Subnet Manager on the switch and have the partitions defined there - at least according to my best guess; the switch documentation doesn't have anything about assigning partitions on a per port basis. I was able to create the partition keys, and they are visible to the infiniband process on the hosts.

Perhaps the answer is that I need to run OpenSM on the ESXi hosts in order to partition the fabric ... But that seems a bit counterintuitive to me. If that's the case I may just run vmkernel and iscsi on the same partition.