Mellanox/Voltaire 4036 Subnet Manager - partition.conf

markpower28

Active Member
Apr 9, 2013
405
103
43
I have been try to make subnet manager to work with vSphere vLAN for a while but did not have much luck until couple month ago. There is not many good documentation out there and I saw quite a few people have the same issue as I had. I like to share my 2 cents here to get people started.

1. 4036's subnet manager is controlled by a file named partition.conf. it's located in /usr/voltarire/config. The easiest way to access the file is use WinSCP login as root then you can edit the partition.conf file directly without the need to import/export file via FTP (make sure change the file permission first)

2. partition.config file. By default, it has
Default=0xffff , ipoib: ALL=full;
This line actually apply the first 0-7 pkeys(vLAN ID) and this is where SRP (SCSI RDMA Protocol) use.

Then that's the fun part of the customize pkey/vLAN access (vMotion, iSCSI, NFS, FT, etc). For the pkey/vLan ID in the partition.config file it needs to be Hex format. And vLan ID in the vSphere needs to be decimal. The first 0-7 pkey/vLAN belong to the default. So 8001 to 8007 is out. And because it's in hex format, the first one I would use is 8010 which is vLan 16.

Below is what I have
Default=0xffff , ipoib: ALL=full;
Partition16=0x8010 , ipoib , defmember=full: ALL=full;
Partition17=0x8011 , ipoib , defmember=full: ALL=full;
Partition18=0x8012 , ipoib , defmember=full: ALL=full;
Partition19=0x8013 , ipoib , defmember=full: ALL=full;
Partition20=0x8014 , ipoib , defmember=full: ALL=full;

From vSphere vkernel vLan ID should be,
vLan = 16
vLan = 17
vLan = 18
vLan = 19
vLan = 20

My reference is based on post from the following vSphere 5.1 U1 IPoIB VLAN support | Mellanox Interconnect Community

I have the following in my lab
Switch: 4036
Server: C6100
IB card: ConnectX-2 VPI firmware version 2.10.700
ZFS: Solaris 11.2 with Napp-it using SRP on ConnectX-2 VPI firmware version 2.10.700.

I am able to push 3000 MB/s via SRP https://forums.servethehome.com/index.php?threads/ipoib-vs-srp.5048/