The post is based on utilize Mellanox 4036 build-in subnet manager and only install IB driver 1.8.2.4 on vSphere 5.5/6.0
I used Solaris 11.2 express for ZFS platform. Then use the following configured IB interface.
IPoIB
dladm show-ib
dladm show-phys
dladm show-part
dladm create-part -l net3 -P 0x8001 p8001.ib1 (defined in 4036 subnet manager)
dladm show-
ipadm create-ip p8001.ib1
ipadm create-addr -a 10.x.x.x/24 p8001.ib1/ipv4 (iSCSI interface)
I always prefer GUI for easy management, so I installed napp-it on top of it.
Even Napp-it does not officially support IB(if I remember correctly), Comstar works perfectly with the iSCSI interface above. vSphere works fine with IBoIP but performance is not impressive at all. https://forums.servethehome.com/index.php?threads/ipoib-vs-srp.5048/
Then I start explorer with SRP option. I did not see any native option with napp-it. I did the following.
svcadm enable ibsrp/target
stmfadm list-target -v
stmfadm create-tg tgsrp
stmfadm add-tg-member -g tgsrp eui.0024E89097xxxxxx (ZFS server)
stmfadm create-hg hgsrp
stmfadm add-hg-member -g hgsrp eui.0024E89097xxxxxx (ESXi host 1)
stmfadm add-hg-member -g hgsrp eui.0024E89097xxxxxx (ESXi host 2)
stmfadm add-hg-member -g hgsrp eui.0024E89097xxxxxx (ESXi host 3)
stmfadm add-view -h hgsrp -t tgsrp 600144F0685E4600000054F9ECxxxxxx
Then SRP target shows in napp-it Comstar GUI option.
After install IB 1.8.2.4 driver on vSphere 5.5/6. It recognize SRP target right the way. And performance is night and day (from 500 MB/s to 3000 MB/s)
I am using ConnectX2 VPI (2.10.700) from both vSphere and Solaris.
Subnet Manager is 4036 build-in SM.
*This is for a lab environment, if you plan to do this in production please make sure your ZFS solution support both IB and SRP.
I used Solaris 11.2 express for ZFS platform. Then use the following configured IB interface.
IPoIB
dladm show-ib
dladm show-phys
dladm show-part
dladm create-part -l net3 -P 0x8001 p8001.ib1 (defined in 4036 subnet manager)
dladm show-
ipadm create-ip p8001.ib1
ipadm create-addr -a 10.x.x.x/24 p8001.ib1/ipv4 (iSCSI interface)
I always prefer GUI for easy management, so I installed napp-it on top of it.
Even Napp-it does not officially support IB(if I remember correctly), Comstar works perfectly with the iSCSI interface above. vSphere works fine with IBoIP but performance is not impressive at all. https://forums.servethehome.com/index.php?threads/ipoib-vs-srp.5048/
Then I start explorer with SRP option. I did not see any native option with napp-it. I did the following.
svcadm enable ibsrp/target
stmfadm list-target -v
stmfadm create-tg tgsrp
stmfadm add-tg-member -g tgsrp eui.0024E89097xxxxxx (ZFS server)
stmfadm create-hg hgsrp
stmfadm add-hg-member -g hgsrp eui.0024E89097xxxxxx (ESXi host 1)
stmfadm add-hg-member -g hgsrp eui.0024E89097xxxxxx (ESXi host 2)
stmfadm add-hg-member -g hgsrp eui.0024E89097xxxxxx (ESXi host 3)
stmfadm add-view -h hgsrp -t tgsrp 600144F0685E4600000054F9ECxxxxxx
Then SRP target shows in napp-it Comstar GUI option.
After install IB 1.8.2.4 driver on vSphere 5.5/6. It recognize SRP target right the way. And performance is night and day (from 500 MB/s to 3000 MB/s)
I am using ConnectX2 VPI (2.10.700) from both vSphere and Solaris.
Subnet Manager is 4036 build-in SM.
*This is for a lab environment, if you plan to do this in production please make sure your ZFS solution support both IB and SRP.
Last edited: