SRPT error on mlnx_ofed 2.4-1.0.4

[EluRex]

Hi All,

I have a setup ZFS Server connect to a vSphere ESXi 6.0 host via SRP. Here is some setup info

• OS: Ubuntu 14.04
• Kernel: linux-headers-3.19.5-031905-generic
• MLNX_OFED Driver: 2.4-1.0.4
• SCST Version: v3.1.0-pre#MOFED
• SCST SVN Tag: 6187

When I use Ubuntu distro IB driver (version 2.2-1) I was able to insmod and mobprobe ib_srpt. However, if I use the latest ofed driver from mellanox (version 2.4-1.0.4) was not able to load ib_srpt.

Does anyone has success creating SRPT on OFED 2.4 driver? (Kernel 3.19.x should not be an issue here)

Everything else (except srpt) on SCST 3.0.1 compiled and installed fine without error, scst scstadm iscsi all works flawlessly.

The dmesg error as follow

scst: Init thread started, PID 5330
scst: Task management thread started, PID 5331
scst: SCST version 3.1.0-pre1 loaded successfully (max mem for commands 4011MB, per device 1604MB)
ib_srpt: disagrees about version of symbol ib_unregister_client
ib_srpt: Unknown symbol ib_unregister_client (err -22)
ib_srpt: disagrees about version of symbol ib_create_cq
ib_srpt: Unknown symbol ib_create_cq (err -22)
ib_srpt: disagrees about version of symbol ib_modify_port
ib_srpt: Unknown symbol ib_modify_port (err -22)
ib_srpt: disagrees about version of symbol ib_cm_listen
ib_srpt: Unknown symbol ib_cm_listen (err -22)
..........

 

[tjk]

EluRex,

I don't have an answer to your problem, but it would be awesome if you wrote up a how-to on how you got this working. There are quite a few IB folks on here, testing everything under the sun.

Welcome to STH btw!

Tom

 

[markpower28]

Is iSER working? If you are running ZFS on Linux, I would stay with iSER. That's where all the development focused on. esxi consistently has driver update for Ethernet but it has been over a year from ib side of the house.

 

[tjk]

esxi consistently has driver update for Ethernet but it has been over a year from ib side of the house.

Mark, is there an official ESX 5.5 or 6.0 driver that supports iSER? I thought only a beta driver was out for ESX 5.5/6 that supported iSER?

 

[markpower28]

For ib, 1.8.3 beta is the only option, but for Ethernet latest 1.9.10 support iSER.

 

[EluRex]

Mini How to

1. Assuming all firmware for the IB card is updated to the latest (for mellanox card is 2.33 as of today)
2. Download and install the latest supported 3.19 kernel from ubuntu main ppa which is 3.19.5 as of today
3. Reboot
4. Uninstall all ubuntu distro IB driver

   apt-get -y --force-yes remove libibcm1 libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils ibsim-utils ibutils libcxgb3-1 libibmad5 libibumad3 libmlx4-1 libmthca1 libnes1 rds-tools infiniband-diags libibcommon1 mstflint opensm libopensm5 perftest srptools

5. Download mlnx_ofed 2.4-1 for ubuntu 14.10 driver and extract it
6. Change the content of distro file in the mlnx 2.4 install folder from ubuntu 14.10 to ubuntu 14.04 (if you are using 14.10, skip this step)
7. run the mlnxofedinstall and it will compile and install everything.
8. If you need ibnetdiag, please install it from the mlnxofed 2.4.1 ubuntu 14.04 package, it's in the DEBS folder (if you are using 14.10, skip this step)
9. Download the latest SCST (svn tag 6187)

   svn checkout svn://svn.code.sf.net/p/scst/svn/trunk scst-trunk

10. cd scst-trunk
11. make 2perf
12. make scst scst_install scstadm scstadm_install srpt srpt_install iscsi iscsi_install

The install of the ib_srp on VMWare vSphere ESXi 6.o is fairly simple. Just follow http://www.mellanox.com/related-doc...or_VMware_vSphere_User_Manual_Rev_1_8_2_4.pdf

More system info

root@nas:~/Downloads/scst-trunk# uname -r
3.19.5-031905-generic

root@nas:~/Downloads/scst-trunk# modinfo mlx4_ib
filename:       /lib/modules/3.19.5-031905-generic/updates/dkms/mlx4_ib.ko
version:        2.4-1.0.4
license:        Dual BSD/GPL
description:    Mellanox ConnectX HCA InfiniBand driver
author:         Roland Dreier
srcversion:     1755A26977F138E4A45EE3D

root@nas:~/Downloads/scst-trunk# lsmod |grep ib
libcrc32c              12644  0
ib_iser                52133  0
libiscsi_tcp           25377  1 iscsi_tcp
libiscsi               57498  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   100628  4 iscsi_tcp,ib_iser,libiscsi
ib_ucm                 18450  0
rdma_cm                44266  2 ib_iser,rdma_ucm
ib_ipoib              136308  0
ib_cm                  52426  3 rdma_cm,ib_ucm,ib_ipoib
ib_uverbs              66418  2 ib_ucm,rdma_ucm
ib_umad                22172  0
mlx5_ib               133901  0
mlx5_core             108054  1 mlx5_ib
mlx4_ib               199832  0
ib_sa                  43635  5 rdma_cm,ib_cm,mlx4_ib,rdma_ucm,ib_ipoib
ib_mad                 51544  4 ib_cm,ib_sa,mlx4_ib,ib_umad
ib_core               130450  13 rdma_cm,ib_cm,ib_sa,iw_cm,mlx4_ib,mlx5_ib,ib_mad,ib_ucm,ib_iser,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib
ib_addr                18702  3 rdma_cm,ib_core,ib_uverbs
mlx4_core             332195  2 mlx4_en,mlx4_ib

root@nas:~/Downloads/scst-trunk# modinfo ib_iser
filename:       /lib/modules/3.19.5-031905-generic/updates/dkms/ib_iser.ko
version:        1.5
author:         Alex Nezhinsky, Dan Bar Dov, Or Gerlitz
license:        Dual BSD/GPL
description:    iSER (iSCSI Extensions for RDMA) Datamover
srcversion:     7C0346B762179682760473C
depends:        compat,libiscsi,ib_core,scsi_transport_iscsi,rdma_cm
vermagic:       3.19.5-031905-generic SMP mod_unload modversions

 

[EluRex]

For ib, 1.8.3 beta is the only option, but for Ethernet latest 1.9.10 support iSER.

SCST still very active in development of SRPT and its able to support 3.19.5 kernel and up to ofed 2.2-1.

The performance SRP vs iSER made me stick with SRP. ZFS volume share via SRPT forvSphere ESXi 6.0 vm guest's RAW DEVICE MAPPING with low latency is crucial for my production environment. Sorry I couldn't provide solid benchmark, but for Oracle DB Enterprise Manager Performance Chart, there is huge difference when I test it in the lab.