Proxmox VE 4.1 setup with scst srpt & srptools over Infiniband

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

EluRex

Active Member
Apr 28, 2015
218
78
28
Los Angeles, CA
I don't know how many still using SRPT but I am using it as my main protocol to share LUN instead of iscsi/iser.

Package Info
Code:
root@nas:~# pveversion --verbose
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-28
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-42
pve-firmware: 1.1-7
libpve-common-perl: 4.0-42
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-18
pve-container: 1.0-35
pve-firewall: 2.0-14
pve-ha-manager: 1.0-16
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie
openvswitch-switch: 2.3.2-2
In addition, the following packages are needed

build-essential, subversion, opensm, pve-headers-4.2.6-1-pve

Then run following command, the SCST version that I use to compile is 6776
Code:
svn checkout svn://svn.code.sf.net/p/scst/svn/trunk scst-trunk.6776
cp scst-trunk.6776 scst-trunk -R
cd scst-trunk
make 2perf
make scst srpt scstadm scst_install srpt_install scstadm_install
This should compile and install scst.ko and ib_srpt.ko

Follow config will load the necessary module when starting scst service
Code:
nano /etc/init.d/scst

parse_scst_conf() {
--  SCST_MODULES="scst"
++ SCST_MODULES="scst scst_vdisk ib_srpt"
following config is a sample of /etc/scst.conf
Code:
root@nas:~# cat /etc/scst.conf
# Automatically generated by SCST Configurator v3.1.0-pre1.

HANDLER vdisk_blockio {
        DEVICE asrs_disk {
                filename /dev/zvol/rpool/vm/asrs_disk
        }

        DEVICE erp_ap {
                filename /dev/zvol/rpool/vm/erp_ap
        }
}

TARGET_DRIVER ib_srpt {
        TARGET fe80:0000:0000:0000:0002:c903:00a0:b671 {
                comp_v_mask ff
                enabled 1
                rel_tgt_id 1

                GROUP pve4 {
                        LUN 0 asrs_disk
                        LUN 1 erp_ap

                        INITIATOR *
                }
        }
}
 
  • Like
Reactions: Patrick and T_Minus

EluRex

Active Member
Apr 28, 2015
218
78
28
Los Angeles, CA
For SRPTOOLS setup is more difficult than SRP Target. The default srptool from debian 8.1 repo does not work. Also the key component ibsrpdm -c failed to probe and return any ib device, which is very frustrating.

The work around is to use the srptools from Mellanox, the package that I use is

MLNX_OFED_LINUX-3.1-1.0.3-debian8.1-x86_64.tgz

Download it and extract it.

Code:
cd MLNX_OFED_LINUX-3.1-1.0.3-debian8.1-x86_64/DEBS/
dpkg -i libibumad*.deb
dpkg -i srptools_1.0.2-12_amd64.deb
By default ibsrpdm -c should output following info
Code:
root@san:~# ibsrpdm -c
id_ext=0002c903009f4480,ioc_guid=0002c903009f4480,dgid=fe800000000000000002c903009f4481,pkey=ffff,service_id=0002c903009f4480
However due to PVE failed to produce those info, ibstat needs to be used instead
Code:
root@san:~# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.34.5000
        Hardware version: 0
        Node GUID: 0x0002c903009f4480
        System image GUID: 0x0002c903009f4483
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 2
                LMC: 0
                SM lid: 1
                Capability mask: 0x0259486a
                Port GUID: 0x0002c903009f4481
                Link layer: InfiniBand

root@san:~# ibstatus
Infiniband device 'mlx4_0' port 1 status:
  default gid:  fe80:0000:0000:0000:0002:c903:009f:4481
  base lid:  0x2
  sm lid:  0x1
  state:  4: ACTIVE
  phys state:  5: LinkUp
  rate:  40 Gb/sec (4X QDR)
  link_layer:  InfiniBand
id_ext=0002c903009f4480,ioc_guid=0002c903009f4480,dgid=fe800000000000000002c903009f4481

Code:
root@vm-ha:~/script# nano srptools.bsh
echo "id_ext=0002c903009f4480,ioc_guid=0002c903009f4480,dgid=fe800000000000000002c903009f4481" > /sys/class/infiniband_srp/srp-mlx4_0-1/add_target

root@vm-ha:~/script# nano /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

mlx4_ib
ib_umad
ib_ipoib
ib_srp

root@vm-ha:~/script# nano /etc/rc.local

/root/script/srptools.bsh
exit 0
Then when you lsscsi, you should see your SRPT disk like following

Code:
root@vm-ha:~/script# lsscsi
[0:0:1:0]    cd/dvd  MATSHITA DVD-ROM SR-8178  PZ16  /dev/sr0
[2:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sda
[3:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdb
[4:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdc
[5:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdd
[6:0:0:0]    disk    ATA      INTEL SSDSC2BW12 RG21  /dev/sde
[9:0:0:0]    disk    SCST_BIO asrs_disk         320  /dev/sdf
[9:0:0:1]    disk    SCST_BIO erp_ap            320  /dev/sdg
[9:0:0:2]    disk    SCST_BIO erp_kac           320  /dev/sdh
[9:0:0:3]    disk    SCST_BIO mail_cfm          320  /dev/sdi
[9:0:0:4]    disk    SCST_BIO mail_disk         320  /dev/sdj
[9:0:0:5]    disk    SCST_BIO mail_data         320  /dev/sdk
[9:0:0:6]    disk    SCST_BIO oracle            320  /dev/sdl
[9:0:0:7]    disk    SCST_BIO oracle_data       320  /dev/sdm
[9:0:0:8]    disk    SCST_BIO report_disk       320  /dev/sdn
 
Last edited:
  • Like
Reactions: Patrick