IPoIB vs. SRP

[markpower28]

It take forever, but finally able to get Solaris 11.2 ZFS server build.

I am running a pool with 4 x mirrors with 8 x 450 GB SAS drive. I have 4 x 250 GB SSD for L2ARC. HBA is LSI 9207-8e with ConnectX-2 VPI

This is for a vSphere 5.5 cluster with ConnectX-2 VPI. Test was performed on a 2012R2 VM.

The initial benchmark was not impressive at all.

Then the search for RDMA for vSphere and Solaris begin but I hit a wall when try to get iSER to work. Looks like Mellanox still working on the iSER driver for IB and it's not ready yet.

Then by chance I realize 1.8.2.4 driver does support SRP. The setup was very easy. Then I used the same Storage Pool, same volume, same LUN and even the same VM did the same test except using SRP target instead of iSCSI target. The performance was night and day!

This is just the initial number, but it looks promising!

 

[capn_pineapple]

And now for some tuning

 

[gizmo15]

i'm a newbie in this domain, but when i look your screenshots, i see the cpu utilization and the time response are rather high in SRP, not a problem?

 

[markpower28]

10% is not noticeable for throughput testing. When you do a random testing, you will see over 95% CPU utilization. (this apply to any file system include MS SMB 3.0)
ZFS host is a node in C6100 with 2 x L5630 and 24 GB memory.

 

[Entz]

You will also find that while the CPU usage in the VM is lower (granted it is also processing less) when using IPoIB, the CPU usage in both ESXi and your storage node will be significantly higher. SRP is pretty amazing, really too bad iSER is so poorly supported.

 

[JeffroMart]

Guys, I was wondering if you could give me a hand with my configuration?

I have a Voltaire 4036, I got it up and running with IPoIB, just fine, but I would like to setup SRP since everyone seems to mention it is getting better performance.

I am using the 1.8.2.4 OFED drivers on VMware 6 side, and I am using OSNexus/QuantaStor on the storage side.

I got the iSCSI up fairly easy as I am more familiar with this, however I cannot seem to get it to be recognized by the Storage Adapter in VMWare, which I will assume is going to be the SRP side?

I believe it may have something to do with Hosts initiator config with QuantaStor, it seems to be asking for the following:

FC Initiator WWPN or IB GID: 00:00:00:00:00:00:00:00

So, I checked the logs inside of OSNEXUS, and I can see things like this:

kern.log:Jul 25 17:01:50 quantastor kernel: [ 0.000000] BRK [0x01fe8000, 0x01fe8fff] PGTABLE
kern.log:Jul 25 17:01:50 quantastor kernel: [ 7.108454] scst: Target fe80:0000:0000:0000:0002:c903:000d:d19b for template ib_srpt registered successfully
kern.log:Jul 25 17:01:50 quantastor kernel: [ 7.108754] scst: Target fe80:0000:0000:0000:0002:c903:000d:d19c for template ib_srpt registered successfully
kern.log:Jul 25 17:02:30 quantastor kernel: [ 46.521085] ib_srpt: Received SRP_LOGIN_REQ with i_port_id 8102:c903:000d:d19b:0002:c903:000d:d543, t_port_id 0002:c903:000d:d19a:0002:c903:000d:d19a and
it_iu_len 580 on port 1 (guid=fe80:0000:0000:0000:0002:c903:000d:d19b)
kern.log:Jul 25 17:02:30 quantastor kernel: [ 46.521097] ib_srpt: rejected SRP_LOGIN_REQ because target fe80:0000:0000:0000:0002:c903:000d:d19b is not enabled
kern.log:Jul 25 17:02:43 quantastor kernel: [ 60.169707] scst: Using autogenerated relative target id 1 for target fe80:0000:0000:0000:0002:c903:000d:d19b
kern.log:Jul 25 17:02:43 quantastor kernel: [ 60.169714] ib_srpt: Enabling target fe80:0000:0000:0000:0002:c903:000d:d19b
kern.log:Jul 25 17:02:43 quantastor kernel: [ 60.169867] scst: Using autogenerated relative target id 2 for target fe80:0000:0000:0000:0002:c903:000d:d19c
kern.log:Jul 25 17:02:43 quantastor kernel: [ 60.169869] ib_srpt: Enabling target fe80:0000:0000:0000:0002:c903:000d:d19c
kern.log:Jul 25 17:03:30 quantastor kernel: [ 106.533200] ib_srpt: Received SRP_LOGIN_REQ with i_port_id 8102:c903:000d:d19b:0002:c903:000d:d543, t_port_id 0002:c903:000d:d19a:0002:c903:000d:d19a and
it_iu_len 580 on port 1 (guid=fe80:0000:0000:0000:0002:c903:000d:d19b)
kern.log:Jul 25 17:03:30 quantastor kernel: [ 106.536558] scst: Using security group "fe80:0000:0000:0000:0002:c903:000d:d19b" for initiator "fe80:0000:0000:0000:0002:c903:000d:d543" (target fe80:000
0:0000:0000:0002:c903:000d:d19b)
kern.log:Jul 25 17:11:30 quantastor kernel: [ 586.607858] ib_srpt: Received SRP_LOGIN_REQ with i_port_id 8102:c903:000d:d19c:0002:c903:000d:d543, t_port_id 0002:c903:000d:d19a:0002:c903:000d:d19a and
it_iu_len 580 on port 2 (guid=fe80:0000:0000:0000:0002:c903:000d:d19c)
kern.log:Jul 25 17:11:30 quantastor kernel: [ 586.611142] scst: Using security group "fe80:0000:0000:0000:0002:c903:000d:d19c" for initiator "fe80:0000:0000:0000:0002:c903:000d:d543" (target fe80:000
0:0000:0000:0002:c903:000d:d19c)
kern.log:Jul 25 17:21:55 quantastor kernel: [ 1211.701682] ib_srpt: Received SRP_LOGIN_REQ with i_port_id 8102:c903:000d:d19c:0002:c903:000d:d543, t_port_id 0002:c903:000d:d19a:0002:c903:000d:d19a and
it_iu_len 580 on port 2 (guid=fe80:0000:0000:0000:0002:c903:000d:d19c)
kern.log:Jul 25 17:21:55 quantastor kernel: [ 1211.704996] scst: Using security group "fe80:0000:0000:0000:0002:c903:000d:d19c" for initiator "fe80:0000:0000:0000:0002:c903:000d:d543" (target fe80:000
0:0000:0000:0002:c903:000d:d19c)
kern.log:Jul 25 17:21:55 quantastor kernel: [ 1211.705101] ib_srpt: Relogin - closed existing channel fe80:0000:0000:0000:0002:c903:000d:d543
kern.log:Jul 25 17:21:55 quantastor kernel: [ 1211.705161] ib_srpt: Received CM DREP message for ch fe80:0000:0000:0000:0002:c903:000d:d543.
kern.log:Jul 25 17:25:55 quantastor kernel: [ 1451.723093] ib_srpt: Received SRP_LOGIN_REQ with i_port_id 8102:c903:000d:d19b:0002:c903:000d:d543, t_port_id 0002:c903:000d:d19a:0002:c903:000d:d19a and
it_iu_len 580 on port 1 (guid=fe80:0000:0000:0000:0002:c903:000d:d19b)
kern.log:Jul 25 17:25:55 quantastor kernel: [ 1451.726462] scst: Using security group "fe80:0000:0000:0000:0002:c903:000d:d19b" for initiator "fe80:0000:0000:0000:0002:c903:000d:d543" (target fe80:000
0:0000:0000:0002:c903:000d:d19b)

I tried adding: fe80:0000:0000:0000:0002:c903:000d:d543 to the hosts initiator config inside of QuantaStor, however after refreshing the storage controllers inside of VMware, still can't see any Targets/Paths/Devices

Any comments/ideas?

 

[markpower28]

Is OSNexus/QuantaStor IB or FC?

 

[JeffroMart]

It supports both IB and FC

 

[markpower28]

I have not work with FC target yet. But from your config is that a FC target instead of SRP target? Target fe80:0000:0000:0000:0002:c903:000d:d19c

For SRP I am expecting naming as follow: eui.0002C90300509Dxx