qdr infiniband slowness.

[RimBlock]

Hi,

I am currently looking for suggestions on how to troubleshoot what may be going on.

I have a Windows 2012r2 server connected via a connectx-2 qdr vpi adaptor to a Voltaire 4036 ib switch and then to a Solaris 11 server via the same type of connectx-2 vpi adaptor.

I have two luns shared via iscsi targets.
5x Hitachi ultrastar 2TB drives (raidz) with a 120GB ssd zil.
2x 120GB ssd drives (stripped).

All drives are connected to a Sata III Supermicro chassis backplane and then through lsi 9211-8i HBA s (original LSI, not reflashed).

The two ib cards have been flashed with an RDMA version of their firmware as per DBA's guide.

Issues I am seeing.
Large file transfers from the iscsi mapped Hitachi drives usually hovers around 50 -》 100MB/s. It also seems to go in waves dropping to 0 for a few seconds (maybe once every 15s).

Using the ssd iscsi share and the Hitachi share at the same time compounds the 'drop to 0MB/s' issue.

Any ideas to help find out what is causing such poor performance ?.

 

[markpower28]

Iboip's performance is not the best out there compare to other protocol. If you have to stay within windows, have you try windows based SMB 3 yet?

 

[RimBlock]

It's for home and is convenient for it to be a windows server.

Does Solaris support SMB 3 ?.

The IB connection between the Windows sever and the Solaris San is where the slowdown seems to be occurring (unless the switch is causing issues).

It almost looks like it is filling up a cache / ram and then clearing it then repeating over and over again.

One other thing that may have an effect is that the Windows server is running on a E3-1220Lv2. I am wondering if it is a underpowered for this connectivity (cpu load via taskmaster is low though).

Are there any other alternatives that can handle ZFS and ib and play with a Windows server without needing to use IPoIB ?.

 

[markpower28]

Solaris support SMB 2.1 I belive. When I testing with IBoIP vs. SRP is difference is 550 MB/s vs 3000 MB/s IPoIB vs. SRP

The best RDMA performance for Windows based is SMB 3 which is very similar to SRP, but that means you need to change from ZFS to Windows.

The next closest thing will be iSER, but there is no iSER driver for Windows...

 

[RimBlock]

Thanks Mark.

I thought there was no srp for windows either ?.

I used to go from Solaris SRP -》 ESXI and pass the drives to the VMs via RDM. Saw a noticeable boost when I moved from ESXi to bare metal for windows build though (os speed wise not drive speed).

Looks like there is another NAS OS called rockstor which uses CentOS (which I like and has IB packages available) and BTRFS which I need to have a read up on.

If windows drivers do support SRP then that would be a good thing for me to check and use though.

 

[markpower28]

The last Windows support SRP was Windows 2008. It's not going to work with 2012 R2.

BTW, the testing with SRP was done on a VM sit on ESXi. The LUN was presented to ESXi via SRP instead of IPoIB.

 

[RimBlock]

Thanks Mark.

Shame things have not moved on over the last 3 years on this issue of different os only supporting some IB protocols.

I may go back to the VM way again but will need to up my E3-1220Lv2 to something a bit more robust.