New SAN/NAS, ZFS OmniOS Napp-It Infiniband ESXI, horrible performance

[cmpufxr]

SAN/NAS

Dell Compellent Controller Head
24GB ECC RAM
4x 4TB HSGT SAS Drives
9x WD 250GB RE3 SATA Drives
2x LSI MegaRAID 9240-8i 8-port RAID cards
Mellanox MHQH29-XTC 40Gb/s Dual Port QFSP ConnectX

ESXI HOST
C6100 with 4 nodes
2 Processors
24GB ECC ram per node
Dell/Mellanox 2port 40GB Mezzanine

SWITCH
HP VOLTAIRE INFINIBAND 4X QDR 36 PORTS MANAGED SWITCH

I have installed ONMI and NAPP-IT on a 16GB flash drive (2x USB3 64GB flash drives on order) setup the HSGT drives in a pool raidz1 (share) and shared it out with SMB via the onboard nics and it seems to be working with good performance 100MB write/100MB read. (120GB SSD on order for ZIL)

I have setup the WD drives raidz1 (esxi)(240GB SSD on order for ZIL) and connected it to the ESXI HOST via the Infiniband. It is connected and I can add the storage to the hosts. I can migrate virtual machine to the data store successfully but it is way slow… 6000Kbps throughput. If I open a data store browser and try to move a ISO it is very slow, If I try to upload an ISO it succeeds but with the same throughput. But if I share the ESXI via SMB then I get 110 write/110 read copy files from my windows box to the share.

I am new to OmniOS, Napp-IT, ZFS and infiniband so I don’t think I am doing too bad, but don’t know what to do next to fix the performance issue.. Any ideas what to look for would be helpful.

 

[CreoleLakerFan]

You didn't specify, but since you mentioned VMware data stores I assume you are experiencing the slowness with an NFS/iSCSI connection via IP over Infiniband. If this is the case you will likely find your solution has to do with disabling synchronous writes for the

Putting the ZIL (SLOG) on the SSD you have ordered may help depending on how performant (new word!) the flash device is. If the slowness persists, you can either upgrade to a better model (many enjoy the Intel S3700) or disable sync writes entirely. This does come with a risk of data loss, so this is inadvisable to use in a production scenario, but could work for a study lab.

I am far from an expert on ZFS, but these suggestions should get you pointed in the right direction. Perhaps some others more knowledgeable about these things can weigh in with additional suggestions.

 

[cmpufxr]

I configure the connection via IB_SRPT with a ISCSI target that show up on ESXI as a SCSI.

 

[markpower28]

how did you install OmniOS on Compellent?

SAN/NAS

Dell Compellent Controller Head
24GB ECC RAM
4x 4TB HSGT SAS Drives
9x WD 250GB RE3 SATA Drives
2x LSI MegaRAID 9240-8i 8-port RAID cards
Mellanox MHQH29-XTC 40Gb/s Dual Port QFSP ConnectX

 

[cmpufxr]

The compellent controller is a supermicro system with scsi cards and a 16 bay SAS backplane. The sas backplane is powered but not connected so the drive bays do not work. The compellent software is loaded on a 4gb sata2 flash module.

I removed the flash module and put a ssd drive in its place. I also removed the scsi cards and added two lsi sas cards and connected the 16 drive bay to them. I loaded omnios on a USB3 flash drive.

From what I can tell the original compellent software (which was corrupt) may be a version of FreeBSD. At least that was what was trying to load from the 4gb flash module.

 

[markpower28]

Well done! Think of if you could reverse engineer a Dell/Supermicro server and turn that into a Compellent

Are you using 2.10 firmware on your ConnectX 2 card?

 

[cmpufxr]

Well done! Think of if you could reverse engineer a Dell/Supermicro server and turn that into a Compellent

Are you using 2.10 firmware on your ConnectX 2 card?

 

[cmpufxr]

You could, it is just software but the drives have to have the dell compellent firmware to work with the compellent software. or so i understand, I may be wrong.

Latest firmware on the connectx 2 cards and the latest firmware on the connectx card