I tried all different options on the server, but each has its own problem.
FreeNAS - LACP issue (workaround to use load balancing, bug reported in 11.2Ux and upstream FreeBSD). NAS features (SMB/CIFS) are great. iSCSI performance is fantastic, but not able to delete a lun is a road blocker until it's fixed. (the bug was reported in 11.2Ux)
Linux - a few crashes on iSER, could be SCST version, haven't got enough time to figure out.
Solaris 11.4 - crash/coredump when Comstar lun is accessed.
In the end, I decided to virtualized the storage servers on ESXi since using the entire server for a storage server is overkill, and I can have multiple storage servers to serve different protocols.
It's getting really really really complicated now with the configuration. lol.
ESXi version - 6.7U3
Disk - 2*64GB Supermicro SataDom (planned for OS mirror but now runs ESXi).
NVME - HGST SN260 6.4TB AIC
SAS - 8*10TB HGST SAS connected to onboard SAS3008 for data zpool
CPU - EPYC 7302P
Memory - 4*32GB Micron PC3200. 128GB was planed for the storage server to start, will add another 128GB later since it's running ESXi now.
Motherboard - SuperMicro H11SSL-NC, onboard SAS3008 flashed to IT mode.
Network adapters. I have front end and backend storage traffic sperated on different nic and switches.
- Intel X550-T2 - didn't plan to use it, but this is the card I have besides X540-T2 that has SR-IOV and Virtual Function work and supported by guest VMs.
- HPE 530SFP (BCM57810) - supports SR-IOV and nic partitioning (4 Physical Function per port and 16 virtual function per PH.). Unfortunately wans't able to make SR-IOV work in guest except pass through the entire physical nic to guest. Thus, added X550.
- Melanox CX-3. - will passthrough to Linux if I want use iSER. SR-IOV is no longer supported in ESX 6.7 for CX-3 cards.
The mother BIOS needs some setting explicated on to make PCI passthrough work properly.
- IOMMU - Enable. Default is Auto
- ACS - Enable. Default Auto.
The SN260 AIC has the following namespace created
- ns1 600GB for ESXi local datastore
- ns2 1800GB for zpool (RDM to VM)
- ns3 1800GB for zpool (RDM to VM)
- ns4 1800GB for zpool (RDM to VM)
- ns5 24GB (RDM to VM) for slog for data pool (only 3 non-VM datastore NFS volumes will be using slog)
The namespace needs to be formated to 512B for ESXi to see them, default is 4K.
Only 1 NVME AIC is used but the VMs on them are non-critical and I have daily backup for the VMs. Will probably change to U2 NVME in the future since the motherboard has 2 onboard NVME ports.
In order to manage namespace, I need a Windows machine with HDM 3.4 and WD NVME driver. I build a Windows VM and passthrough the NVME to the VM and I was able to manage them.
Interesting performance stat here
- Read 6.2GB/s, write 2.2GB/s when the NVME card is passed through to VM directly
- Read 5.8GB/s write 2.2GB/s when namespace is RDMed to VM
- Read 1.7GB write 1.7GB when the disk/namespace is used as datastore and test inside VM
To be continued......
