Hi!
Thank you for your answer!
From ZFS-view, that is clear, but I do not fully understand the "VMWare-view". There seem to be different queues and VMWare recommends to use smaller LUNs instead of some big ones. For ZFS it should be equal to have on zvol on a pool instead of multiple ones.
Do you know, if VMWare handles 1 target with 5LUNs like 5 targets with 1 LUN?
In the esxi host, there is a queue depth limit per VM, per LUN, and per iscsi adapter.
In the driver:
Code:
esxcli system module list -m iscsi_vmk
iscsivmk_HostQDepth int Maximum Outstanding Commands Per Adapter
iscsivmk_LunQDepth int Maximum Outstanding Commands Per LUN
In the adapter:
Code:
esxcli iscsi adapter param get -A vmhba64
Name Current Default Min Max Settable Inherit
-------------------- ---------- ---------- --- -------- -------- -------
MaxCmds 128 128 2 2048 false false
In the LUN:
Code:
esxcli storage core device list --device naa.624a93704bfab20d4b694df7000143ac
Device Max Queue Depth: 113
No of outstanding IOs with competing worlds: 32
'outstanding IOs with competing worlds' is a per VM limit. Increasing it to the max (113 here) can cause very high latency, even if a single VM is active.
After trying to tune a bunch of these parameters, I've found only one parameter that I tune in general, and that is the multipathing. By default, only a single path is active when using multipath iscsi. You can make all paths active by setting the policy to round robin (below). By default, esxi switches path ever 1000 iops. Most of the storage vendors recommend switching paths every single iop (iops=1). I see better random mixed read/write IOPS with somewhere in between (e.g. iops=64). Here's how to set what the vendors recommend.
Code:
esxcli storage nmp device set -d vmhba64 --psp=VMW_PSP_RR
esxcli storage nmp psp roundrobin deviceconfig set -d vmhba64 -t iops -I 1
Lastly, note that you can multipath NFS 4.1, and it will increase the performance, even if the paths are on the same link, subnet, etc., as long is there's enough network bandwidth. With jumbo frames, I see multipath NFS 4.1 outperform mulitpath iscsi, plus you can use one huge store and it doesn't require any tuning.
