This is a continuation of previous discussion here:
NFSv3 vs NFSv4 vs iSCSI for ESXI datastores
OK, so there is good news and bad news about connectx-3 and iser on esxi.
First the bad news.
The bad news is, there are issues with flow control which might or might not be able to be resolved. More importantly, when iser hangs on esxi, it is often fatal to the extent that the system cannot boot. So use this info at your own risk, with the following warning:
If the iSER client or target is mis-configured in such a way that the LUN attaches but cannot transfer data, it may hang the system. It may then hang during shutdown, and at reboot. This can require a reinstall of esxi (and motivate use of kickstart installs).
If the client connects to a target, but cannot send/receive RDMA, or the target or lun go offline:
- Certain functions in esxi hang, including any file-system access.
- Esxi will hang during shutdown at "stopping scsi periodic probe".
- Esxi will hang during boot. It will not reach a shell or a prompt.
This can occur when a roce V1 client tries to connect to a roce V2 only server. It will silently fail.
Even with a valid configuration, if an iser datastore is mounted when shutdown occurs, the system will hang at "stopping scsi periodic probe". I just hit reset.
So, on to the good news.
The inbox drivers and iSER does work with CX3, on both RoCEv1 and v2.
One of the most crucial peices of missing information, for me, was, after a static discovery target is configured, next, one or more VMkernel adapters (vmkN) must be 'bound' to the adapter, and each of the vmkN must be tagged 'VSAN' (see 'interface tag add' below).
Second, if you add only one path to an iscsi target, the device may be marked 'degraded' because it has nofailover candidate; however, it will work. Just be aware that 'degraded' does not mean dead. (see 'statictarget add' below)
If you have a CX3Pro, consider using RoCE V2 if possible. It provides better performance and far more consistent bandwidth/throughput than V1.
I found no RDMA performance or troubleshooting tools for esxi 6.7. Mellanox has a installable RDMA sniffer, but it doesn not work with cx3/3Pro cards.
You need either global flow control (pause all traffic on switch), or better yet, priority flow control, which pauses only the recipient of the pause request. I have tested primarily using RoCEv2 and CX3 Pro cards connected to an Arista DCS-7050QX.
There are still flow control issues even with the configuration shown below, however given the lack of troubleshooting tools, I haven't yet been able to identify whether the issue is on the initiator or target side. Capturing RoCE traffic would help. I haven't yet done that, but I didn't want to delay getting this info out any further.
Here is some example steps for configuring iser on esxi 6.7 with a CX3 Pro card, just to fill in some of the gaps discussed above. This installs no additional drivers. It uses only the inbox drivers provided by the install CD. Be careful if you attempt these, and consider using kickstart installs if you want to pursue it.
Resources:
https://community.mellanox.com/s/article/what-is-iser-x
https://community.mellanox.com/s/ar...r-for-vmware-vsphere-esxi-6-5-6-7-host--cli-x
http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers
http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_ESXi_User_Manual_v2.4.0.pdf
NFSv3 vs NFSv4 vs iSCSI for ESXI datastores
OK, so there is good news and bad news about connectx-3 and iser on esxi.
First the bad news.
The bad news is, there are issues with flow control which might or might not be able to be resolved. More importantly, when iser hangs on esxi, it is often fatal to the extent that the system cannot boot. So use this info at your own risk, with the following warning:
If the iSER client or target is mis-configured in such a way that the LUN attaches but cannot transfer data, it may hang the system. It may then hang during shutdown, and at reboot. This can require a reinstall of esxi (and motivate use of kickstart installs).
If the client connects to a target, but cannot send/receive RDMA, or the target or lun go offline:
- Certain functions in esxi hang, including any file-system access.
- Esxi will hang during shutdown at "stopping scsi periodic probe".
- Esxi will hang during boot. It will not reach a shell or a prompt.
This can occur when a roce V1 client tries to connect to a roce V2 only server. It will silently fail.
Even with a valid configuration, if an iser datastore is mounted when shutdown occurs, the system will hang at "stopping scsi periodic probe". I just hit reset.
So, on to the good news.
The inbox drivers and iSER does work with CX3, on both RoCEv1 and v2.
One of the most crucial peices of missing information, for me, was, after a static discovery target is configured, next, one or more VMkernel adapters (vmkN) must be 'bound' to the adapter, and each of the vmkN must be tagged 'VSAN' (see 'interface tag add' below).
Second, if you add only one path to an iscsi target, the device may be marked 'degraded' because it has nofailover candidate; however, it will work. Just be aware that 'degraded' does not mean dead. (see 'statictarget add' below)
If you have a CX3Pro, consider using RoCE V2 if possible. It provides better performance and far more consistent bandwidth/throughput than V1.
I found no RDMA performance or troubleshooting tools for esxi 6.7. Mellanox has a installable RDMA sniffer, but it doesn not work with cx3/3Pro cards.
You need either global flow control (pause all traffic on switch), or better yet, priority flow control, which pauses only the recipient of the pause request. I have tested primarily using RoCEv2 and CX3 Pro cards connected to an Arista DCS-7050QX.
There are still flow control issues even with the configuration shown below, however given the lack of troubleshooting tools, I haven't yet been able to identify whether the issue is on the initiator or target side. Capturing RoCE traffic would help. I haven't yet done that, but I didn't want to delay getting this info out any further.
Here is some example steps for configuring iser on esxi 6.7 with a CX3 Pro card, just to fill in some of the gaps discussed above. This installs no additional drivers. It uses only the inbox drivers provided by the install CD. Be careful if you attempt these, and consider using kickstart installs if you want to pursue it.
Code:
# warning! This can and may hang your system if mis-configured.
esxcli system module parameters set -m nmlx4_en -p "pfctx=0x08 pfcrx=0x08"
esxcli system module parameters set -m nmlx4_core -p "enable_qos=1 enable_rocev2=1"
esxcli system module parameters set -m nmlx4_rdma -p "pcp_force=3 dscp_force=24 dscp_to_pcp=0"
esxcli network firewall ruleset set -e true -r iSCSI
esxcli network firewall ruleset set -e true -r pvrdma
esxcli iscsi software set -e true
esxcli system module set --enabled=true --module=iser
vmkload_mod iser
esxcli rdma iser add
/sbin/auto-backup.sh
reboot
esxcli network ip interface tag add -i vmk4 -t VSAN
esxcli iscsi networkportal add -A vmhba67 -n vmk4
esxcli iscsi adapter discovery statictarget add -A vmhba67 -a 172.16.1.99 -n iqn.2001-04.com.my-server
esxcli iscsi adapter discovery statictarget add -A vmhba67 -a 172.16.2.99 -n iqn.2001-04.com.my-server
esxcli iscsi adapter discovery rediscover -A vmhba67
esxcli storage core device list
esxcli storage vflash device list
https://community.mellanox.com/s/article/what-is-iser-x
https://community.mellanox.com/s/ar...r-for-vmware-vsphere-esxi-6-5-6-7-host--cli-x
http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers
http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_ESXi_User_Manual_v2.4.0.pdf
Last edited: