iSCSI ZFS block storage options for ESXi with RDMA / RoCE / iSER support

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
L

lunadesign

Active Member
Aug 7, 2013
256
34
28
I'm currently using FreeNAS for iSCSI ZFS block storage for my ESXi server. The FreeNAS box is getting old so I'm looking for a replacement. I'm not currently using any of the other FreeNAS features besides ZFS and iSCSI.

I was initially planning to upgrade the FreeNAS box to TrueNAS Core or TrueNAS Scale but it appears neither has RDMA support. I have a pair of ConnectX-5 100G adapters that I plan to use to connect the iSCSI box to ESXi so I'd really like to take advantage of RDMA to get better performance and less CPU utilization.

As a proof-of-concept, I set up an Ubuntu 22.04.1 LTS box with ZFS and LinuxIO (LIO) / targetcli with iSER. I need to do some more testing, but so far, it does look better than regular iSCSI.

Are there any other options I should be looking at?

Are there any downsides to using a general purpose OS (i.e., Ubuntu) vs a storage-focused platform (i.e., TrueNAS) besides losing the GUI?
 
  • Like
Reactions: sprice82
L

lunadesign

Active Member
Aug 7, 2013
256
34
28
BackupProphet said:
With Linux you can also use NVMe-oF for block devices. It should provide slightly better performance on SAS/SATA devices, and massive performance boost for NVMe devices.
Click to expand...
That's a very intriguing idea, especially now that ESXi has NVMe-oF support in ESXi 7.

Can NVMe-oF serve SATA/SAS devices? I wasn't aware of that.

What Linux package would you recommend for the NVMe-oF target? The only one I've found so far is from SPDK.io.
 
You must log in or register to reply here.
Top Bottom