Options for Illumos based storage server with RDMA?

dswartz

Active Member
Jul 14, 2011
389
33
28
I currently have an NFS server running ZoL 0.7.11. Works okay, but I'd prefer some Illumos based distro. My ESXi host is connected to the storage server using a point to point QSFP cable and 2 Mellanox Connectx-3 EN cards. Works okay, but I'd like to dabble with RDMA. The most visible products are Mellanox Connectx-4 LX cards, but Mellanox seems long ago to have given up any interesting in any Solaris based product. Chelsio makes a couple of interesting cards (t5 and t6 asics), but chelsio seems to be riding the iwarp bandwagon, which is broken down by the side of the road (even Intel seems to have ditched it.) So now, chelsio is pushing things like iSCSI offload, although I guess you might be able to do RDMA NFS from ESXi => Solarish server. But vmware has stated that from 6.7 forward, they will only support RoCE (implying no support for iWARP). If illumos provides drivers for esxi and solarish, I guess I'd be fine with that, but it's hard to tell from reading illumos HCL what they in fact support. The chelsio driver claims to support everything up to t6 asic, but that doesn't mean it supports all the fancy offload stuff. Sorry for the TL;DR :) If anyone has any concrete recommendations, I'd much appreciate them. If I can't find out anything useful, I might need to stick with Linux and go with the recent Mellanox cards.
 

gea

Well-Known Member
Dec 31, 2010
2,438
818
113
DE
Solaris and the free Illumos fork have many advantages, wide hardware support is not among. I doubt that you will find support for newer Mellanox cards while Chelsio is quite well supported on Illumos.

For your quite special questions you should ask at the Illumos-discuss maillist where you have a chance that a dev that is involved in driver development may answer. (ex Joyent who added Chelsio T6 support)
 

dswartz

Active Member
Jul 14, 2011
389
33
28
Yeah, I'm leaning toward chelsio, but I need to know if they support RDMA/iSER. I will check out the mailing list, as that seems like the best option to get useful info. Thanks!
 

zxv

The more I C, the less I see.
Sep 10, 2017
153
51
28
I've been running both ZoL and Omniosce (OmniOS Community Edition) for some time. Omniosce is an illumos distro that initially focused on the storage server OS, but has moved closer to illumos and now focuses on general server OS.

I'm just beginning to look at doing NFS over RDMA on Omniosce. I don't yet have any info on which NICs work best, and that's a question I'm interested in and will almost certainly explore.