Options for Illumos based storage server with RDMA?

Discussion in 'Solaris, Nexenta, OpenIndiana, and napp-it' started by dswartz, Sep 26, 2018.

  1. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    376
    Likes Received:
    28
    I currently have an NFS server running ZoL 0.7.11. Works okay, but I'd prefer some Illumos based distro. My ESXi host is connected to the storage server using a point to point QSFP cable and 2 Mellanox Connectx-3 EN cards. Works okay, but I'd like to dabble with RDMA. The most visible products are Mellanox Connectx-4 LX cards, but Mellanox seems long ago to have given up any interesting in any Solaris based product. Chelsio makes a couple of interesting cards (t5 and t6 asics), but chelsio seems to be riding the iwarp bandwagon, which is broken down by the side of the road (even Intel seems to have ditched it.) So now, chelsio is pushing things like iSCSI offload, although I guess you might be able to do RDMA NFS from ESXi => Solarish server. But vmware has stated that from 6.7 forward, they will only support RoCE (implying no support for iWARP). If illumos provides drivers for esxi and solarish, I guess I'd be fine with that, but it's hard to tell from reading illumos HCL what they in fact support. The chelsio driver claims to support everything up to t6 asic, but that doesn't mean it supports all the fancy offload stuff. Sorry for the TL;DR :) If anyone has any concrete recommendations, I'd much appreciate them. If I can't find out anything useful, I might need to stick with Linux and go with the recent Mellanox cards.
     
    #1
  2. gea

    gea Well-Known Member

    Joined:
    Dec 31, 2010
    Messages:
    2,241
    Likes Received:
    742
    Solaris and the free Illumos fork have many advantages, wide hardware support is not among. I doubt that you will find support for newer Mellanox cards while Chelsio is quite well supported on Illumos.

    For your quite special questions you should ask at the Illumos-discuss maillist where you have a chance that a dev that is involved in driver development may answer. (ex Joyent who added Chelsio T6 support)
     
    #2
  3. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    376
    Likes Received:
    28
    Yeah, I'm leaning toward chelsio, but I need to know if they support RDMA/iSER. I will check out the mailing list, as that seems like the best option to get useful info. Thanks!
     
    #3
  4. zxv

    zxv The more I C, the less I see.

    Joined:
    Sep 10, 2017
    Messages:
    152
    Likes Received:
    46
    I've been running both ZoL and Omniosce (OmniOS Community Edition) for some time. Omniosce is an illumos distro that initially focused on the storage server OS, but has moved closer to illumos and now focuses on general server OS.

    I'm just beginning to look at doing NFS over RDMA on Omniosce. I don't yet have any info on which NICs work best, and that's a question I'm interested in and will almost certainly explore.
     
    #4
Similar Threads: Options Illumos
Forum Title Date
Solaris, Nexenta, OpenIndiana, and napp-it ZFS Tuning options Mar 28, 2013
Solaris, Nexenta, OpenIndiana, and napp-it SMB3 development on Illumos (NexentaStor, OmniOS, OpenIndiana and SmartOS) Aug 10, 2019
Solaris, Nexenta, OpenIndiana, and napp-it Native (Open-) ZFS encryption is in Illumos now Jun 26, 2019
Solaris, Nexenta, OpenIndiana, and napp-it Shout out to OmniOS / IllumOS platform Apr 22, 2019
Solaris, Nexenta, OpenIndiana, and napp-it CIFS Illumos Kernel Server and Permissions on Windows Jul 14, 2016

Share This Page