ConnectX3 VPI on Solaris/Omnios

[markpower28]

I tried a fresh install with Solaris 11.3 using ConnectX3 VPI connect to a 40Gbe switch, it shows IB as connection instead of ethernet. Same result with OmniOS.

Is there a way to change the ConnectX3 VPI to ethernet? It works fine in Windows OS.

Thanks in advance!

 

[gea]

I do not use them but I when they are not on the HCL they are not supported

Without looking on quality/stability of drivers:
- on Windows everything has drivers
- on Linux, most parts have drivers
- on Unix (BSD, OSX, Solarish) you must check prior use

 

[markpower28]

Thanks Gea, looks like this will not work on Solaris and OmniOS. I tried 11.3 and 11.4 beta both shows as InfiniBand same as OmniOS. I loaded Ubuntu 18.04 without any issues. NIC shows as 40Gbe, the problem I am facing now is after install napp-it // webbased ZFS NAS/SAN appliance for OmniOS, OpenIndiana, Solaris and Linux : Linux it does not show iSCSI target (I installed TGT). How can I make iSCSI target/LUN configuration shows up in Napp-It GUI?

 

[gea]

The iSCSI functionality in napp-it is based on Comstar
Configuring iSCSI Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems

This framework is part of the Solaris (and Illumos) OS. The Linux version of napp-it
supports only features that are common between Solaris and Linux like ZFS and Jobfeatures.

 

[Rand__]

@markpower28

Had you tried running IPoIB? Or how did you end up?

Looking to speed up Solaris networking and passing through the card was one of my options bu I also only saw them as IB (and they don't show as such in NappIt)

 

[markpower28]

IPoIB's performance sub-par compares to any RDMA solution. I have been enjoying RDMA via SRP for vSpehre until 6.0. With vSphere driver stack change, the only RDMA option for 6.5 and above is iSER. vSphere 6.7 has the iSER build-in which leaves us not many options.

This is what I end up having.
vSphere with iSER - How to release the full potential of your iSCSI storage! - VMware VROOM! Blog

 

[Rand__]

Using ZoL? or what FS/OS for the storage box?

 

[markpower28]

Ubuntu 18.04 LTS server with ZFS and TGT for iSER.

 

[Rand__]

and manual zfs config then i assume as Napp-It is not supported iirc?

Thanks

 

[markpower28]

correct, manual zfs config

 

[gea]

Manual ZFS config is easy as long as it means create a pool and add vdevs or replace a disk.

But when it comes to the whole feature set that Sun and now Oracle, BSD, Illumos or ZoL had build into ZFS (snaps, replication, recovery, clone, hold, release, disk detection, ACL, SMB, NFS, iSCSI, virtual networking to name only a few) the associated knowledge covers a few meters of manuals and even experienced storage admins have difficulties to manage this from command line. This is why advanced storage appliances are so important (my napp-it, Nexentastor but also Freenas or others).

 

[Rand__]

Yeah.
Too bad Napp-It is not supported on Linux

 

[markpower28]

Gea, thanks for the follow up. I have been using Solaris/SRP until it's mission impossible for vSphere 6.5 due to driver stack change. I switch to the Linux based ZFS for iSER support for better Ethernet driver on the target and ESXi. I agreed with you in terms of performance and functionality, but Mellanox and VMware's road map does not have IB support with RDMA. Ethernet is the only way and there is no Ethernet driver for Solaris platform. I highly recommend you looking into further Linux development. (Once we teast the beauty of RDMA we can't go back...)

Manual ZFS config is easy as long as it means create a pool and add vdevs or replace a disk.

But when it comes to the whole feature set that Sun and now Oracle, BSD, Illumos or ZoL had build into ZFS (snaps, replication, recovery, clone, hold, release, disk detection, ACL, SMB, NFS, iSCSI, virtual networking to name only a few) the associated knowledge covers a few meters of manuals and even experienced storage admins have difficulties to manage this from command line. This is why advanced storage appliances are so important (my napp-it, Nexentastor but also Freenas or others).

 

[Arthur Krewat]

I'm very late to this, and I don't remember the specifics, but there is a way to set the "dual mode" ConnectX-3 to default to Ethernet, and this works on Solaris 11.3 and 11.4. I had to boot Linux, and run a Mellanox utility to set it. Once set, the card stays that way even if you power down and remove it from the server. Linux, and even Windows, seem to auto-switch to whatever is connected to it. Solaris, however, defaults to IB if the card is set to auto-detect.

 

[gea]

Info

Support for Mellanox ConnectX-4/5/6 NICs will be in next OmniOS 151034 (may 2020)
Connect X-3 is only supported in Solaris

omniosorg/omnios-build

 

[markpower28]

very exciting news, is that means RDMA (iSER) finally comes to the stable platform for ZFS?

 

[Rand__]

Not sure you can deduct that - even CX3 had RDMA (RoCE) HW support (v2 on Pro).
That also needs software to support it, NIC/Driver only support is not sufficient unfortunately.
But given that RDMA support might be there due to IB one can hope

 

[Arthur Krewat]

Info

Support for Mellanox ConnectX-4/5/6 NICs will be in next OmniOS 151034 (may 2020)
Connect X-3 is only supported in Solaris

omniosorg/omnios-build

Official statement from Oracle about Connectx-4/5 support in Oracle Solaris: "Here is the current status in 2019:Currently only Mellanox CX-2/CX-3 are supported in Solaris11.4. Mellanox CX-4/CX-5 will be supported in future running in ethernet mode (not IB mode), and only in ZFSSA, and not for general use."

https://support.oracle.com/epmos/faces/DocContentDisplay?id=2610928.1

So if anyone has a newer ZFSSA box from Oracle and it uses a ConnectX-4 or 5, PM me