ConnectX3 VPI on Solaris/Omnios

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

markpower28

Active Member
Apr 9, 2013
413
104
43
I tried a fresh install with Solaris 11.3 using ConnectX3 VPI connect to a 40Gbe switch, it shows IB as connection instead of ethernet. Same result with OmniOS.

Is there a way to change the ConnectX3 VPI to ethernet? It works fine in Windows OS.

Thanks in advance!
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
I do not use them but I when they are not on the HCL they are not supported

Without looking on quality/stability of drivers:
- on Windows everything has drivers
- on Linux, most parts have drivers
- on Unix (BSD, OSX, Solarish) you must check prior use
 
Last edited:

markpower28

Active Member
Apr 9, 2013
413
104
43

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
@markpower28

Had you tried running IPoIB? Or how did you end up?

Looking to speed up Solaris networking and passing through the card was one of my options bu I also only saw them as IB (and they don't show as such in NappIt)
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
and manual zfs config then i assume as Napp-It is not supported iirc?

Thanks
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Manual ZFS config is easy as long as it means create a pool and add vdevs or replace a disk.

But when it comes to the whole feature set that Sun and now Oracle, BSD, Illumos or ZoL had build into ZFS (snaps, replication, recovery, clone, hold, release, disk detection, ACL, SMB, NFS, iSCSI, virtual networking to name only a few) the associated knowledge covers a few meters of manuals and even experienced storage admins have difficulties to manage this from command line. This is why advanced storage appliances are so important (my napp-it, Nexentastor but also Freenas or others).
 

markpower28

Active Member
Apr 9, 2013
413
104
43
Gea, thanks for the follow up. I have been using Solaris/SRP until it's mission impossible for vSphere 6.5 due to driver stack change. I switch to the Linux based ZFS for iSER support for better Ethernet driver on the target and ESXi. I agreed with you in terms of performance and functionality, but Mellanox and VMware's road map does not have IB support with RDMA. Ethernet is the only way and there is no Ethernet driver for Solaris platform. I highly recommend you looking into further Linux development. (Once we teast the beauty of RDMA we can't go back...)

Manual ZFS config is easy as long as it means create a pool and add vdevs or replace a disk.

But when it comes to the whole feature set that Sun and now Oracle, BSD, Illumos or ZoL had build into ZFS (snaps, replication, recovery, clone, hold, release, disk detection, ACL, SMB, NFS, iSCSI, virtual networking to name only a few) the associated knowledge covers a few meters of manuals and even experienced storage admins have difficulties to manage this from command line. This is why advanced storage appliances are so important (my napp-it, Nexentastor but also Freenas or others).
 

Arthur Krewat

New Member
Apr 29, 2016
15
2
3
57
I'm very late to this, and I don't remember the specifics, but there is a way to set the "dual mode" ConnectX-3 to default to Ethernet, and this works on Solaris 11.3 and 11.4. I had to boot Linux, and run a Mellanox utility to set it. Once set, the card stays that way even if you power down and remove it from the server. Linux, and even Windows, seem to auto-switch to whatever is connected to it. Solaris, however, defaults to IB if the card is set to auto-detect.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
Info

Support for Mellanox ConnectX-4/5/6 NICs will be in next OmniOS 151034 (may 2020)
Connect X-3 is only supported in Solaris

omniosorg/omnios-build
 
Last edited:

markpower28

Active Member
Apr 9, 2013
413
104
43
very exciting news, is that means RDMA (iSER) finally comes to the stable platform for ZFS? :)
 
Last edited:

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Not sure you can deduct that - even CX3 had RDMA (RoCE) HW support (v2 on Pro).
That also needs software to support it, NIC/Driver only support is not sufficient unfortunately.
But given that RDMA support might be there due to IB one can hope :)
 

Arthur Krewat

New Member
Apr 29, 2016
15
2
3
57
Info

Support for Mellanox ConnectX-4/5/6 NICs will be in next OmniOS 151034 (may 2020)
Connect X-3 is only supported in Solaris

omniosorg/omnios-build
Official statement from Oracle about Connectx-4/5 support in Oracle Solaris: "Here is the current status in 2019:Currently only Mellanox CX-2/CX-3 are supported in Solaris11.4. Mellanox CX-4/CX-5 will be supported in future running in ethernet mode (not IB mode), and only in ZFSSA, and not for general use."

https://support.oracle.com/epmos/faces/DocContentDisplay?id=2610928.1

So if anyone has a newer ZFSSA box from Oracle and it uses a ConnectX-4 or 5, PM me ;)