Mellanox ConnectX-2 not supported on new OmniOS HCL

[push3r]

I had recently purchased several ConnectX-2 MNPA19-XTR 10GbE SFP+ NICs to be used on ZFS storage with OmniOS and Napp-It. Before I purchased them several weeks ago, I saw that they were on the OmniOS/IllumOS HCL but now they are not.

I did some digging and some people did say that the ConnectX-2 are no longer supported. Is this true? If so, that really sucks! And my only option is Solaris 11? What about Solaris Express? It would obviously be nicer to use a free OS. I'm not too crazy about FreeNAS. I really need Solaris based ZFS for its ComStar iSCSI as I will also be installing a QLogic 8Gb Fiber Channel HBA on the same box presenting LUN Target (FC over Ethernet). I know this will work using the QLT drivers and I found many document on setting it up.

I am still waiting for the ConnectX-2 NICs to arrive in order to really test.

Anyone has an idea on this?

Thank you

 

[whitey]

There is a BIG difference between being 'supported' and simply working. Of course we'd prefer both! haha All I can advise is test and see. I have only ever used Intel/Broadcom nic's w/ phys OmniOS setups and use vmxnet3 vnics for my AIO Omni setups, all work very well.

I'd be interested in hearing your results or seeing what @gea weighs in with.

 

[push3r]

@whitey, Well, if they removed the ConnnectX-2 support then doesn't it means that they have also removed the drivers?

It would be great of @gea can jump in and shed some light.

 

[gea]

In my own setups I do not use any IB.
All is 10 Gb ethernet with a first 40 Gb switch.

IB on OmniOS is EoL. Only advantage are cheap offers.

Try it.
When it works, you can expect that the driver will work
in near future but no bugfixes and no support.

 

[whitey]

OUCH, wow Gea OmniOS has thrown in the towell on IB or is that the WHOLE Illumos dev community saying 'bigger fish to fry/higher priorities)...not a big issue for me as I am a 10GbE guy but have been thinking abt IB for 6 months or so...by the time another year or two rolls by I will prolly be 'over it' and see no benefit to IB as 40G ports become more plentiful/avail at reasonable pricepoints.

 

[gea]

This is not related to napp-it or OmniOS.
Hardware support is done either at Oracle or Illumos.

The IB support of Illumos is the same like years ago. Old hardware is supported by old drivers, newer hardware like Connectix X3 is not supported and will most propably never be supported, maybe from Oracle on Solaris but I would doubt that either Connectix or any of the enterprises behind Illumos like Delphix, Joyent, Nexenta or OmniTi have interest in IB on Illumos.

If you like it or not.
Ethernet is the future and mainstream .
On newer server boards its included without a serious premium over 1 Gb.
In one year you will hardly find a newer server mainboard without 10G Ethernet.

As long as there is interest in IB, I will try to support that in napp-it with some community help
but interest and request for IB is very low.

 

[push3r]

I'd just installed the newly arrived ConnectX-2 MNPA19-XTR on a fresh OmniOS latest stable release and no luck!

"prtconf -D" or "prtconf -D | grep ml" did not show anything Mellanox related. bummer!

I'll play with it some more, but as of right now, it doesn't look good.

On the other hand, my Qlogic FC HBA (QLE2560) card showed up with the QLC drivers. Note: Can disable this QLC and install/use QLT drivers to use for FC Target via iSCSI.

No other free solution for Solaris based ZFS now.

Now, I might have to go to FreeNAS 10 (in Alpha right now) where FreeBSD 10 has support for ConnectX-2 and Qlogic FC HBA.

 

[brianmat]

Looks like we will have to stick on our current versions of OmniOS as well. We just added ConnectX-2 cards due to the cost and speed . We'll be ok with not upgrading for a while as I ran into APD issues with the current OmniOS. It's still a bummer to hear this though.

 

[push3r]

Yeh, I spent some time yesterday searching and trying to get OmniOS (latest stable release) to see the ConnectX-2 card but no luck.

Today, I tried the same card on FreeBSD 10.2 Release and got it working with a few modifications with regard to enabling the mlx4, mlxen drivers and loader.conf file.

As for OmniOS options, you can get the Intel X520-DA1 card as I see it on the IllumOS HCL. It only costs a bit more.

Now, I'm gonna try Solaris 11.3 and see.

UPDATE: With Solaris 11.3, the ConnectX-2 card showed up immediately, no stress required.

 

[whitey]

This is not related to napp-it or OmniOS.
Hardware support is done either at Oracle or Illumos.

The IB support of Illumos is the same like years ago. Old hardware is supported by old drivers, newer hardware like Connectix X3 is not supported and will most propably never be supported, maybe from Oracle on Solaris but I would doubt that either Connectix or any of the enterprises behind Illumos like Delphix, Joyent, Nexenta or OmniTi have interest in IB on Illumos.

If you like it or not.
Ethernet is the future and mainstream .
On newer server boards its included without a serious premium over 1 Gb.
In one year you will hardly find a newer server mainboard without 10G Ethernet.

As long as there is interest in IB, I will try to support that in napp-it with some community help
but interest and request for IB is very low.

I concur, IB seems to still have a foothold/base in HPC and some stg cluster technologies for back-end interconnect but other than that I have lost my allure/fascination with it. Seems like a MAJOR PITA that I don't need.

10G/40G ethernet ftw :-D

 

[push3r]

Just pulled the trigger and got Intel X520-DA1 with Intel branded transceiver for $100 on the bay.

 

[whitey]

Any reason you didn't go w/ this?

Intel X520-DA1 E10G42BTDAG1P5 10GbE Ethernet Converged Network Adapter

I thought dual port X520-da2's went for abt a bill/$100?

Fake possibly, looks like a U.S. seller. Smokin' deal if they are legit!

EDIT: Another:

Intel X520-DA1 E10G41BTDA 10Gb Gbe Gbit 10 gigabit Network Adapter

 

[push3r]

Add intel branded transceiver, aroubd $30 bucks, which is supposedly required and you are up to around $70 bucks.

I looked and most dual ports, card only, are around $130. And then you have to add 2 intel branded transceivers, it's more like $200.

I'm buying for my work and saving a few bucks is not a big deal.

 

[push3r]

Btw, why do they make the transceivers removable? Why not just make it a permanent part of the card?

You can't do anything without the transceiver anyway. I know they want vendor lock in but the cards can be branded in the first place. Maybe a card can use for single and multimode so the transceivers dictate that? Well, again, specify that from the get go and people can buy either.

It's like having ethernet cards with removable RJ45.

It gets annoying really fast when you have to hunt for compatible transceivers.

 

[downtownHippie]

UPDATE: With Solaris 11.3, the ConnectX-2 card showed up immediately, no stress required.

So you're confirming the card that newegg has on sale (refurbed) for $24 works as a 10GB network option for Solaris 11.3?
Mellanox ConnectX-2 10GbE PCIe 2.0 x8 Low Profile Network Interface Card, MNPA19-XTR - Newegg.com
and $40 with a 3m cable
Refurbished: HP 10GbE Mellanox ConnectX-2 PCIe 2.0 x8 Standard Height Network Interface Card, 671798-001 666172-001 MNPA19-XTR with Cisco 3 Meter 10Gb Copper Twinax Cable SFP-H10GB-CU3M 37-0961-03 - Newegg.com

Do I need a "switch" in there or can I plug one card into the other or can I stick on in my server and another in my win10 desktop and have a 10gb point to point network?

 

[Hadrien]

Btw, why do they make the transceivers removable? Why not just make it a permanent part of the card?

You can't do anything without the transceiver anyway. I know they want vendor lock in but the cards can be branded in the first place. Maybe a card can use for single and multimode so the transceivers dictate that? Well, again, specify that from the get go and people can buy either.

It's like having ethernet cards with removable RJ45.

It gets annoying really fast when you have to hunt for compatible transceivers.

As far as I know, there is no way to have a transceiver that would support the myriad of connection options for 10Gb. You need options for singlemode fiber, a few for the different wavelengths of multimode fiber, direct attach copper, etc. (not sure what else is out there).

If you want something simpler, take a gander at 10Gb Base T NICs which use CAT6 or better for 10Gb connections. Last time we used them here though the fiber optic 10Gb cards seem to provide slightly lower latency as well as more throughput. 10Gb Base T should be good enough for any home network though.

 

[ehfortin]

Yeh, I spent some time yesterday searching and trying to get OmniOS (latest stable release) to see the ConnectX-2 card but no luck.

Today, I tried the same card on FreeBSD 10.2 Release and got it working with a few modifications with regard to enabling the mlx4, mlxen drivers and loader.conf file.

As for OmniOS options, you can get the Intel X520-DA1 card as I see it on the IllumOS HCL. It only costs a bit more.

Now, I'm gonna try Solaris 11.3 and see.

UPDATE: With Solaris 11.3, the ConnectX-2 card showed up immediately, no stress required.

Was moving to OmniOS latest because Solaris 11.3 doesn't support my LSI card in passthrough on ESXI 6U1... On this side of the fence, the LSI is working great but not my ConnectX-2 10 Gbps. So, it seems I'll move back to Solaris 11.3 but on physical where both card (LSI and Mellanox) work great. Bummer...