Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

esses

New Member
Mar 12, 2018
16
4
3
48
10gbe performance iperf from Linux (Fedora 27) is AWFUL - 13 Mbit/s to the same client.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
weird, should not be that bad. My perf is significant better in FreeNas.
Can you do ESX to FN iperf check?
 

esses

New Member
Mar 12, 2018
16
4
3
48
Looks like it was the Finisar gbic / optical cable. Moving back to twinax gives me a more respectable iperf score and a file xfer in the 300-400mb/sec range.
 
  • Like
Reactions: Rand__

esses

New Member
Mar 12, 2018
16
4
3
48
ConnectX3 cards arrived, flashed in Windows 10 correct firmware aaand..

FreeNAS still comes up with a "NO IRQ's left" error message on passthrough, which is still my preferred solution. What a pain in the neck.
 

diskdiddler

Member
Mar 3, 2017
99
7
8
46
Did you write that FreeNAS Guide? Thank you.
I'm looking at going to 10G one day and I've been really adamant about having it over copper.
I've started re-thinking that in the last few days when I hear you can buy 10GB SFP cards for $40 on ebay and the modules can be cheap.
 

JustinClift

Member
Oct 5, 2014
35
14
8
@diskdiddler When you say adamant about having it over copper, are you meaning (say) direct connections between two computers only?

With the stuff I've been doing for the last year or so, it's mostly just direct connections between a FreeNAS box and various other PC's using direct attach cables. That's pretty easy to do, and yep the cards are very cheap on Ebay.
 

JustinClift

Member
Oct 5, 2014
35
14
8
Yeah, FreeNAS 9.3 is very, very old now. That didn't have the Mellanox drivers in it, so it's not surprising they weren't seen. Any of the recent FreeNAS releases (eg 11.x) should see them fine. If they're not seen with that, it'll probably just mean the firmware needs updating on the cards. There has been the occasional report like that, and it's 99.5% of the time due to really old firmware. Fixable by flashing.

Does that help? :)
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
I can confirm that the CX2 and CX3 10G cards work in Freenas as does the CX3 40G cards.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
also confirming Connectx3 40gbE cards work great in freenas, this is an iperf test from my hypervisor, through a stack of ICX6610's to the freenas box with 8 threads:

Code:
[SUM]  0.0-10.0 sec  46.1 GBytes  39.6 Gbits/sec
 

diskdiddler

Member
Mar 3, 2017
99
7
8
46
@diskdiddler When you say adamant about having it over copper, are you meaning (say) direct connections between two computers only?

With the stuff I've been doing for the last year or so, it's mostly just direct connections between a FreeNAS box and various other PC's using direct attach cables. That's pretty easy to do, and yep the cards are very cheap on Ebay.
I was not aware of the low price of fibre / sfp+ gear.
Looking up prices, the cheapest I can possibly get a 10Gbit 10-BASET card from Aquantia is about $80 US and I'm not convinced it's got FreeBSD support either.

Whereas if I get over my obsession with copper only, I could lay a second cable to my NAS (fibre) and use the existing copper for IPMI when I upgrade to a real NAS.

The Mellanox cards look to be older than Aquantia, supported, right now in FreeBSD and one of my 2 systems could just run DAC / Twinax, saving me even more money.
 
  • Like
Reactions: JustinClift

esses

New Member
Mar 12, 2018
16
4
3
48
So weirdly enough, I upgraded esxi to 6.7, completely reconfigured the network with seperate vmkernels and port groups, and now the dual 40gbe cards are humming along perfectly.

I think it might have been a combination VMware issue with their new html5 ui (because frankly the thing is full of very strange bugs all over the shop) and my own incompetence.

Passthrough on the other hand still doesn't work at all, with the same error message about IRQs.

I guess for my purposes this might work better since at least now the physical 40gbe links are also available to other VMs, not just Freenas.

Thanks everyone for taking the time to help!
 
  • Like
Reactions: JustinClift