ConnectX crash on freenas 11.1

vrod

Active Member
Jan 18, 2015
233
35
28
29
Hi folks,

Just got 4 ConnectX dual 10gb sfp+ adapters and wanted to start building my new freenas box. Installation went fine, however I am getting kernel panics as soon as the dhclient tries to touch any of the adapters. having the mgmt 1gb configured, I re-inserted the adapters and the system boots up fine (as the dhclient now doesn't try to do anything with them anymore).

The interfaces shows up in FreeNAS UI but as soon as I configure one of them, kernel panic again.

Now, the seller stated it was "IBM Mellanox ConnectX EN Dual-Port 10GbE PCI-E x8 w/o GBIC - 59Y1906" adapters, however during bootup I see that the "VPI" driver is booted up? I also tried to use ESOS with these and there it looks like I am being given 4 Infiniband addresses (since I have 2 adapters with 2 ports).

I have seen people having posted that the ConnectX adapters work in FreeNAS, but perhaps this was only for the "EN" version and not "VPI"? I guess I could return the cards and get some X520-DA2's instead but is there perhaps any way to flash these to "Ethernet-Only" mode? I do not wish to use Infiniband.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
Ok so by a little luck I had a single port ConnectX adapter laying around which I know is EN-only. This works just fine, I can configure an interface without having the system crash... Now the question is: can I flash the dual port adapters to EN-only mode or will it be more simple to just return them and get some "real" EN-adapters?

On a side note, anyone here which would know why the HDD's in a SC847 chassis would only connect with SATA1-speeds? FreeNAS shows a speed for all the drives of 150MB/s. I am using a IBM M1115 adapter.
 

Rand__

Well-Known Member
Mar 6, 2014
6,088
1,463
113

vrod

Active Member
Jan 18, 2015
233
35
28
29
I'm from europe and the prices here are a bit different from connectx to x-2. However I'll probably just return these cards... makes no sense to try anything with them.

I did think about 40gbe adapters since they are not priced so much higher than the dual port 10gbe, however I do not have a 40gbe switch and I would want to run redundant lines from the servers to the storage box (a couple of esxi hosts, many ubuntu hosts). The use-case for all this is iSCSI storage.

In regards to the sata speed, I looked around some more and it seems like I gotta get the SAS2-version of the expanders. https://forums.servethehome.com/index.php?threads/supermicro-backplane-firmwares-woes.3646/
 

Rand__

Well-Known Member
Mar 6, 2014
6,088
1,463
113
So am I, still would consider those I linked or similar ones ;) You can haggle a bit and still get those at €100 delivered and get way more recent tech (although some x4 cards just went out of support as well not sure it were these).
Or get the X3 and get some QSFp->SFP adapters, they come up cheap every now and then. Maybe @zeynel has some left from hist last snag;)


Backplane - just open the case and look at it ;) If it says BPN-SAS-xyz then its SAS1 and there is your reason, if it says BPN-SAS2-xyz then there is a link issue.
 
Last edited:

vrod

Active Member
Jan 18, 2015
233
35
28
29
it is a SAS1. :)

I have seen a lot of posts about QSFP+ breakout cables, however this seems like something you can only do on the switch side. Would a ConnectX-3 40gb adapter be able to break out to 4x10gb links?
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
hmmm i see.... just 30 bucks for a small piece of metal... that's something.

I think I'll be good for now, I am zeroing 12 drives right now and are hitting 1150MB/s across the 2 backplanes. The box will be doing ZFS mirrors with a front-end optane cache and 750 ssd cache, so I think bandwidth-wise I'll be good for now. I'm gonna be relying a bit on the cache because of that but hopefully it will work. If not then I will get a SAS2 backplane.
 

Rand__

Well-Known Member
Mar 6, 2014
6,088
1,463
113
Thats why I said get them when they are cheap. Every now and then they go for 10 bucks or 15.

Ok, hit me up if you need one, maybe I still have it by then;)
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
I did look at the 40Gbe adapters but it seems like all on ebay are VPI-adapters and not pure ethernet as I'm looking for. I have found a "SuperMicro AOC-STGN-i2S" which is based on the Intel 82599ES adapter. I have a box running a Freenas 11.1 instance with this card without issues.... so that might be the most logical choice?
 

Rand__

Well-Known Member
Mar 6, 2014
6,088
1,463
113
VPI is theoretically the better version as it allows you to use either IB or EN.
You should be able to set the type via the link I posted above, if need be on a windows or linux box.
 

vrod

Active Member
Jan 18, 2015
233
35
28
29
There's too many types of adapters :D

I don't know if I would be able to set this in FreeNAS... Would it work if I changed the mode in Windows and then inserted the adapter into the Freenas box? Would the change be persistent?
 

Rand__

Well-Known Member
Mar 6, 2014
6,088
1,463
113
if it gets flashed to the card then yes, if its a driver thing no.

but there must be a lot of ppl using cx-3 cards in Freenas... I don't use them directly so have not tested it sorry