ConnectX crash on freenas 11.1

Discussion in 'FreeBSD and FreeNAS' started by vrod, Feb 11, 2018.

  1. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    Hi folks,

    Just got 4 ConnectX dual 10gb sfp+ adapters and wanted to start building my new freenas box. Installation went fine, however I am getting kernel panics as soon as the dhclient tries to touch any of the adapters. having the mgmt 1gb configured, I re-inserted the adapters and the system boots up fine (as the dhclient now doesn't try to do anything with them anymore).

    The interfaces shows up in FreeNAS UI but as soon as I configure one of them, kernel panic again.

    Now, the seller stated it was "IBM Mellanox ConnectX EN Dual-Port 10GbE PCI-E x8 w/o GBIC - 59Y1906" adapters, however during bootup I see that the "VPI" driver is booted up? I also tried to use ESOS with these and there it looks like I am being given 4 Infiniband addresses (since I have 2 adapters with 2 ports).

    I have seen people having posted that the ConnectX adapters work in FreeNAS, but perhaps this was only for the "EN" version and not "VPI"? I guess I could return the cards and get some X520-DA2's instead but is there perhaps any way to flash these to "Ethernet-Only" mode? I do not wish to use Infiniband.
     
    #1
  2. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    Ok so by a little luck I had a single port ConnectX adapter laying around which I know is EN-only. This works just fine, I can configure an interface without having the system crash... Now the question is: can I flash the dual port adapters to EN-only mode or will it be more simple to just return them and get some "real" EN-adapters?

    On a side note, anyone here which would know why the HDD's in a SC847 chassis would only connect with SATA1-speeds? FreeNAS shows a speed for all the drives of 150MB/s. I am using a IBM M1115 adapter.
     
    #2
  3. Jannis Jacobsen

    Jannis Jacobsen Active Member

    Joined:
    Mar 19, 2016
    Messages:
    218
    Likes Received:
    34
    for Freenas Chelsio is the best choice

    -J
     
    #3
  4. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    #4
  5. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    I'm from europe and the prices here are a bit different from connectx to x-2. However I'll probably just return these cards... makes no sense to try anything with them.

    I did think about 40gbe adapters since they are not priced so much higher than the dual port 10gbe, however I do not have a 40gbe switch and I would want to run redundant lines from the servers to the storage box (a couple of esxi hosts, many ubuntu hosts). The use-case for all this is iSCSI storage.

    In regards to the sata speed, I looked around some more and it seems like I gotta get the SAS2-version of the expanders. https://forums.servethehome.com/index.php?threads/supermicro-backplane-firmwares-woes.3646/
     
    #5
  6. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    So am I, still would consider those I linked or similar ones ;) You can haggle a bit and still get those at €100 delivered and get way more recent tech (although some x4 cards just went out of support as well not sure it were these).
    Or get the X3 and get some QSFp->SFP adapters, they come up cheap every now and then. Maybe @zeynel has some left from hist last snag;)


    Backplane - just open the case and look at it ;) If it says BPN-SAS-xyz then its SAS1 and there is your reason, if it says BPN-SAS2-xyz then there is a link issue.
     
    #6
    Last edited: Feb 11, 2018
  7. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    it is a SAS1. :)

    I have seen a lot of posts about QSFP+ breakout cables, however this seems like something you can only do on the switch side. Would a ConnectX-3 40gb adapter be able to break out to 4x10gb links?
     
    #7
  8. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    #8
  9. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    hmmm i see.... just 30 bucks for a small piece of metal... that's something.

    I think I'll be good for now, I am zeroing 12 drives right now and are hitting 1150MB/s across the 2 backplanes. The box will be doing ZFS mirrors with a front-end optane cache and 750 ssd cache, so I think bandwidth-wise I'll be good for now. I'm gonna be relying a bit on the cache because of that but hopefully it will work. If not then I will get a SAS2 backplane.
     
    #9
  10. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    Thats why I said get them when they are cheap. Every now and then they go for 10 bucks or 15.

    Ok, hit me up if you need one, maybe I still have it by then;)
     
    #10
  11. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    I did look at the 40Gbe adapters but it seems like all on ebay are VPI-adapters and not pure ethernet as I'm looking for. I have found a "SuperMicro AOC-STGN-i2S" which is based on the Intel 82599ES adapter. I have a box running a Freenas 11.1 instance with this card without issues.... so that might be the most logical choice?
     
    #11
  12. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    VPI is theoretically the better version as it allows you to use either IB or EN.
    You should be able to set the type via the link I posted above, if need be on a windows or linux box.
     
    #12
  13. vrod

    vrod Active Member

    Joined:
    Jan 18, 2015
    Messages:
    204
    Likes Received:
    29
    There's too many types of adapters :D

    I don't know if I would be able to set this in FreeNAS... Would it work if I changed the mode in Windows and then inserted the adapter into the Freenas box? Would the change be persistent?
     
    #13
  14. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,235
    Likes Received:
    264
    if it gets flashed to the card then yes, if its a driver thing no.

    but there must be a lot of ppl using cx-3 cards in Freenas... I don't use them directly so have not tested it sorry
     
    #14
Similar Threads: ConnectX crash
Forum Title Date
FreeBSD and FreeNAS Mellanox CX4 ConnectX VPI cards and FreeNAS Jul 23, 2016
FreeBSD and FreeNAS Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS? May 3, 2015
FreeBSD and FreeNAS NAS4Free Mellanox ConnectX support missing in new version Feb 22, 2015

Share This Page