Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS?

Discussion in 'FreeBSD and FreeNAS' started by Jeggs101, May 3, 2015.

  1. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,381
    Likes Received:
    194
    Looking to see if anyone has tried these yet. I want to use a ConnectX-2 card in an upcoming FreeNAS box if this works.
     
    #1
  2. tjk

    tjk Member

    Joined:
    Mar 3, 2013
    Messages:
    159
    Likes Received:
    17
    I tried to get them working in FreeNAS 9.3, and FreeNAS does not see the card(s). Not sure if there are "hacks" to make em work, I tried the latest 9.3 vanilla.
     
    #2
  3. timothy.lazarus

    Joined:
    Mar 7, 2015
    Messages:
    46
    Likes Received:
    5
    No. The Mellanox drivers work fine in FreeBSD 10.1. Since FreeNAS is based on 9.x, the cards won't work. I am currently running my storage server with 10.1 and a ConnectX-2 card. I did have an odd iscsi hiccup once, but I think that was a hardware issue with one of the ports on the card and not a FreeBSD or driver issue. After running 2012 for a bit, FreeBSD 10.1 seems like a great storage server OS.
     
    #3
  4. tjk

    tjk Member

    Joined:
    Mar 3, 2013
    Messages:
    159
    Likes Received:
    17
    Are you doing IPoIB or NFS with FreeBSD 10.1? ZFS, etc?

    Do you know if 10.1 supports IB SRP?
     
    #4
  5. timothy.lazarus

    Joined:
    Mar 7, 2015
    Messages:
    46
    Likes Received:
    5
    My 2 cards are ConnectX-2 EN, so no Infiniband. I'm using iSCSI right now. I don't believe SRP is supported, but RoCE may be coming if it isn't already supported:

    SRP + Infiniband + FreeBSD 10

    I am a bit frustrated with Mellanox's driver support so far, however. According to the spec sheet, my cards support RoCE. However, I was never able to get it working in 2012, and I don't believe anyone else here was either. The current plan is to upgrade to ConnectX-3 cards once they come down in price a little more.
     
    #5
  6. Baddreams

    Baddreams New Member

    Joined:
    Mar 15, 2011
    Messages:
    21
    Likes Received:
    7
    I'm in the same boat as you. If you look closely at the release notes for RoCE support you'll see it says ConnectX-3 only.

    I'd like to see iSER support in FreeBSD but who knows when that will happen. Someone started it, you can see the configuration options for it in ctl.conf, but there isn't anything in the kernel to make use of it.

    I've tried pretty much every combination of hosts to get iSER working including ESXi 5.5, FreeBSD 10.0/1, Linux (Ubuntu 14.04, LIO target), and XenServer 6.1. In the end, the only combination I could get to work was two Linux hosts talking to each other, then iSER worked. And it was glorious. Never seen performance like that.

    Couldn't get SMBDirect working between two Windows Server 2012 R2 hosts.

    I think these cards are just a bit too old for Mellanox to put much effort into supporting the fancy features.
     
    #6
    timothy.lazarus likes this.
  7. Baddreams

    Baddreams New Member

    Joined:
    Mar 15, 2011
    Messages:
    21
    Likes Received:
    7
    Support for these cards has been in FreeBSD for quite some time, and in 9.X for sure because I was using mine back then. You might have to compile a custom kernel for FreeNAS though.
     
    #7
  8. neo

    neo Well-Known Member

    Joined:
    Mar 18, 2015
    Messages:
    671
    Likes Received:
    359
    For current versions of pfSense and FreeNAS they don't work out the box.
     
    #8
    Last edited: May 12, 2015
  9. Baddreams

    Baddreams New Member

    Joined:
    Mar 15, 2011
    Messages:
    21
    Likes Received:
    7
    Which is why I said you may have to compile a custom kernel. And by custom I mean add two lines to the kernel conf. Is that activity not allowed in FreeNAS?
     
    #9
  10. neo

    neo Well-Known Member

    Joined:
    Mar 18, 2015
    Messages:
    671
    Likes Received:
    359
    That is a cumbersome solution upgrading. I personally prefer native compatibility.
     
    #10
  11. Baddreams

    Baddreams New Member

    Joined:
    Mar 15, 2011
    Messages:
    21
    Likes Received:
    7
    Bum deal.
     
    #11
  12. mervincm

    mervincm Active Member

    Joined:
    Jun 18, 2014
    Messages:
    138
    Likes Received:
    30
    I recall it is included in a test build, you might want to try that.
     
    #12
  13. Levi

    Levi Member

    Joined:
    Mar 2, 2015
    Messages:
    56
    Likes Received:
    2
    What 2 lines?
     
    #13
  14. BackupProphet

    BackupProphet Active Member

    Joined:
    Jul 2, 2014
    Messages:
    641
    Likes Received:
    222
    You don't need to compile a custom kernel, you need to compile the driver though. In FreeBSD it is located at
    /usr/src/sys/modules/mlxen

    then you write:
    make && make install && kldload mlxen

    And to load the driver for each reboot you add the following to /boot/loader.conf
    mlxen_load="YES"
     
    #14
    Last edited: Feb 4, 2016
    Lighti likes this.
  15. Lighti

    Lighti New Member

    Joined:
    Apr 2, 2016
    Messages:
    1
    Likes Received:
    3
    Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).

    I created a new (!) jail in FreeNAS 9.10, used only make in the mlxen directory you pointed out and copied the created mlxen.ko out of the jail to one of my FreeNAS datasets.

    I also needed to compile the mlx4 driver to get the mlx4.ko (mlxen.ko would not start without) under the folder:
    /usr/src/sys/modules/mlx4

    With the mlxen.ko and mlx4.ko in one of my datasets under FreeNAS, i then opened a FreeNAS shell and installed them with these commands:

    install -o root -g wheel -m 555 mlxen.ko /boot/kernel
    install -o root -g wheel -m 555 mlx4.ko /boot/kernel
    kldxref /boot/kernel

    Afterwards, i could load the mlxen.ko sucessfully with

    kldload mlxen.ko

    Because my connectx-2 card is not shipped to me yet, i can not verify if this really works.
    But thats the most promising solution to the mellanox connectx-2 & freenas problem i have found so far...
     
    #15
    amatveev, ttabbal and Jannis Jacobsen like this.
  16. rockitlikeithott

    Joined:
    May 8, 2015
    Messages:
    142
    Likes Received:
    24
    Can confirm this works with a connectX-3 40gbe HP LOM.
    As an aside. You don't need to copy anything over. You can navigate into the jails directory where the the .ko files are located and just run the commands above straight from there. Also add the mlxen.ko to the loader.conf so its persistent across reboots. Thanks for your post though. Made it very easy to get this card working.
     
    #16
  17. amatveev

    amatveev New Member

    Joined:
    May 4, 2016
    Messages:
    4
    Likes Received:
    0
    I've tried to do the same operation for ConnectX-2 VDI in IB mode without success.
    I've compiled mlx4.ko, ml4ib.ko and ibcore.ko, installed them with:
    install -o root -g wheel -m 555 mlx4ib.ko /boot/kernel
    install -o root -g wheel -m 555 mlx4.ko /boot/kernel
    install -o root -g wheel -m 555 ibcore.ko /boot/kernel
    kldxref /boot/kernel

    But when I try to: kldload mlx4ib.ko
    I get:
    mlx4_core0: <mlx4_core> mem 0xfbb00000-0xfbbfffff,0xfa800000-0xfaffffff irq 16 at device 0.0 on pci3
    mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (May 4 2016)

    link_elf_obj: symbol ipport_firstauto undefined
    linker_load_file: Unsupported file type
    KLD mlx4ib.ko: depends on ibcore - not available or version mismatch
    mlx4_core0: detached
    pci3: <serial bus> at device 0.0 (no driver attached)
    linker_load_file: Unsupported file type

    Any ideas how this could be fixed?
     
    #17
  18. BackupProphet

    BackupProphet Active Member

    Joined:
    Jul 2, 2014
    Messages:
    641
    Likes Received:
    222
    You need to install the ibcore kernel module.

    /usr/src/sys/modules/ibcore/
     
    #18
  19. amatveev

    amatveev New Member

    Joined:
    May 4, 2016
    Messages:
    4
    Likes Received:
    0
    It's done with: install -o root -g wheel -m 555 ibcore.ko /boot/kernel
    (see my command log)

    But when I try to load it with: kldload ibcore.ko

    link_elf_obj: symbol ipport_firstauto undefined
    linker_load_file: Unsupported file type
     
    #19
  20. BackupProphet

    BackupProphet Active Member

    Joined:
    Jul 2, 2014
    Messages:
    641
    Likes Received:
    222
    You need to renew module references
    Code:
    kldxref /boot/kernel


    kldxref(8)
     
    #20
Similar Threads: Mellanox ConnectX-2
Forum Title Date
FreeBSD and FreeNAS Mellanox CX4 ConnectX VPI cards and FreeNAS Jul 23, 2016
FreeBSD and FreeNAS NAS4Free Mellanox ConnectX support missing in new version Feb 22, 2015

Share This Page