Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS?

Discussion in 'FreeBSD and FreeNAS' started by Jeggs101, May 3, 2015.

  1. amatveev

    amatveev New Member

    Joined:
    May 4, 2016
    Messages:
    4
    Likes Received:
    0
    It's done right after I install all the drivers:

    install -o root -g wheel -m 555 mlx4ib.ko /boot/kernel
    install -o root -g wheel -m 555 mlx4.ko /boot/kernel
    install -o root -g wheel -m 555 ibcore.ko /boot/kernel
    kldxref /boot/kernel

    Problem is here:
    [root@storage-b] /mnt/tank/temp/install# kldload mlx4ib.ko
    kldload: mlx4ib.ko will be loaded from /boot/kernel, not the current directory
    kldload: an error occurred while loading the module. Please check dmesg(8) for more details.

    [root@storage-b] /mnt/tank/temp/install# dmesg
    ...
    mlx4_core0: <mlx4_core> mem 0xfbb00000-0xfbbfffff,0xfa800000-0xfaffffff irq 16 at device 0.0 on pci3
    mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (May 4 2016)
    link_elf_obj: symbol ipport_firstauto undefined
    linker_load_file: Unsupported file type
    KLD mlx4ib.ko: depends on ibcore - not available or version mismatch
    mlx4_core0: detached
    pci3: <serial bus> at device 0.0 (no driver attached)
    linker_load_file: Unsupported file type
     
    #21
  2. BackupProphet

    BackupProphet Active Member

    Joined:
    Jul 2, 2014
    Messages:
    662
    Likes Received:
    230
    Are you sure that the /usr/src tree is aligned with your kernel?
    Maybe this page can help you, InfiniBand - FreeBSD Wiki
     
    #22
  3. amatveev

    amatveev New Member

    Joined:
    May 4, 2016
    Messages:
    4
    Likes Received:
    0
    Could you give me a hint how to check that out?

    I'm trying to do it on FreeNas 9.10 system, so FreeBSD WIki is not very useful (((
     
    #23
  4. JustinClift

    JustinClift New Member

    Joined:
    Oct 5, 2014
    Messages:
    24
    Likes Received:
    11
    If it's of any interest, support for Mellanox adapters was added into the main development code for FreeNAS 9.10 (and upcoming 10) a few days ago. It's available in the official 9.10 nightlies:

    Index of /9.10/MASTER/

    Grab anything from about 201605221730 onwards (or just grab the latest), and you should be good.

    For native IB mode support, that will take further effort. Some work is happening unofficially here, but it's not as far along and I don't have tonnes of time spare for it at the moment:

    GitHub - justinclift/freenas-infiniband: FreeNAS 9.10/10 with drivers for Mellanox Ethernet & Infiniband adapters

    Feedback on the ISO available there for download is welcome though. :)
     
    #24
    Last edited: May 29, 2016
    Patrick and Aestr like this.
  5. inbusiness

    inbusiness New Member

    Joined:
    Jul 31, 2016
    Messages:
    5
    Likes Received:
    3
    Mellanox FreeBSD drver 2.1.5 is a last support for ConnectX-2 VPI HCA.
    You must use a driver 2.1.5, not 2.1.6 that support ConnectX-3 or above only.
     
    #25
  6. JustinClift

    JustinClift New Member

    Joined:
    Oct 5, 2014
    Messages:
    24
    Likes Received:
    11
    That might be true for the driver manually downloaded from the Mellanox website, but it's not the case for the Mellanox driver in mainline FreeBSD. That supports the adapters listed here:

    ConnectX-2 is supported from FreeBSD 9.x onwards.
     
    #26
  7. unwind-protect

    Joined:
    Mar 7, 2016
    Messages:
    118
    Likes Received:
    9
    The way I detangled this so far is that the mlx4 driver needs some functions out of the Linux compatibility layer. The 64 bit Linux compat support is not ready and doesn't compile in its entirety. You can compile the 32 bit Linux module into a 64 bit FreeBSD kernel but that doesn't make those functions available to 64 bit code. I assume that this works on a 32 bit FreeBSD as it is now. This is all on -current.

    I haven't had time to see specifically what is going on and isolate the parts of the Linux module that we need and compile only that into a separate module.

    To clarify this is for Mellanox 10 Gb Ethernet cards, not Infiniband.
     
    #27
  8. JustinClift

    JustinClift New Member

    Joined:
    Oct 5, 2014
    Messages:
    24
    Likes Received:
    11
    @unwind-protect That's a big confusing. Mellanox 10GbE cards work with FreeNAS 9.10-STABLE already. It's native Infiniband mode that isn't working yet.
     
    #28
  9. unwind-protect

    Joined:
    Mar 7, 2016
    Messages:
    118
    Likes Received:
    9
    My current fight might be with a temporary -current breakage on amd64. It is also possible that FreeNAS already did split out the required functions. Do you have a specific model and driver/module that is working for you?
     
    #29
  10. JustinClift

    JustinClift New Member

    Joined:
    Oct 5, 2014
    Messages:
    24
    Likes Received:
    11
    @unwind-protect Mellanox 10/40/56GbE support was added to the 9.10 build on 22nd May 2016:


    That was only in the FreeNAS 9.10 Nightlies for a while, to make sure it didn't introduce breakage.

    It was then added to the FreeNAS 9.10 STABLE releases, starting with the 201606270534 build (eg 27th June 2016), though I didn't test that one.

    Models wise, in theory all of the ConnectX, ConnectX-2, ConnectX-3 (including Pro), and ConnectX-4 cards should work.

    Personally, I've only tested with ConnectX series ones (MHGH28-XTC), but other people have definitely tested with ConnectX-3 Pro's, and someone here shows usage with a ConnectX-2. Another report of ConnectX-2 working here (HW: HP 592520-B21 / MHQH29B-XTR).

    Is it possible you're working with an older version of the FreeNAS 9.10 ISO's? Maybe the release download or something?

    If not, then yeah, it sounds like a bug or something wrong with the hardware. :(
     
    #30
    Last edited: Aug 5, 2016
  11. unwind-protect

    Joined:
    Mar 7, 2016
    Messages:
    118
    Likes Received:
    9
    No, I think my problem is that compilation of the mlx4 driver looks broken in FreeBSD-current. As I said it looks like it requires functions that are (now?) in the Linux64 module, however you currently cannot compile the Linux64 module. Compiling the Linux32 module doesn't give the desired functions to the mlx4 driver.
     
    #31
  12. JustinClift

    JustinClift New Member

    Joined:
    Oct 5, 2014
    Messages:
    24
    Likes Received:
    11
    Ahhh, ok. Not sure why you're trying to do that, but ok. :)

    Any interest in raising that on the freebsd-infiniband mailing list?


    The Mellanox driver developers are subscribed there, and are generally pretty proactive about reported problems. :)
     
    #32
  13. esses

    esses New Member

    Joined:
    Mar 12, 2018
    Messages:
    8
    Likes Received:
    2
    Apologies for the thread necro;

    Did anyone ever find a solution to this?

    I just reinstalled Freenas 11-U2 running on an older ConnectX2-en, and the v2.1.6 driver spits this out at me;

    I had it working on my old build, however it looks like the fresh install has thoroughly broken running the ConnectX2. I'd sooner not spend another few hundred on a ConnectX3 or X4 card.
     
    #33
  14. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,362
    Likes Received:
    294
  15. esses

    esses New Member

    Joined:
    Mar 12, 2018
    Messages:
    8
    Likes Received:
    2
    Looks like they're all sold already. Oh well.
     
    #35
  16. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,362
    Likes Received:
    294
    Ah prices have come down a lot. You alternatively could virtualize and use the cx2 in the hypervisor and pass through a virtual nic?

    Or which version of FN did you have on your old build?
     
    #36
  17. esses

    esses New Member

    Joined:
    Mar 12, 2018
    Messages:
    8
    Likes Received:
    2
    Original was 9.10, but upgraded subsequently to 11.1 U2.

    I installed 11.1 U2 fresh for the new build because I wanted to decommission an old HP Proliant I had previously used as an esxi host, and virtualize the FreeNAS instance under that.

    It all went quite well, the HBA is passed through and FreeNAS is humming along quite happily right now as a VM serving all of the other VM's without any of them being any the wiser. Go me! Smaller power bill!

    Now here's what the problem is;
    The new FreeNAS installation won't start the mellanox card at all, I keep getting the IRQ error. I don't know if VMware is interfering even though the MLX card is on passthrough mode, but even copying the kernel modules from the old (IDENTICAL) installation don't seem to help. I haven't had time to pop the old freenas boot media in and try running it bare metal again.

    Right now, I'm seeing if I can get VMware to pass traffic from the 10gbe card to the freenas VM but I'm finding the way ESXi does networking to be very confusing.

    From the FreeNAS VM it should be;
    192.168.0.x - 1gbe normal network traffic
    192.168.254.x - 10gbe peer-2-peer right to my workstation.

    On ESXi I had assumed to get the extra subnet working I'd need to add both physical adapters to the vswitch, assign the correct IP addresses on either end and I'd be good to go. Didn't work. Setting up a different port group/switch would be another way to go but I keep running out of time.

    Subsequent iterations I've tried are adding a VMkernel for the 254.x subnet.

    So, it seems because FreeNAS won't play nice with the mellanox card on passthrough I'm left struggling trying to route traffic intelligently (guffaw) through a vswitch infrastructure to emulate the switched network + peer to peer setup I had before.
     
    #37
  18. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,362
    Likes Received:
    294
    well easiest would be two vswitches, one with the 1g interfaces and one with the 10g card. have a client network on each and pass taht with a vmxnet3 nic to FN, shouldnt be too difficult. lots of guides for that here or online
     
    #38
  19. esses

    esses New Member

    Joined:
    Mar 12, 2018
    Messages:
    8
    Likes Received:
    2
    Well, that was odd. Nothing worked four times over with the vswitch configuration, and then suddenly started to. Performance is a little lower than hoped for - iperf3 throws out 2.96-4.21gbits/sec to the client, but it's sure better than 1GbE.

    I'll keep my eye out for some X3 cards though...
     
    #39
  20. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,362
    Likes Received:
    294
    There might be some 10G optimizations you can add to FN. I had a few values here and on the FN forum but that was from v9, not sure they are still applicable tbh.
     
    #40
Similar Threads: Mellanox ConnectX-2
Forum Title Date
FreeBSD and FreeNAS Mellanox CX4 ConnectX VPI cards and FreeNAS Jul 23, 2016
FreeBSD and FreeNAS NAS4Free Mellanox ConnectX support missing in new version Feb 22, 2015

Share This Page