Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

amatveev

New Member
May 4, 2016
4
0
1
41
You need to renew module references
Code:
kldxref /boot/kernel

kldxref(8)
It's done right after I install all the drivers:

install -o root -g wheel -m 555 mlx4ib.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
install -o root -g wheel -m 555 ibcore.ko /boot/kernel
kldxref /boot/kernel

Problem is here:
[root@storage-b] /mnt/tank/temp/install# kldload mlx4ib.ko
kldload: mlx4ib.ko will be loaded from /boot/kernel, not the current directory
kldload: an error occurred while loading the module. Please check dmesg(8) for more details.

[root@storage-b] /mnt/tank/temp/install# dmesg
...
mlx4_core0: <mlx4_core> mem 0xfbb00000-0xfbbfffff,0xfa800000-0xfaffffff irq 16 at device 0.0 on pci3
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (May 4 2016)
link_elf_obj: symbol ipport_firstauto undefined
linker_load_file: Unsupported file type
KLD mlx4ib.ko: depends on ibcore - not available or version mismatch
mlx4_core0: detached
pci3: <serial bus> at device 0.0 (no driver attached)
linker_load_file: Unsupported file type
 

JustinClift

Member
Oct 5, 2014
35
14
8
If it's of any interest, support for Mellanox adapters was added into the main development code for FreeNAS 9.10 (and upcoming 10) a few days ago. It's available in the official 9.10 nightlies:

Index of /9.10/MASTER/

Grab anything from about 201605221730 onwards (or just grab the latest), and you should be good.

For native IB mode support, that will take further effort. Some work is happening unofficially here, but it's not as far along and I don't have tonnes of time spare for it at the moment:

GitHub - justinclift/freenas-infiniband: FreeNAS 9.10/10 with drivers for Mellanox Ethernet & Infiniband adapters

Feedback on the ISO available there for download is welcome though. :)
 
Last edited:
  • Like
Reactions: Patrick and Aestr

inbusiness

New Member
Jul 31, 2016
5
3
3
48
It's done right after I install all the drivers:

install -o root -g wheel -m 555 mlx4ib.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
install -o root -g wheel -m 555 ibcore.ko /boot/kernel
kldxref /boot/kernel

Problem is here:
[root@storage-b] /mnt/tank/temp/install# kldload mlx4ib.ko
kldload: mlx4ib.ko will be loaded from /boot/kernel, not the current directory
kldload: an error occurred while loading the module. Please check dmesg(8) for more details.

[root@storage-b] /mnt/tank/temp/install# dmesg
...
mlx4_core0: <mlx4_core> mem 0xfbb00000-0xfbbfffff,0xfa800000-0xfaffffff irq 16 at device 0.0 on pci3
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (May 4 2016)
link_elf_obj: symbol ipport_firstauto undefined
linker_load_file: Unsupported file type
KLD mlx4ib.ko: depends on ibcore - not available or version mismatch
mlx4_core0: detached
pci3: <serial bus> at device 0.0 (no driver attached)
linker_load_file: Unsupported file type
Mellanox FreeBSD drver 2.1.5 is a last support for ConnectX-2 VPI HCA.
You must use a driver 2.1.5, not 2.1.6 that support ConnectX-3 or above only.
 

JustinClift

Member
Oct 5, 2014
35
14
8
Mellanox FreeBSD drver 2.1.5 is a last support for ConnectX-2 VPI HCA.
You must use a driver 2.1.5, not 2.1.6 that support ConnectX-3 or above only.
That might be true for the driver manually downloaded from the Mellanox website, but it's not the case for the Mellanox driver in mainline FreeBSD. That supports the adapters listed here:

ConnectX-2 is supported from FreeBSD 9.x onwards.
 

unwind-protect

Active Member
Mar 7, 2016
415
156
43
Boston
The way I detangled this so far is that the mlx4 driver needs some functions out of the Linux compatibility layer. The 64 bit Linux compat support is not ready and doesn't compile in its entirety. You can compile the 32 bit Linux module into a 64 bit FreeBSD kernel but that doesn't make those functions available to 64 bit code. I assume that this works on a 32 bit FreeBSD as it is now. This is all on -current.

I haven't had time to see specifically what is going on and isolate the parts of the Linux module that we need and compile only that into a separate module.

To clarify this is for Mellanox 10 Gb Ethernet cards, not Infiniband.
 

unwind-protect

Active Member
Mar 7, 2016
415
156
43
Boston
@unwind-protect That's a big confusing. Mellanox 10GbE cards work with FreeNAS 9.10-STABLE already. It's native Infiniband mode that isn't working yet.
My current fight might be with a temporary -current breakage on amd64. It is also possible that FreeNAS already did split out the required functions. Do you have a specific model and driver/module that is working for you?
 

JustinClift

Member
Oct 5, 2014
35
14
8
@unwind-protect Mellanox 10/40/56GbE support was added to the 9.10 build on 22nd May 2016:


That was only in the FreeNAS 9.10 Nightlies for a while, to make sure it didn't introduce breakage.

It was then added to the FreeNAS 9.10 STABLE releases, starting with the 201606270534 build (eg 27th June 2016), though I didn't test that one.

Models wise, in theory all of the ConnectX, ConnectX-2, ConnectX-3 (including Pro), and ConnectX-4 cards should work.

Personally, I've only tested with ConnectX series ones (MHGH28-XTC), but other people have definitely tested with ConnectX-3 Pro's, and someone here shows usage with a ConnectX-2. Another report of ConnectX-2 working here (HW: HP 592520-B21 / MHQH29B-XTR).

Is it possible you're working with an older version of the FreeNAS 9.10 ISO's? Maybe the release download or something?

If not, then yeah, it sounds like a bug or something wrong with the hardware. :(
 
Last edited:

unwind-protect

Active Member
Mar 7, 2016
415
156
43
Boston
No, I think my problem is that compilation of the mlx4 driver looks broken in FreeBSD-current. As I said it looks like it requires functions that are (now?) in the Linux64 module, however you currently cannot compile the Linux64 module. Compiling the Linux32 module doesn't give the desired functions to the mlx4 driver.
 

JustinClift

Member
Oct 5, 2014
35
14
8
Ahhh, ok. Not sure why you're trying to do that, but ok. :)

Any interest in raising that on the freebsd-infiniband mailing list?


The Mellanox driver developers are subscribed there, and are generally pretty proactive about reported problems. :)
 

esses

New Member
Mar 12, 2018
16
4
3
48
Apologies for the thread necro;

Did anyone ever find a solution to this?

I just reinstalled Freenas 11-U2 running on an older ConnectX2-en, and the v2.1.6 driver spits this out at me;

mlx4_core0: <mlx4_core> mem 0xfd000000-0xfd0fffff,0xe4000000-0xe5ffffff irq 19 at device 0.0 on pci6
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Feb 20 2018)

mlx4_core0: No IRQs left, device can't be started.
device_attach: mlx4_core0 attach returned 28
I had it working on my old build, however it looks like the fresh install has thoroughly broken running the ConnectX2. I'd sooner not spend another few hundred on a ConnectX3 or X4 card.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
Ah prices have come down a lot. You alternatively could virtualize and use the cx2 in the hypervisor and pass through a virtual nic?

Or which version of FN did you have on your old build?
 

esses

New Member
Mar 12, 2018
16
4
3
48
Original was 9.10, but upgraded subsequently to 11.1 U2.

I installed 11.1 U2 fresh for the new build because I wanted to decommission an old HP Proliant I had previously used as an esxi host, and virtualize the FreeNAS instance under that.

It all went quite well, the HBA is passed through and FreeNAS is humming along quite happily right now as a VM serving all of the other VM's without any of them being any the wiser. Go me! Smaller power bill!

Now here's what the problem is;
The new FreeNAS installation won't start the mellanox card at all, I keep getting the IRQ error. I don't know if VMware is interfering even though the MLX card is on passthrough mode, but even copying the kernel modules from the old (IDENTICAL) installation don't seem to help. I haven't had time to pop the old freenas boot media in and try running it bare metal again.

Right now, I'm seeing if I can get VMware to pass traffic from the 10gbe card to the freenas VM but I'm finding the way ESXi does networking to be very confusing.

From the FreeNAS VM it should be;
192.168.0.x - 1gbe normal network traffic
192.168.254.x - 10gbe peer-2-peer right to my workstation.

On ESXi I had assumed to get the extra subnet working I'd need to add both physical adapters to the vswitch, assign the correct IP addresses on either end and I'd be good to go. Didn't work. Setting up a different port group/switch would be another way to go but I keep running out of time.

Subsequent iterations I've tried are adding a VMkernel for the 254.x subnet.

So, it seems because FreeNAS won't play nice with the mellanox card on passthrough I'm left struggling trying to route traffic intelligently (guffaw) through a vswitch infrastructure to emulate the switched network + peer to peer setup I had before.
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
well easiest would be two vswitches, one with the 1g interfaces and one with the 10g card. have a client network on each and pass taht with a vmxnet3 nic to FN, shouldnt be too difficult. lots of guides for that here or online
 

esses

New Member
Mar 12, 2018
16
4
3
48
Well, that was odd. Nothing worked four times over with the vswitch configuration, and then suddenly started to. Performance is a little lower than hoped for - iperf3 throws out 2.96-4.21gbits/sec to the client, but it's sure better than 1GbE.

I'll keep my eye out for some X3 cards though...
 

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
There might be some 10G optimizations you can add to FN. I had a few values here and on the FN forum but that was from v9, not sure they are still applicable tbh.