Do Mellanox ConnectX-2 EN 10GbE Cards work in FreeNAS?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Jeggs101

Well-Known Member
Dec 29, 2010
1,529
241
63
Looking to see if anyone has tried these yet. I want to use a ConnectX-2 card in an upcoming FreeNAS box if this works.
 

tjk

Active Member
Mar 3, 2013
481
199
43
I tried to get them working in FreeNAS 9.3, and FreeNAS does not see the card(s). Not sure if there are "hacks" to make em work, I tried the latest 9.3 vanilla.
 
Mar 7, 2015
56
6
8
No. The Mellanox drivers work fine in FreeBSD 10.1. Since FreeNAS is based on 9.x, the cards won't work. I am currently running my storage server with 10.1 and a ConnectX-2 card. I did have an odd iscsi hiccup once, but I think that was a hardware issue with one of the ports on the card and not a FreeBSD or driver issue. After running 2012 for a bit, FreeBSD 10.1 seems like a great storage server OS.
 

tjk

Active Member
Mar 3, 2013
481
199
43
No. The Mellanox drivers work fine in FreeBSD 10.1. Since FreeNAS is based on 9.x, the cards won't work. I am currently running my storage server with 10.1 and a ConnectX-2 card. I did have an odd iscsi hiccup once, but I think that was a hardware issue with one of the ports on the card and not a FreeBSD or driver issue. After running 2012 for a bit, FreeBSD 10.1 seems like a great storage server OS.
Are you doing IPoIB or NFS with FreeBSD 10.1? ZFS, etc?

Do you know if 10.1 supports IB SRP?
 
Mar 7, 2015
56
6
8
My 2 cards are ConnectX-2 EN, so no Infiniband. I'm using iSCSI right now. I don't believe SRP is supported, but RoCE may be coming if it isn't already supported:

SRP + Infiniband + FreeBSD 10

I am a bit frustrated with Mellanox's driver support so far, however. According to the spec sheet, my cards support RoCE. However, I was never able to get it working in 2012, and I don't believe anyone else here was either. The current plan is to upgrade to ConnectX-3 cards once they come down in price a little more.
 

Baddreams

New Member
Mar 15, 2011
21
7
3
My 2 cards are ConnectX-2 EN, so no Infiniband. I'm using iSCSI right now. I don't believe SRP is supported, but RoCE may be coming if it isn't already supported:

SRP + Infiniband + FreeBSD 10

I am a bit frustrated with Mellanox's driver support so far, however. According to the spec sheet, my cards support RoCE. However, I was never able to get it working in 2012, and I don't believe anyone else here was either. The current plan is to upgrade to ConnectX-3 cards once they come down in price a little more.
I'm in the same boat as you. If you look closely at the release notes for RoCE support you'll see it says ConnectX-3 only.

I'd like to see iSER support in FreeBSD but who knows when that will happen. Someone started it, you can see the configuration options for it in ctl.conf, but there isn't anything in the kernel to make use of it.

I've tried pretty much every combination of hosts to get iSER working including ESXi 5.5, FreeBSD 10.0/1, Linux (Ubuntu 14.04, LIO target), and XenServer 6.1. In the end, the only combination I could get to work was two Linux hosts talking to each other, then iSER worked. And it was glorious. Never seen performance like that.

Couldn't get SMBDirect working between two Windows Server 2012 R2 hosts.

I think these cards are just a bit too old for Mellanox to put much effort into supporting the fancy features.
 
  • Like
Reactions: timothy.lazarus

Baddreams

New Member
Mar 15, 2011
21
7
3
Looking to see if anyone has tried these yet. I want to use a ConnectX-2 card in an upcoming FreeNAS box if this works.
Support for these cards has been in FreeBSD for quite some time, and in 9.X for sure because I was using mine back then. You might have to compile a custom kernel for FreeNAS though.
 

neo

Well-Known Member
Mar 18, 2015
672
363
63
Support for these cards has been in FreeBSD for quite some time, and in 9.X for sure because I was using mine back then. You might have to compile a custom kernel for FreeNAS though.
For current versions of pfSense and FreeNAS they don't work out the box.
 
Last edited:

Baddreams

New Member
Mar 15, 2011
21
7
3
For currently versions of pfSense and FreeNAS they don't work out the box.
Which is why I said you may have to compile a custom kernel. And by custom I mean add two lines to the kernel conf. Is that activity not allowed in FreeNAS?
 

neo

Well-Known Member
Mar 18, 2015
672
363
63
Which is why I said you may have to compile a custom kernel. And by custom I mean add two lines to the kernel conf. Is that activity not allowed in FreeNAS?
That is a cumbersome solution upgrading. I personally prefer native compatibility.
 

BackupProphet

Well-Known Member
Jul 2, 2014
1,083
640
113
Stavanger, Norway
olavgg.com
You don't need to compile a custom kernel, you need to compile the driver though. In FreeBSD it is located at
/usr/src/sys/modules/mlxen

then you write:
make && make install && kldload mlxen

And to load the driver for each reboot you add the following to /boot/loader.conf
mlxen_load="YES"
 
Last edited:
  • Like
Reactions: Lighti

Lighti

New Member
Apr 2, 2016
1
3
3
40
You don't need to compile a custom kernel, you need to compile the driver though. In FreeBSD it is located at
/usr/src/sys/modules/mlxen

then you write:
make && make install && kldload mlxen

And to load the driver for each reboot you add the following to /boot/loader.conf
mlxen_load="YES"
Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).

I created a new (!) jail in FreeNAS 9.10, used only make in the mlxen directory you pointed out and copied the created mlxen.ko out of the jail to one of my FreeNAS datasets.

I also needed to compile the mlx4 driver to get the mlx4.ko (mlxen.ko would not start without) under the folder:
/usr/src/sys/modules/mlx4

With the mlxen.ko and mlx4.ko in one of my datasets under FreeNAS, i then opened a FreeNAS shell and installed them with these commands:

install -o root -g wheel -m 555 mlxen.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
kldxref /boot/kernel

Afterwards, i could load the mlxen.ko sucessfully with

kldload mlxen.ko

Because my connectx-2 card is not shipped to me yet, i can not verify if this really works.
But thats the most promising solution to the mellanox connectx-2 & freenas problem i have found so far...
 
May 8, 2015
142
24
18
33
Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).

I created a new (!) jail in FreeNAS 9.10, used only make in the mlxen directory you pointed out and copied the created mlxen.ko out of the jail to one of my FreeNAS datasets.

I also needed to compile the mlx4 driver to get the mlx4.ko (mlxen.ko would not start without) under the folder:
/usr/src/sys/modules/mlx4

With the mlxen.ko and mlx4.ko in one of my datasets under FreeNAS, i then opened a FreeNAS shell and installed them with these commands:

install -o root -g wheel -m 555 mlxen.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
kldxref /boot/kernel

Afterwards, i could load the mlxen.ko sucessfully with

kldload mlxen.ko

Because my connectx-2 card is not shipped to me yet, i can not verify if this really works.
But thats the most promising solution to the mellanox connectx-2 & freenas problem i have found so far...
Can confirm this works with a connectX-3 40gbe HP LOM.
As an aside. You don't need to copy anything over. You can navigate into the jails directory where the the .ko files are located and just run the commands above straight from there. Also add the mlxen.ko to the loader.conf so its persistent across reboots. Thanks for your post though. Made it very easy to get this card working.
 

amatveev

New Member
May 4, 2016
4
0
1
41
Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).

I created a new (!) jail in FreeNAS 9.10, used only make in the mlxen directory you pointed out and copied the created mlxen.ko out of the jail to one of my FreeNAS datasets.

I also needed to compile the mlx4 driver to get the mlx4.ko (mlxen.ko would not start without) under the folder:
/usr/src/sys/modules/mlx4

With the mlxen.ko and mlx4.ko in one of my datasets under FreeNAS, i then opened a FreeNAS shell and installed them with these commands:

install -o root -g wheel -m 555 mlxen.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
kldxref /boot/kernel

Afterwards, i could load the mlxen.ko sucessfully with

kldload mlxen.ko

Because my connectx-2 card is not shipped to me yet, i can not verify if this really works.
But thats the most promising solution to the mellanox connectx-2 & freenas problem i have found so far...
I've tried to do the same operation for ConnectX-2 VDI in IB mode without success.
I've compiled mlx4.ko, ml4ib.ko and ibcore.ko, installed them with:
install -o root -g wheel -m 555 mlx4ib.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
install -o root -g wheel -m 555 ibcore.ko /boot/kernel
kldxref /boot/kernel

But when I try to: kldload mlx4ib.ko
I get:
mlx4_core0: <mlx4_core> mem 0xfbb00000-0xfbbfffff,0xfa800000-0xfaffffff irq 16 at device 0.0 on pci3
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (May 4 2016)

link_elf_obj: symbol ipport_firstauto undefined
linker_load_file: Unsupported file type
KLD mlx4ib.ko: depends on ibcore - not available or version mismatch
mlx4_core0: detached
pci3: <serial bus> at device 0.0 (no driver attached)
linker_load_file: Unsupported file type

Any ideas how this could be fixed?
 

amatveev

New Member
May 4, 2016
4
0
1
41
You need to install the ibcore kernel module.

/usr/src/sys/modules/ibcore/
It's done with: install -o root -g wheel -m 555 ibcore.ko /boot/kernel
(see my command log)

But when I try to load it with: kldload ibcore.ko

link_elf_obj: symbol ipport_firstauto undefined
linker_load_file: Unsupported file type