Are you doing IPoIB or NFS with FreeBSD 10.1? ZFS, etc?No. The Mellanox drivers work fine in FreeBSD 10.1. Since FreeNAS is based on 9.x, the cards won't work. I am currently running my storage server with 10.1 and a ConnectX-2 card. I did have an odd iscsi hiccup once, but I think that was a hardware issue with one of the ports on the card and not a FreeBSD or driver issue. After running 2012 for a bit, FreeBSD 10.1 seems like a great storage server OS.
I'm in the same boat as you. If you look closely at the release notes for RoCE support you'll see it says ConnectX-3 only.My 2 cards are ConnectX-2 EN, so no Infiniband. I'm using iSCSI right now. I don't believe SRP is supported, but RoCE may be coming if it isn't already supported:
SRP + Infiniband + FreeBSD 10
I am a bit frustrated with Mellanox's driver support so far, however. According to the spec sheet, my cards support RoCE. However, I was never able to get it working in 2012, and I don't believe anyone else here was either. The current plan is to upgrade to ConnectX-3 cards once they come down in price a little more.
Support for these cards has been in FreeBSD for quite some time, and in 9.X for sure because I was using mine back then. You might have to compile a custom kernel for FreeNAS though.Looking to see if anyone has tried these yet. I want to use a ConnectX-2 card in an upcoming FreeNAS box if this works.
For current versions of pfSense and FreeNAS they don't work out the box.Support for these cards has been in FreeBSD for quite some time, and in 9.X for sure because I was using mine back then. You might have to compile a custom kernel for FreeNAS though.
Which is why I said you may have to compile a custom kernel. And by custom I mean add two lines to the kernel conf. Is that activity not allowed in FreeNAS?For currently versions of pfSense and FreeNAS they don't work out the box.
That is a cumbersome solution upgrading. I personally prefer native compatibility.Which is why I said you may have to compile a custom kernel. And by custom I mean add two lines to the kernel conf. Is that activity not allowed in FreeNAS?
Bum deal.That is a cumbersome solution upgrading. I personally prefer native compatibility.
What 2 lines?Which is why I said you may have to compile a custom kernel. And by custom I mean add two lines to the kernel conf. Is that activity not allowed in FreeNAS?
Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).You don't need to compile a custom kernel, you need to compile the driver though. In FreeBSD it is located at
/usr/src/sys/modules/mlxen
then you write:
make && make install && kldload mlxen
And to load the driver for each reboot you add the following to /boot/loader.conf
mlxen_load="YES"
Can confirm this works with a connectX-3 40gbe HP LOM.Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).
I created a new (!) jail in FreeNAS 9.10, used only make in the mlxen directory you pointed out and copied the created mlxen.ko out of the jail to one of my FreeNAS datasets.
I also needed to compile the mlx4 driver to get the mlx4.ko (mlxen.ko would not start without) under the folder:
/usr/src/sys/modules/mlx4
With the mlxen.ko and mlx4.ko in one of my datasets under FreeNAS, i then opened a FreeNAS shell and installed them with these commands:
install -o root -g wheel -m 555 mlxen.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
kldxref /boot/kernel
Afterwards, i could load the mlxen.ko sucessfully with
kldload mlxen.ko
Because my connectx-2 card is not shipped to me yet, i can not verify if this really works.
But thats the most promising solution to the mellanox connectx-2 & freenas problem i have found so far...
I've tried to do the same operation for ConnectX-2 VDI in IB mode without success.Thanks! That actually worked to load the driver in the new FreeNAS 9.10 (based on FreeBSD 10.3).
I created a new (!) jail in FreeNAS 9.10, used only make in the mlxen directory you pointed out and copied the created mlxen.ko out of the jail to one of my FreeNAS datasets.
I also needed to compile the mlx4 driver to get the mlx4.ko (mlxen.ko would not start without) under the folder:
/usr/src/sys/modules/mlx4
With the mlxen.ko and mlx4.ko in one of my datasets under FreeNAS, i then opened a FreeNAS shell and installed them with these commands:
install -o root -g wheel -m 555 mlxen.ko /boot/kernel
install -o root -g wheel -m 555 mlx4.ko /boot/kernel
kldxref /boot/kernel
Afterwards, i could load the mlxen.ko sucessfully with
kldload mlxen.ko
Because my connectx-2 card is not shipped to me yet, i can not verify if this really works.
But thats the most promising solution to the mellanox connectx-2 & freenas problem i have found so far...
It's done with: install -o root -g wheel -m 555 ibcore.ko /boot/kernelYou need to install the ibcore kernel module.
/usr/src/sys/modules/ibcore/