FreeNAS 11.3 U1 and Mellanox ConnectX-3 issue

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

chrisk2305

Member
Jan 25, 2017
38
2
8
40
Hi guys,

I would need your help. I have a FreeNAS 11.3 machine with a Mellanox ConnectX-3 40Gbe / IB Single Port installed.

The interface does not show up in the list of network interfaces but the driver seems to be loaded:

Code:
pciconf -lv | grep Mellanox -C 3

   class      = bridge
   subclass   = PCI-PCI
mlx4_core0@pci0:129:0:0:        class=0x028000 card=0x005115b3 chip=0x100315b3 rev=0x00 hdr=0x00
   vendor     = 'Mellanox Technologies'
   device     = 'MT27500 Family [ConnectX-3]'
   class      = network
none6@pci0:132:0:0:     class=0x130000 card=0x148a1022 chip=0x148a1022 rev=0x00 hdr=0x00



Code:
dmesg | grep Mellanox
mlx5en: Mellanox Ethernet driver 3.5.1 (April 2019)
mlx4_core: Mellanox ConnectX core driver v3.5.1 (April 2019)


This card was set to ETH mode in Windows previously. Maybe I need to set the card to ETH mode in FreeNAS again? I read something about a system tunable to accomplish that (sys.device.mlx4_core0.mlx4_port0 = eth) but I am not sure how to do that and find the right "device path".

Can somebody please assist?

Thanks in advance!
Chris
 

chrisk2305

Member
Jan 25, 2017
38
2
8
40
Ok, that would explain why the Interface isn't showing up. Thanks for the link but this does seem to apply for linux.

In the FreeNAS Forums somebody pointed out to set the system tuneable to something like this:

Code:
sys.device.mlx4_core0.mlx4_port0 = eth
But I am unsure how find (a) the correct device path and (b) how to set that at all.

Maybe someone has done this before and can assist me?

thanks!
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Connect-x3 cards don't save the port configuration, my guess is the port automatically changed itself to infiniband.
they can absolutely save the port config, it just won't do it if you use driver options/flags. If you use mellanox firmware tools to set the ports (and other options), it's written into the card's EEPROM, and will persist forever even if you take it out and put it into another system.

Because of this, every time I buy one of these cards I boot a debian live cd with MFT installed, and update the card firmware and configure the ports and other things before putting it into use. It completely avoids these quite common threads of people waking up one day and finding out their freenas/esxi/xenserver/etc system updated some driver and they suddenly have an infiniband card. If anyone wants the exact commands, second part of this post: https://forums.servethehome.com/ind...net-dual-port-qsfp-adapter.20525/#post-198015

across all the forums I browse this has become such a common issue that causes so much grief I've been toying with the idea of building a custom live ISO people can just boot once and it does all this automatically, then the card is good forever
 

Tim111

New Member
Jan 9, 2019
11
1
3
I have created a script for freenas that allows you to flash cx2 and cx3 cards configure ports and some other stuff for cx3. Also made one for Windows that works well. Also as per Mellanox dual port cards need to be eth eth or ib eth not eth ib.
 
  • Like
Reactions: nikalai

tjk

Active Member
Mar 3, 2013
481
199
43
across all the forums I browse this has become such a common issue that causes so much grief I've been toying with the idea of building a custom live ISO people can just boot once and it does all this automatically, then the card is good forever
Please do!
 

Tim111

New Member
Jan 9, 2019
11
1
3
Here is the link to my files.
Dropbox - Scripts.rar - Simplify your life
You will need to install mft first. tjk that would be awesome i have firmware for loads of cx2 and cx3 but sadly you must create them. if you create iso with Linux or freebsd I think my script would be a good start. Please let me know if anyone finds bugs. I created this for cross flashing cards so I know windows script will need adjusting for your needs.
 

netdude

New Member
Jan 28, 2021
1
0
1
It turned out to be in my HPE BIOS : an option to manage the Mellanox port settings ... I'm used to setting using the mlxconfig tool that Mellanox provided but it seems the older cards had the settings in the BIOS.