TrueNAS Scale + ConnectX-3 - only infiniband driver loading?

[AveryFreeman]

Hey, I have a server at a place I help with IT stuff running TrueNAS Scale 24.10 - it was at 22.04, part of the reason I upgraded it was hoping it would solve this issue

lspci shows the Mellanox 27500 model, and it's under /sys/bus/pci/devices, so it's absolutely there and operational. TrueNAS scale is only loading the infiniband driver for the Mellanox Connectx-3, mlx4_ib, not the ethernet one, mlx4_en.

Loading the mlx4_en driver works with modprobe, but the NICs still aren't detected - not sure if I need to pack drivers into initramfs, no idea how to do this on scale since interacting with it is so decidedly un-linuxish now.

I swapped it out with another one, I've got 4 CX354-QCBT I bought in 2019 and flashed from IB to ethernet. All were conformed working fine.

I figured all the heads on this board love their CX3s, so I thought it'd be the place to ask - any ideas what to do here?

Thanks!

Update: I wasn't sure where else to put this, since Scale is not really like Debian, its syntax and methodology is completely different due to being an appliance, but there's no Scale forum, and this Forum doesn't mention it. My thought was perhaps the moderators never updated this forum to reflect Scale's existence, which is why I chose it instead of a Linux forum, in addition to Ix Systems having a very particular, let's say... Unique ... interface due to its intended use as an appliance. Thank you for your understanding.

 

[i386]

flashed from IB to ethernet.

Just to be sure: you mean you flashed them with the en firmware that's ethernet only?
(I did this with all my cx-3 because I'm not using ib and had problems with ubuntu 22.04 and older)

 

[sko]

There is no "modprobe" in FreeBSD, that is a linux thing. You also don't have to reflash or fiddle around in any other way with the firmware on FreeBSD. Simply load the appropriate driver (mlx4en) via the kld_list in rc.conf (or manually via kldload) and the NIC will work. For IB you'd use the mlx4ib driver accordingly.

 

[i386]

There is no "modprobe" in FreeBSD, that is a linux thing.

Truenas scale is the linux based product, truenas core is the freebsd based product

 

[sko]

Truenas scale is the linux based product, truenas core is the freebsd based product

Given that this is the subforum for the FreeBSD based Free/TrueNAS variants I didn't give the 'scale/core' much attention as it is very often mixed up anyways...

 

[AveryFreeman]

Just to be sure: you mean you flashed them with the en firmware that's ethernet only?
(I did this with all my cx-3 because I'm not using ib and had problems with ubuntu 22.04 and older)

Yeah, both ports are 40Gb ethernet on all of them. Worked fine with everything else I'd used it with, but I think I'd only tried Arch, Fedora and vSphere + vCenter

 

[AveryFreeman]

There is no "modprobe" in FreeBSD, that is a linux thing. You also don't have to reflash or fiddle around in any other way with the firmware on FreeBSD. Simply load the appropriate driver (mlx4en) via the kld_list in rc.conf (or manually via kldload) and the NIC will work. For IB you'd use the mlx4ib driver accordingly.

The modprobe command is present, and the modules it either loads or removes from memory are reflected in lsmod, if that's more clear

Whether or not you're supposed to use it, I have no idea because I am completely new to TrueNAS, I only thought it'd be easier for my friend to support his ISCSI SAN if he needed to, since he's not so tech savvy.

Do you know how to rebuild initramfs the TrueNAS™️ way? Thanks

 

[AveryFreeman]

Truenas scale is the linux based product, truenas core is the freebsd based product

Yeah I know, I don't see a forum for it, sorry. This seemed like the best alternative. Scale is decidedly *not* Debian (that's the whole reason for this thread).

 

[AveryFreeman]

There is no "modprobe" in FreeBSD, that is a linux thing. You also don't have to reflash or fiddle around in any other way with the firmware on FreeBSD. Simply load the appropriate driver (mlx4en) via the kld_list in rc.conf (or manually via kldload) and the NIC will work. For IB you'd use the mlx4ib driver accordingly.

Oh I gotcha sorry, yeah I apologize wasn't a better looking option, didn't mean to confuse you. I'd post in the linux forums, but it doesn't really act like FreeBSD or Linux TBH - I mean, it has modprobe but tf if I can get it working.

 

[sko]

From how butchered the FreeBSD based TrueNAS is underneath I guess it's the same with the linux based variant. I absolutely can't recommend it as it makes standard configurations that are easy 1-line changes in config files tedious and annoying because you have to find the option in the GUI - config files get overwritten or tampered with, so normal system configuration and administration simply isn't possible. If they didn't think of a use case you simply can't do it - or only very cumbersome and definately not in a sane and reasonable way you can figure out a few months later when you need to change it.

 

[AveryFreeman]

I knew I should have just installed debian or freebsd, now I can't use my 40GbE adapters I set up for backups and migrations from the other servers. Of all the machines for the NIC not to be supported well, the storage server's the worst. I'l just run TN on ESXi next time I get back so the hardware will be supported OOB. Shocking, I've never imagined HW being better supported by ESXi. What's even more shocking is nobody has an easy fix for TrueNAS to build some different modules into a new initramfs. WTAF.

 

[sko]

...or just use FreeBSD (or the TrueNAS Core if you must...) where that NIC has been supported for over a decade:

from man(4) mlx4en:

HISTORY
     The mlx4en device driver first appeared in FreeBSD 9.x.

even if you'd need another driver version you wouldn't have to fiddle around with something like a initramfs - just build the driver module for the kernel version and you are done.