OPNSense on MS-01 -- error: network is down

[coatmaker618]

I'm trying to do a baremetal install of OPNSense on a MINISFORUM MS-01 with a 25 Gbe Mellanox NIC,

When I use the liveusb (running 'root' instead of 'installer') the network interface works fine (can get DHCP lease from current router, ping current router, etc) -- however when I try to actually install from the liveUSB and do EXACTLY the same thing it seemingly installs ok but then fails to get the same network interfaces working telling me "ping: sendto: Network is down" when I try to ping anything on the network.

I'm hoping someone else has gotten OPNSense working on an MS-01!

 

[Scott Laird]

Odds are your NIC is overheating and shutting down. Check your system log? Most Mellanox NICs are only good for 5-10 minutes without forced air cooling, then they'll start logging gripes (at around 105C) and power themselves off (around 115C). The quick fix is to add a USB fan. Just set it on top of the MS-01 blowing down through the existing holes in the case. I have a 2x100G ConnectX-5 in my MS-01 home router, and it's been fine this way for >6 months.

 

[coatmaker618]

UGH! That actually makes a lot of sense, that card sure does get warm!

Any idea where I can see the error/warning logs and/or the temperature? Ideally both of the NIC and the MS-01 itself? But hey, I'll take whatever subset of this I can get.

 

[coatmaker618]

Ok, I THINK I finally got it working consistently. As far as I can tell, there were multiple problems:

1. Overheating(as mentioned above)
2. I'm still learning vPro (spoiled with iDrac). I noticed that vPro had a setting to disable the network port for the OS & isolate it for vPro (like a real IPMI should be), so I obviously checked that because I can spare a port. However OPNSense happily reported that port as a potential interface. I believe it not be a reliable port because it shouldn't even be visibile to the OS (at least, per my reading of the vPro checkbox). Nonetheless, this si a potential issue.
3. Some firewall rule is blocking connection to the webpage port whether I use LAN or OPT1 (I swapped it to 8443). It's obviously local since running `pfctl -d` is sufficient to allow access to the webpage (so glad it's not internet facing yet). Turns out this is because I made a default gateway on the LAN (allows me to use the LAN to get to the internet, eg: updates/package-management).