I spent quite a while here over the last 24 hours trying to figure out what was causing my brandy new pfsense build to drop it's WAN connection and go offline, unable to recover. I figured I'd drop a post that may help someone else from chasing their tail.
My Supermicro board has 5x LAN ports on it. 1 dedicated port for the BMC, 4x Intel i354 1gbe ports available to the system. The issue is that Supermicro defaults the IPMI to "Failover" mode. It seems that over some undetermined amount of time, if the BMC doesn't detect a connection on the IPMI ethernet port, it fails over to LAN1 (which is assigned to my pfsense WAN interface, which is the default pfsense config) and boots whatever was connected to it, off. As such, my IPMI was getting a public IP from Verizon on LAN1, siloing pfsense from using it. I never bothered with the IPMI on this since any form of remote admin on this machine at the IPMI level would be pretty worthless, as if pfsense isn't running, I wouldn't be able to access it in the first place. This seems like a pretty big security risk. I would have never imagined that a default setting would risk putting a machine directly to a public facing IP with nothing in between. But here we are.
The fix is easy, login to the IPMI and change the IPMI port from "Failover" to "Dedicated". From there on out, the BMC will never take control of LAN1. There is also a "Shared" mode, which uses an internal switch on LAN1 to split traffic to LAN1 and the IPMI. This would reduce some cabling (if you want to use IPMI), but as I can't see any way for the ONT to know which of the two MAC addresses on the single interface to spit out an IP to, I can't see any real way this would work when configured as a WAN connection port.
I could have reconfigured the WAN port to use LAN4 on the board, but then I would lose the blinky LED on my chassis fro the LAN and WAN connections and we just can't have that.
Hope this helps someone in the future!
My Supermicro board has 5x LAN ports on it. 1 dedicated port for the BMC, 4x Intel i354 1gbe ports available to the system. The issue is that Supermicro defaults the IPMI to "Failover" mode. It seems that over some undetermined amount of time, if the BMC doesn't detect a connection on the IPMI ethernet port, it fails over to LAN1 (which is assigned to my pfsense WAN interface, which is the default pfsense config) and boots whatever was connected to it, off. As such, my IPMI was getting a public IP from Verizon on LAN1, siloing pfsense from using it. I never bothered with the IPMI on this since any form of remote admin on this machine at the IPMI level would be pretty worthless, as if pfsense isn't running, I wouldn't be able to access it in the first place. This seems like a pretty big security risk. I would have never imagined that a default setting would risk putting a machine directly to a public facing IP with nothing in between. But here we are.
The fix is easy, login to the IPMI and change the IPMI port from "Failover" to "Dedicated". From there on out, the BMC will never take control of LAN1. There is also a "Shared" mode, which uses an internal switch on LAN1 to split traffic to LAN1 and the IPMI. This would reduce some cabling (if you want to use IPMI), but as I can't see any way for the ONT to know which of the two MAC addresses on the single interface to spit out an IP to, I can't see any real way this would work when configured as a WAN connection port.
I could have reconfigured the WAN port to use LAN4 on the board, but then I would lose the blinky LED on my chassis fro the LAN and WAN connections and we just can't have that.
Hope this helps someone in the future!