Supermicro IPMI interface suddenly loses IP address on X9DRD-7LN4F

[BLinux]

One of my Supermicro servers recently lost it's IP address assigned to the IPMI interface. The board is a X9DRD-7LN4F. It originally had a static IP address assigned to the IPMI interface and was working. Suddenly, that IP address went offline.

- I checked the switch port, and it shows there is link.
- I checked the server, and the IPMI port does have link lights.
- Currently, the IPMICFG shows the IP address is 0.0.0.0. I tried to set it manually again, but it keeps returning 0.0.0.0 no matter what i do.
- I reset the with IPMICFG -r, but still same issue.
- I tried a factory configuration reset with clean LAN (IPMICFG -fdl) and it is still having the same issue.

The BMC/IPMI is responsive to IPMICFG from the host itself, but obviously I cannot reach it over the network right now.

Any ideas how to fix this?

 

[RageBone]

i sadly have no clue how to fix that jet, or what that would be.
But i have an idea how to possibly diagnose the problem.

So, we have a few components playing together at this point. The BMC, its Flash on which everything is stored, the dedicated Nic, and the other Platform Nics. The communication channel between BMC and all the Nics is the same i think, so any fault in any component on that, could also cause the symptoms.

First thing to check is if you can get it to work on the other nics, i think.
If it is, dedicated Nic just died, probably.
if it isn't, well. Could still be anything in that list.


Was the board moved or worked on close to the problem ?

 

[BLinux]

do you know the option in IPMICFG to move to one of the other NICs? (I didn't see it, but maybe I missed it)

The dedicated NIC does seem to have link, at least the LED and the switch both seem to indicate so.

No, the system has been running for over a year and I don't really touch it physically so it hasn't been moved. It's really weird, it just suddenly dropped off the network, that's when I noticed the IP is 0.0.0.0.

 

[RageBone]

i sadly don't know the setting for that.

And i guess this is RMA territory because i assume that there is some kind of hardware fault as the root of this.
My Gigabyte MD80 behaves the same, so do a few other of my still dead boards if i remember correctly, but i haven't come around to debug them jet.

if you can get a shell on the BMC, you could look if the Nic is still detected and if the driver still does its thing.
I don't know how to get a shell on the SM ipmi though.
It was once possible and then they patched it away i think.


Since the Nic detects a cable, and lights up, we can assume that the realtek Nic chip isn't fully dead. so it could be something on the way between the rj45 to the chip, some periphery thing around the Nic, the com-channel between BMC and nic, or the BMC, that could cause this.

Oh, and here is a datasheet for the rtl8211 that is on my x10drif.
http://download3.dvd-driver.cz/realtek/datasheets/pdf/rtl8211e(g)-vb(vl)-cg_datasheet_1.6.pdf

On some ASrack boards, there was this issue of killing the flash by writing to many logs to it.
No idea if a corrupted configuration for the bmc could also cause this behavior.
But you should be able to verify that by reading back the flashed config and comparing it.
If it differs, the flash has probably gone bad.

On the ASRack thing, doing weird things like adding a new user and then login in as that one, would temporarily fix the ipmi for a few minutes and restarts.
But the ipmi was still basically working on that one.

 

[EffrafaxOfWug]

I don't know if it's applicable to the X9 series, but I've used the following ipmitool raws to change the IPMI NIC bonding configuration to allow the other onboard NICs to be used for IPMI:

# get current NIC bonding setting
ipmitool raw 0x30 0x70 0x0c 0

# set dedicated
ipmitool raw 0x30 0x70 0x0c 1 00

# set shared
ipmitool raw 0x30 0x70 0x0c 1 01

# set failover
ipmitool raw 0x30 0x70 0x0c 1 02

I've no idea why the NIC would suddely lose its config like that, but my usual first step in debugging weird things like this is replacing the BIOS battery and seeing if things improve. I don't see any reason why a dicky battery would cause IPMI to lose its config, but it is at least something relatively easy to check.

I assume from within the OS you're able to (re-)apply the IPMI NIC settings and that other IPMI config isn't forgotten?

 

[RageBone]

@BLinux just found the setting for dedicated Nics in the Bios of my X10DRi F.
Its in the BMC Network Configuration.

I guess yours does not have that. : (

 

[BLinux]

@BLinux just found the setting for dedicated Nics in the Bios of my X10DRi F.
Its in the BMC Network Configuration.

I guess yours does not have that. : (

yeah, not sure if X9 has the option.. might, but Supermicro tech support is telling me to boot into BIOS, but I can't shutdown the server at the moment so i have to wait for a maintenance window opportunity....