Integrated NIC Preventing Boot of a X9SCM-F - NVM Currupt

[Luzer]

I'm lucky enough to get a x9scm-f from a friend that said it didn't work. The when I first booted it I got the error below and after some troubleshooting I changed the jumper to disable the NIC and it booted. I don't see any other issues as I have TruNAS running for some testing.

I was thinking of booing into Windows and force a flash, but the motherboard wont POST past this until I disable the onload NIC which makes it unavailable in the OS.

Error:


Is there a way to force a reflash the built in NIC? I emailed Supermicro Tech Support and hopefully have a way without an RMA.

 

[Luzer]

So somehow I was able to get it to boot with the NIC enbaled but TrueNas doesn't see it as an interface to assign to. The activity light is on and LSPCI sees it but nothing under ifconfig.

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:06.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) (rev 05)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C204 Chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
01:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
02:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
04:00.0 Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 02)
05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

dmesg:
nas% dmesg | grep net
pci1: <network, ethernet> at device 0.0 (no driver attached)
igb0: Ethernet address: 9
igb1: Ethernet address: 9
em0: Ethernet address: 0
mlxen0: Ethernet address: f
mlxen1: Ethernet address: f
debugnet_any_ifnet_update: Bad dn_init result from igb0 (ifp 0xfffff80002106800), ignoring.
debugnet_any_ifnet_update: Bad dn_init result from igb1 (ifp 0xfffff80001e09800), ignoring.
debugnet_any_ifnet_update: Bad dn_init result from em0 (ifp 0xfffff80001e0b000), ignoring.

Moved the pin up:

 

[rootwyrm]

Bad dn_init result means you're in a real bad place. Everyone with the real knowledge on that has long ago abandoned FreeBSD for various reasons. And there's all of one person who actually knows what's going on here. Hi. (Yes, I'm that RootWyrm. VMware folks know who. And why that translates to 'the guy who knows damn near as much about the X9SCL/X9SCM as the Supermicro engineers.')

You can attempt the DOS-based AMI flash tool (use 2.39+) but it will probably fail. You need to re-enable the network interface jumper first. Do not do it through the IPMI, do it from an actual cabled connection. But it's probably going to fail. em0's the IPMI shared interface (yes, it's shared, long story there) and it's got a corrupt IME or IPMI. Reflash BIOS, then IPMI - not IPMI and BIOS. Flashing that order will cause it to truly break and then you'll have to go through IPMI recovery, which is very do not want.

If/when the DOS flash fails to improve things, you'll need to do the IME reset two step. PAY VERY CLOSE ATTENTION, THIS IS AT YOUR OWN RISK, MAY CAUSE BLINDNESS IN MICE, AND I AM 150% NOT RESPONSIBLE IF THINGS GO WRONG.

1. Disconnect ALL power, INCLUDING the coin cell. It MUST come out.
2. Set all LAN and BME to ENABLED (1-2)
3. Locate JPME1 and JPME2 (between chipset and BIOS) and set to CLOSED (recovery)
4. Locate J31, immediately above the BIOS. This is the SPI programming interface for the base BIOS
5. Locate JWD (right of the DIMMs) and REMOVE the jumper (disable watchdog)
6. Confirm JI2C1 and JI2C2 are DISABLED (jumper removed)
7. Remove ALL attached devices; cards, drives, TPM if equipped, etc.
8. Locate J29 (SPI Programming Enable) and confirm closed; the header is sometimes absent and a permanent bridge installed

Attempt:

1. Short JBT1 (CMOS clear) for 10+ seconds - yes, even with the battery pulled, these boards have extremely long holdup
2. You need a 4GB OR SMALLER FAT32 USB stick. Anything over 4GB will fail. Every time. Find a 512MB if you can.
3. Download the latest X9SCM-F BIOS from Supermicro
4. Copy that file to "SUPER.ROM" (all caps does matter here!!) on the USB drive
5. Attach a PS/2 keyboard before attaching any power, MUST be PS/2.
6. Now attach power
7. Do NOT turn on the system - wait for the BMC LED to indicate READY.
8. Insert the USB drive into the BOTTOM REAR USB port; it MUST be the bottom rear USB port
9. Power on the system; as SOON as you apply power, press and hold CTRL + HOME to force the system to attempt BIOS recovery

Now, as I said, this is fairly unlikely to work by itself. The next step is performing a forced reset of the IPMI using ipmicfg. (No link, but as a Supermicro customer, you are licensed and can download it directly.) Create or grab a bootable DOS USB, put ipmicfg on it, boot from it, and run "ipmicfg -fde". You may be tempted to use Linux - don't. This will not work. It will say it did. It does not.
WAIT. FIVE. MINUTES. Even if the IPMI LED indicates ready, WAIT FIVE MINUTES.

Now, set JPME1 and JPME2 back to open (normal) and try again. It might still be broken. But it might fix it! And then no mice will go blind. (This is the 'low risk' procedure.)
If you get 5 beeps on the Ctrl+Home portion, this means recovery failed. Do not ask how often this happens on boards that only need a basic recovery process. Try it two or three times, with CMOS clears in between. Changing JPUSB1 to disable or closing JL1 can sometimes help here as well. If it just won't recover, keep the jumpers as described above (especially JPME1 and JPME2,) and try flashing with the DOS utility. The SUPER.ROM method is usually better for IME recovery because it occurs before OROM+PXE init and DOS is post-OROM.

EVERYTHING PAST THIS POINT IS SUPER DANGEROUS AND DO NOT EVEN THINK OF PERFORMING IT UNLESS YOU ARE COMFORTABLE WITH EVERY ITEM IN THE ACRONYM SOUP I AM ABOUT TO DROP ON YOU, BECAUSE YOU WILL PROBABLY BRICK A RECOVERABLE BOARD.
And yes, this is why I am going to be very deliberately obtuse here, to prevent people from destroying their hardware. I'm serious. If you do not already know how to do everything I am about to outline and can fill in the deliberately missing steps unassisted, find a shop comfortable with PCB rework and SPI programming to do it for you. Or buy a newer board. Please. The X9SCM's weren't great when they were new, but they fit a particular niche superbly.

1. (IF NON-SOCKETED) remove J29
2. (IF NON-SOCKETED) Connect 3V SPI to J31 (NOTE: do not use cheap FTDI, do not exceed 3.3V!)
3. Definitively locate BIOS on CSEL (revision dependent!)
4. Definitively locate IPMI (U63) on CSEL
5. (IF IME FAILURE) extract IME from BIOS (OK to use tool to extract from full dump)
6. (IF IME FAILURE) note MAC/PCI association (WARNING: LABELS OFTEN BACKWARDS!)
7. WARNING: DO NOT USE CPOL+CPHA FOR RECOVERY, USE CPHA!
8. WARNING: DO NOT PROGRAM U63 WITH J31!
   NOTE: IF J31 SOCKETED, REMOVAL RECOMMENDED
9. (LICENSE REQUIRED) correct base image IME version if applicable CAUTION L+-F HAS IN REVERSE!!
10. (LICENSE REQUIRED) reprogram IME, SMBIOS, DMI, VPD with correct values
11. (LICENSE REQUIRED) If NMI assert fails in testing, swap TL (optional)
12. Factory reset IPMI and re-flash from real mode

 

[Glock24]

Bad dn_init result means you're in a real bad place. Everyone with the real knowledge on that has long ago abandoned FreeBSD for various reasons. And there's all of one person who actually knows what's going on here. Hi. (Yes, I'm that RootWyrm. VMware folks know who. And why that translates to 'the guy who knows damn near as much about the X9SCL/X9SCM as the Supermicro engineers.')

You can attempt the DOS-based AMI flash tool (use 2.39+) but it will probably fail. You need to re-enable the network interface jumper first. Do not do it through the IPMI, do it from an actual cabled connection. But it's probably going to fail. em0's the IPMI shared interface (yes, it's shared, long story there) and it's got a corrupt IME or IPMI. Reflash BIOS, then IPMI - not IPMI and BIOS. Flashing that order will cause it to truly break and then you'll have to go through IPMI recovery, which is very do not want.

If/when the DOS flash fails to improve things, you'll need to do the IME reset two step. PAY VERY CLOSE ATTENTION, THIS IS AT YOUR OWN RISK, MAY CAUSE BLINDNESS IN MICE, AND I AM 150% NOT RESPONSIBLE IF THINGS GO WRONG.

1. Disconnect ALL power, INCLUDING the coin cell. It MUST come out.
2. Set all LAN and BME to ENABLED (1-2)
3. Locate JPME1 and JPME2 (between chipset and BIOS) and set to CLOSED (recovery)
4. Locate J31, immediately above the BIOS. This is the SPI programming interface for the base BIOS
5. Locate JWD (right of the DIMMs) and REMOVE the jumper (disable watchdog)
6. Confirm JI2C1 and JI2C2 are DISABLED (jumper removed)
7. Remove ALL attached devices; cards, drives, TPM if equipped, etc.
8. Locate J29 (SPI Programming Enable) and confirm closed; the header is sometimes absent and a permanent bridge installed

Attempt:

1. Short JBT1 (CMOS clear) for 10+ seconds - yes, even with the battery pulled, these boards have extremely long holdup
2. You need a 4GB OR SMALLER FAT32 USB stick. Anything over 4GB will fail. Every time. Find a 512MB if you can.
3. Download the latest X9SCM-F BIOS from Supermicro
4. Copy that file to "SUPER.ROM" (all caps does matter here!!) on the USB drive
5. Attach a PS/2 keyboard before attaching any power, MUST be PS/2.
6. Now attach power
7. Do NOT turn on the system - wait for the BMC LED to indicate READY.
8. Insert the USB drive into the BOTTOM REAR USB port; it MUST be the bottom rear USB port
9. Power on the system; as SOON as you apply power, press and hold CTRL + HOME to force the system to attempt BIOS recovery

Now, as I said, this is fairly unlikely to work by itself. The next step is performing a forced reset of the IPMI using ipmicfg. (No link, but as a Supermicro customer, you are licensed and can download it directly.) Create or grab a bootable DOS USB, put ipmicfg on it, boot from it, and run "ipmicfg -fde". You may be tempted to use Linux - don't. This will not work. It will say it did. It does not.
WAIT. FIVE. MINUTES. Even if the IPMI LED indicates ready, WAIT FIVE MINUTES.

Now, set JPME1 and JPME2 back to open (normal) and try again. It might still be broken. But it might fix it! And then no mice will go blind. (This is the 'low risk' procedure.)
If you get 5 beeps on the Ctrl+Home portion, this means recovery failed. Do not ask how often this happens on boards that only need a basic recovery process. Try it two or three times, with CMOS clears in between. Changing JPUSB1 to disable or closing JL1 can sometimes help here as well. If it just won't recover, keep the jumpers as described above (especially JPME1 and JPME2,) and try flashing with the DOS utility. The SUPER.ROM method is usually better for IME recovery because it occurs before OROM+PXE init and DOS is post-OROM.

EVERYTHING PAST THIS POINT IS SUPER DANGEROUS AND DO NOT EVEN THINK OF PERFORMING IT UNLESS YOU ARE COMFORTABLE WITH EVERY ITEM IN THE ACRONYM SOUP I AM ABOUT TO DROP ON YOU, BECAUSE YOU WILL PROBABLY BRICK A RECOVERABLE BOARD.
And yes, this is why I am going to be very deliberately obtuse here, to prevent people from destroying their hardware. I'm serious. If you do not already know how to do everything I am about to outline and can fill in the deliberately missing steps unassisted, find a shop comfortable with PCB rework and SPI programming to do it for you. Or buy a newer board. Please. The X9SCM's weren't great when they were new, but they fit a particular niche superbly.

1. (IF NON-SOCKETED) remove J29
2. (IF NON-SOCKETED) Connect 3V SPI to J31 (NOTE: do not use cheap FTDI, do not exceed 3.3V!)
3. Definitively locate BIOS on CSEL (revision dependent!)
4. Definitively locate IPMI (U63) on CSEL
5. (IF IME FAILURE) extract IME from BIOS (OK to use tool to extract from full dump)
6. (IF IME FAILURE) note MAC/PCI association (WARNING: LABELS OFTEN BACKWARDS!)
7. WARNING: DO NOT USE CPOL+CPHA FOR RECOVERY, USE CPHA!
8. WARNING: DO NOT PROGRAM U63 WITH J31!
   NOTE: IF J31 SOCKETED, REMOVAL RECOMMENDED
9. (LICENSE REQUIRED) correct base image IME version if applicable CAUTION L+-F HAS IN REVERSE!!
10. (LICENSE REQUIRED) reprogram IME, SMBIOS, DMI, VPD with correct values
11. (LICENSE REQUIRED) If NMI assert fails in testing, swap TL (optional)
12. Factory reset IPMI and re-flash from real mode

Very interesting. I'll try this recovery method for a X9SCi-LN4F that takes a long long time to POST and gives a ME error.

 

[Luzer]

Bad dn_init result means you're in a real bad place. Everyone with the real knowledge on that has long ago abandoned FreeBSD for various reasons. And there's all of one person who actually knows what's going on here. Hi. (Yes, I'm that RootWyrm. VMware folks know who. And why that translates to 'the guy who knows damn near as much about the X9SCL/X9SCM as the Supermicro engineers.')

You can attempt the DOS-based AMI flash tool (use 2.39+) but it will probably fail. You need to re-enable the network interface jumper first. Do not do it through the IPMI, do it from an actual cabled connection. But it's probably going to fail. em0's the IPMI shared interface (yes, it's shared, long story there) and it's got a corrupt IME or IPMI. Reflash BIOS, then IPMI - not IPMI and BIOS. Flashing that order will cause it to truly break and then you'll have to go through IPMI recovery, which is very do not want.

If/when the DOS flash fails to improve things, you'll need to do the IME reset two step. PAY VERY CLOSE ATTENTION, THIS IS AT YOUR OWN RISK, MAY CAUSE BLINDNESS IN MICE, AND I AM 150% NOT RESPONSIBLE IF THINGS GO WRONG.

1. Disconnect ALL power, INCLUDING the coin cell. It MUST come out.
2. Set all LAN and BME to ENABLED (1-2)

.....

Thanks for the reply. I'll be able to test this in a day or two. So the first NIC and the IPMI share ROM/firmware?

 

[Glock24]

I did not fins a <4GB flash drive and I did not find a PS/2 keyboard either.

But I noticed that closing JPME1 makes the board boot normally (no excessive boot times) and I see no ME errors. The ME version in the ME info screen in the BIOS shows 0.

I guess this JPME1 jumper disabled the Intel ME?

 

[Luzer]

I've tried to get the motherboard into BIOS recovery mode and it doesnt want to. I did hear the two beeps once or twice but no 'recovery mode'.

I'm using a 1gb USB drive and have the CMOS battery out. When the power button is pressed I hold down a single or both CTRL and Home and it POSTS, the gets stuck on the NVRAM.

I also have the jumpers set per the directions above.

Any tips?

 

[casperghst42]

I used informaton from these pages to reflash the nics on my X10SDV-6C+-TLN4F when the 10GbE ports suddenly became sick.

Missing NIC port, em0: The EEPROM Checksum Is Not Valid

Intel® Ethernet Connections Boot Utility, Preboot Images, and EFI Drivers

Download Intel Drivers and Software

Download new and previously released drivers including support software, bios, utilities, firmware and patches for Intel products.

downloadcenter.intel.com

I first tried with the latest version, but had to go back a few (3 years) to find one which would reflash.

Something like "bootutil -nic=1 -defcfg" could work, but I had to play around with the parameters to get it to work.

Edit: forgot to mention that this is the official Intel way of upgrading the firmware on their NICs.

 

[rootwyrm]

Thanks for the reply. I'll be able to test this in a day or two. So the first NIC and the IPMI share ROM/firmware?

No, that's not the case. It's confusing if you don't have some REAL deep history on internals on these boards.
The X9SCL/M has a minimum three NICs; an RTL8111, and 2+ Intel. Now here's where it gets very, very deeply stupid. (I am still not apologizing for calling it that, because it is.) The board claims in the documentation that the RTL8111 is 'dedicated' to the IPMI - this is accurate. The host has no visibility.
What it leaves OUT is that the IPMI also has access to the first Intel NIC (by PCI order.) And it has a forced link with that Intel NIC in addition to the RTL8111, in fail-over mode. This is no big deal unless you have a complicated VLAN setup. Guess what I had? Yeah. This resulted in a good ... lot of back and forth with a lot of people until somebody finally figured out that oh, if the IPMI decides the RTL8111 isn't 'up' enough, it fails over to the Intel but doesn't and won't fail back unless you pull power completely. And it is VERY easy to trigger the RTL8111 into failure.

Insert a whole lot of screaming here.

That's also why it's so critical to have actual interfaces directly connected. The IPMI will fault in this process at least once or twice, no matter what you do or how simple you try to make it. If your remote I/O gets disconnected during a flash, that really is all she wrote. SPI recovery is the only way then.

I've tried to get the motherboard into BIOS recovery mode and it doesnt want to. I did hear the two beeps once or twice but no 'recovery mode'.

I'm using a 1gb USB drive and have the CMOS battery out. When the power button is pressed I hold down a single or both CTRL and Home and it POSTS, the gets stuck on the NVRAM.

I also have the jumpers set per the directions above.

Any tips?

So, uh, how long did you give it after the 2 beeps, and did anything show up on the monitor?
2 beeps actually indicates that it's attempting to enter recovery mode. Which is very, very, very slow. Literally 6-8MHz 8088 slow. Also, the X9SCL/M has this delightful thing where it doesn't always turn on the display in recovery mode, particularly if the IPMI's at all unhappy or not fully booted or when it is or because of the phase of the moon.

If you can get it to the two beep phase, give it at LEAST 30 minutes. I'm so not joking. If no change there, put the CMOS battery back in and try again. Some boards just refuse to program without the CMOS battery and I honestly can never remember which. Also try holding Ctrl+Home before pressing power and/or before turning on the PSU. As I said; it's finicky. You really will have to experiment and try it a bunch of times and ways.

It's important to have the IME in Recovery Mode because this is what allows the board to actually write/modify the IME firmware 'from scratch.' If the recovery mode jumper is not set, certain regions of the NVRAM are basically write protected. Which if it's a corrupted MAC or the like, would produce the same error or error out on flashing.

 

[Luzer]

So, uh, how long did you give it after the 2 beeps, and did anything show up on the monitor?
2 beeps actually indicates that it's attempting to enter recovery mode. Which is very, very, very slow. Literally 6-8MHz 8088 slow. Also, the X9SCL/M has this delightful thing where it doesn't always turn on the display in recovery mode, particularly if the IPMI's at all unhappy or not fully booted or when it is or because of the phase of the moon.

The two beeps would happen then it would try to boot almost immediately, it would show the Supermicro POST screen.

If you can get it to the two beep phase, give it at LEAST 30 minutes. I'm so not joking. If no change there, put the CMOS battery back in and try again. Some boards just refuse to program without the CMOS battery and I honestly can never remember which. Also try holding Ctrl+Home before pressing power and/or before turning on the PSU. As I said; it's finicky. You really will have to experiment and try it a bunch of times and ways.

Sounds like a plan. Once I have more time to turn off the server I'll give it a try.

I'm debating flashing the BIOS though the SPI and I'm wondering where to start. I think I can get the firmware I need from my friends x9scm and if I break mine a new x9scm isn too much. Do you have any recommendations for the flasher and the connector (no idea what the name are).

Thanks for help.