Natex S5520HC BIOS recovery restores NICs

[techtoys]

Observed symptoms are different that this post so I am starting a new thread.
I purchased 3 systems recently from Natex, 2 x S5520HC and 1 x S2600CP.
Both S5520HC boards have a problem with their on-board NICs.

Short Answer: Perform a BIOS recovery! not just a reflash/upgrade

I suspect the BIOS NVRAM on these systems was modified to remove the NICs from being visible.

Symptoms:

• NIC enable/disable does not appear in BIOS
• NIC only visible in BMC portion of BIOS
• Both NICs light up to display activity
• You can ping the BMC NIC only
• NICs do not appear to ESXi 5.5 or 6.0
• lspci will not find the NIC in ESXi or Ubuntu 14.04

There were some unusual issues on this board like blue screen during Windows Server install. These issues were documented on the Intel site and supposedly fixed in the newest BIOS. The first thing I did was to upgrade to the newest firmware package and restore defaults on the BIOS. This did not fix anything.

You do not actually need to remove the system from the rack and open it up to perform the BIOS recovery. The jumper method is only needed when you can't boot to the EFI. In this case, you should be able to boot into the EFI.

1. Download BIOS recovery package 61
2. Place contents of package in the root of a USB drive
3. Boot to EFI and BIOS should be restored to factory fresh.
4. Cycle AC power. I removed the power cord
5. Remember to let the BIOS finish POST the first time it boots.

Once you boot to EFI it should automatically run startup.nsh and flash your firmware. Simple.
After this I booted into ESXi 6.0 U2 and both NICs were available and connected.

I will see if this fixed other issues like the windows install issue.
I could prioritize this if someone wants but my next step is to get the Mellanox Infiniband up and running on these boxes and test things like diskless boot.

 

[ttabbal]

Thanks for documenting it. In my case, as you noted, the lights didn't work either, nor did the BMC link. It's good to know how to fix it for future users though. My replacement board had NIC access out of the box for Linux. I didn't test anything else. I updated to the latest firmware and so far the machine has been rock solid stable through 48 hours of burn-in testing.

I also picked up a pair of the Mellanox ConnectX-2 cards, transceivers, and fiber to do a point-to-point link to my main workstation. It's working great, iperf is getting 9.3Gbit/s with no tuning. Just plugged everything in and assigned IPs on a different subnet. Hope yours goes as well.

 

[dragonme]

I just picked up a rackable systems s3012 3u with dual l5640s and 48gb ram with 12 drive cage

the units originally came from amazon according to sources.. and it has a proprietary rackable switch/light/management panel in the front right corner connected to the front panel header and has a lan socket connected via the serial b internal header by the look of things

no drives, no drivers, no raid card, just supplied with 12 empty sleds and a reverse 4 to 8087 to one 4 drive backplane

processors are passive heat sink

3 80mm fans connected to what looks like a power brake out board running at full speed (no management)

I was looking to run either OS X server on it via clover boatloader or and ESXI all in one with napp-it providing zfs pools to esxi...

ideally would like to run that esxi/napp-it off a cheap 40gb ssd connected to one of the internal headers.. that would let me pass the ich10 controller to nap-it to use vs passing individual drives.

THE ISSUES.. this is my first rodeo with server class motherboards and after downloading and trying to read all the intel docs I can find. I am getting more confused on how to just boot and run this thing

1 > the bios might be custom as it was advertised that only 1 nic was usable so I was considering doing the bios recovery to get to the latest firmwares... but which to use... there are so many choices.. and then all the other modules, the bmc, etc.. does the recovery image flash all the modules to the latest?

do I just want the non operating specific efi one?

2 > I have used mostly old bios boards in the past.. my newest board is a dx58so.. it has UEFI but I dont think I have used it that way..
i am getting confused with legacy, bios, efi, uefi, etc boot options.. I cant even get sata drives to show up in the boot menu.. possibly due to the bios being custom? I tried booting a hard drive that has OS X on it and it says 'to boot this volume the motheboard needs to be flashed" or something to the effect...

3> I have been told that if I add hfsplus.efi or hfsplus-64.efi to the bios that it might be able to boot OS X native without boot loader but now I just want to get the system working like a regular desktop type board...

4> what utilities to verify hardware, configure hardware, etc should I download to help me out ?

5> all the thermal settings here seem to require it running in a intel chassis ... I also need to move those fans from direct power to some onboard fan management.. should I just get 2 120mm CPU pwm fans and connect them to the cpu headers and let the system pull air based on cpu workload or temp?

any who... I am really overwhelmed here... and advice or walkthroughs getting me going in the right direction would be appreciated..

I have never used a system that has efi shell...

 

[dragonme]

ok looked into this a bit more.. since this is a rackable enclosure intels FRU configurations WILL SPIN ALL MOTHERBOARD FANS at 100% regardless of what you select in bios or what settings you enter through FRU setup. If you select custom enclosure it will only control the cpu fans

additionally, if you try to fake it by using a intel chassis, its looking for the front panel temperature sensor and if the board is reporting it missing it will run at 100, additionally, if you have chasis intrusion on and you are not using a chassis switch it will report the case as open and spin everything at 100%...

so intel thermal monitoring is basically fubar unless you run it in a intel chassis.. so I guess the only way to quiet down a S5520 motherboard is to install a fan controller ...

my setup is a custom chasis that is 3u but the motherboard is sitting under the drive array which basically makes the motherboard area a 1U and uses passive cooling heat sinks on the 2 l5640 processors.. so I guess I have a couple options.. non great

1> install cpu fans on the back of the case and connect them to the CPU fan headers and manage case flow based on proc temps?

2> leave the fans at 100% running off the direct power of the brake out power board.. fairly noisy

3> install a fan control module with some kind of temperature based control...

thoughts anyone>?

anyone.. Bueler.. Bueler..

 

[ttabbal]

I think you didn't get replies before as you're talking about something uncommon around here on that post, OSX and such. I'll take a stab at some things anyway though...

The BIOS recovery this thread is about is a bit specific to server boards, in my experience. It's a method that allows you to boot the board even if the BIOS chip is corrupted. You set a jumper that tells the CPU to load the BIOS from a USB drive. From there, you can re-flash the onboard chip. As it sounds like this process enabled the NICs for the OP, you might try it on your board as well. It could restore your other NIC.

As for hardware verification, I use memtest86 for a basic test. Then I used a Linux based load tester to push the CPUs and memory for a few days.

For the fan controls... I'm not sure. Mine is in a Supermicro SC847, just the power supply fans are louder than the whole machine I'm upgrading from. I have all the case fans plugged into the motherboard and they all work fine, as do the active CPU coolers. The diagnostics in the BIOS say the fans are running about 1500 RPM. I wouldn't think that's full speed. I do hear some fans spin up to 100% during POST, but they settle down after that. I set it to "Other Chassis" and "Medium Ramp". I also disabled the intrusion detection switch as I didn't bother to connect it.

 

[dragonme]

thanks ttabbal..

leaving out osx.. I am just trying to figure out if I should be booting everything though efi shell using scripts.. or if I should enable/dissable efi.. etc.. at the bios level I am at which is 49 I think.. the board may not be UEFI yet since I see no ref to that .. but EFI boot looks similar having to set paths.. etc.. again mostly all new to me.. I am use to bios boot environment.

so to clarify about your super micro chasis setup.. you updated FRUSDR and went though the setup.. told it it was a custom case and answered the questions and you have fan management though BIOS? which fans are spinning down and settling.. the CPU fans I think are the ones based on ramp.. do you have fans connected to the 4 fan headers that are clustered in the corner of the board?

the reason I ask is my cooling setup is just a a suck system.. all fans connected on the back wall pulling air over the 2 passive cpu heatsinks.. they are 3 80mm server chasis fans by sunan 4.6 amp and sound like a vacuum cleaner without power management.

the custom remote management I think is the rackable systems roamer serial port.. its not the IP version and its connected by ribbon cable to the internal serial b header and the front panel led header. I dont plan on using a serial management server to control a single box.. but reading the intel docs leads me to believe that I should be able to get ipmi over one of the NICs.. its a basic ipmi without intel add on board but I would like to be able to cycle physical power and see bios remotely but I think I need the upgrade card for that.

I guess my concern over doing a bios recovery is that it will flush any customizations to the bios, acpi, bmc, serial settings for the custom serial bmc roamer board etc... but it may not really be a loss if I cant use it anyway...

I am trying to dig around and find a good guide for esxi on this motherboard.. not really familiar with it so I dont know the nuances with 5.2, 5.5 and 6.0 from a driver and hardware standpoint but I think this board will really only run well with server operating systems since intels drivers are a bit limited...

all thoughts appreciated and please dont hesitate to call me out if I am barking up the wrong tree...

 

[ttabbal]

Regarding EFI, just think of it as the "new BIOS". You can totally ignore the EFI command line for anything not related to BIOS type stuff. It's handy for a few things, like messing with flashing hardware cards etc.. Diagnostics, upgrading the BIOS, and it's a little more "modern" when it comes to booting the OS, if you want it to anyway. EFI machines can almost always run in a compatibility mode that boots the machine much like a BIOS machine does. For me, it tends to work BETTER that way. I suspect it's just that it's a kind of early EFI implementation and the newer stuff doesn't like something about it. For example, I kept hanging up installing Proxmox. Couldn't figure out why. Booted using BIOS mode, everything worked perfectly. EFI boot can be quite a bit faster, but it's a server, if you're rebooting it all the time, you're doing it wrong.

For the fans, yes, the case fans are using those plugs in the corner. I'm actually using every fan header on the board as there are 5 case fans and 2 CPU fans. I can't say for certain about the speeds, but when I first power up they seem to spin up to 100% then die down after the POST. This could also be BIOS version related. I'm on the latest, I think that's 61, but I just grabbed the latest version from the Intel website.

IPMI, I'm still getting a handle on it. What seems to be happening is that the BMC can get it's own address and traffic on the main NICs without the OS knowing about it. Without the add-on RMM3, you can do power control and serial port redirection. I don't think you can use the BIOS config screens over the serial redirection, but I could be wrong there. You also need something like ipmitool to do it. With the RMM3, you get a web based interface to the IPMI stuff, and a java based KVM. So you can see the video from the system, type as if you are at the console, etc.. It supports graphics consoles as well. The other cool trick is media redirection. You can send an ISO file and it will boot from it over the network. That's a bit flaky on my XP VM, but works well enough.

I've only tried Linux and FreeNAS on it. Both seem to detect all the hardware fine and run well. I imagine most of it would work well in Windows, but I have no reason to try it. The on board video is an ancient Matrox chip, I think Desqview had a driver for it... I suspect someone at Intel found a crate full of them covered in 3 inches of dust and stuffed them on there to get rid of them.

I don't see any reason you should have problems with esxi. I considered running it myself, but decided that the machine is mostly for storage in my case, and I didn't want to virtualize the storage for various reasons. FreeNAS with jails/Virtualbox/beHyve will work for most anything I can think of for the moment. And there are lots of options should I want to switch it up.

 

[dragonme]

I dont think I am going to virtualize all the storage.. I have a 8 disk 12+TB pool that I created under OS X using zfs and a backup pool as well..

was going to run napp-it on esxi but export the HBA into napp-it so it has the drives native.. then share them out to the OS X server. as for the vms.. yes virtualize their machine disks probably..

 

[unwind-protect]

I wonder whether the old BIOS had just gotten stuck in having assigned interrupts in a scheme that made the nic invisible, and you reset that by reinstalling the same BIOS.

I have a similar problem right now, in that a PCI soundcard that I want to use has decided to block other devices based on interrupt conflicts. Started all of the sudden. Hard to diagnose since when the condition applies you can't see all devices. I'll make a different thread about it. Just thinking whether you had a problem with interrupt conflicts.

 

[dragonme]

I dont know.. the more I look at it the more inclined I am to think that rackable did a custom bios to configure the machine to amazons requirements... in the bios and in system reports the 2nd nic on enabled but it does not even report a mac even though intels bios should add one to nic 1 rom for ipmi sideband on nic one and add 2 to get nic 2s mac.

rackable used no intel power management at all, they just slapped 3 80mm server fans at the back of the chassis running off a molex connector and breakout board, turned the board around to have the back sticking out the front, and added their own roamer serial controller to the serial b header for a power button and on/off led... no NMI button even.. just a very cheap design..

since the board is essentially backwards, and using passive heat sinks...I dont think that setting up the FRU from scratch as a custom board will enable fan management. might try it once I get an operating system on there that permits me to see and record cpu temp values.. I might try having one rear case fan run off the motherboard 4 fan cluster of pins and put one each on the CPUs.. select fast ramping and see if its good enough to draw air based on cpu load. although the cpu really needs 4 pin fans and these are 3 pin..

I am really inclined to use the recovery bios to I think 60 or 61.. the latest grouping and let it update bmc, fru, bios, etc... I should not need to set the jumper to recovery either, as that is only required if the bios won't even post which is not the case.

thanks for sharing.. keep it up!!

 

[dragonme]

well I tried the bios recover as well.. reading the intel docs it says you can do a bios recover without the jumpers and that is what I did.. I updated the entire software stack the FRU,BMC, etc all to the latest versions and did a restore bios 64, and my second nic has an enable/disable but it show mac address <not found> or something like that.

the seller has many of these systems for sale and they all say the same thing, due to configuration only one nic active.. I assumed that it was bios controlled and that as soon as bios was refreshed I would get it back.. no luck yet.

oh and the op makes mention of the iscsi option.. mine is still grayed out.. its there but I cant change settings..

I too redid my FRU, BMC settings for fans when I cleared it out.. its basically a 1U since the drives sit on top of the mobo and it uses passive heatsinks.. did a custom enclosure and told it no fans on the cpu, and the 3 fans on the motherboard.. it didnt find a front sensor and this chassis has the back facing front anyway.. I doubt I will get any fan control out of it but set it to med ramp I think

I probably lost the rackable roamer serial out of band controller hooked into serial B as well, but I wasn't going to buy a serial console server anyway.. hope perhaps I can get that sideband ipmi on nic 1 working alongside regular lan connection as its my only working controller.. assuming its working. have not plugged it into the lan yet..

still messing with getting my operating system to load... ugh.. I HATE intel boards....


any other thoughts?

 

[ttabbal]

I'm a bit confused that you would use a serial port thing on an IPMI board anyway. It has serial redirection built in.... If you go to server management in the BIOS you should see some options for the BMC IP addressing. Plug in the LAN and see what it does. It should be able to use DHCP or static IP and once you know the IP, you should be able to connect to it with IPMI software. If the LAN ports work at all, the BMC should be able to use them. It seems to be wired in there somehow. I only have one of them connected and I'm passing traffic for the BMC and the main OS.

What issue are you having with the OS? The only time I had boot issues was trying to use EFI boot with some newer Linux builds. In the boot menu, it shows entries for both EFI and BIOS boot devices. So just don't pick the EFI ones. I can confirm that works with Proxmox, Ubuntu, and FreeNAS.

 

[dragonme]

thanks.. and yes confusing.. I dont know why rackable would put roamer serial controllers on them and why would the kill off a nic on purpose? also, intel sold add on remote management that did proper remote console and would have given the unit 2 fully usable nics plus a dedicated ipmi remote terminal with KVM... seems bizarre .. but then amazon ordered them that way so they must have had a good reason...

I am a bit confused on how a single lan port can communicate with the lan and do sideband ipmi on the same socket using different subnets.. again I am no IT pro.. but I have not gotten there yet...

yeah.. as for intel and there UEFI EFI crap bioses this is not my first run in. the DX58so was also supposed to support UEFI and it is equally as bad.. I would like to hackintosh it like I have other systems but its proving frustrating.. plus there are quite a few server grade options in the bios I dont fully understand and dont know how I should set them to get OS X going.. I am going to give it a bit longer and if I keep butting heads with it.. I will likely just install esxi on it and give that a whirl..

I tried to boot a disk that should have been bootable.. and boots on my DX58 and it says something like " this boot loader requires a flashed motherboard' or something to that effect... its driving me a bit bonkers.. !!!

 

[techtoys]

I just started reading the FRUSDR update document from Intel on this.
It states that it is possible to create a custom configuration file for a 3rd party chassis but recommends against it.
of course, you could wreck the system with insufficient cooling on some unknown chassis so 100% fans is likely the safest.

It would be a chore but there must be a way to check the operating conditions and build your own master.cfg.

 

[techtoys]

I spoke to Patrick at the meetup. We both thought there are reasons that they would purposely turn off a nic. The systems I have were on a 10G network and I think they axed the nic bios. You may be able to recover it but I did have flashing lights so it looked active.

 

[techtoys]

I have ESXi 6 running on this now and did have ESXi 5.5 also running. 6 worked OTB.
Ubuntu live disk booted as well.
I also have booted by attaching an image (but I used the Avocent system)

currently trying to get remote diskless boot via mellanox card or nic

 

[dragonme]

tech toys.. you are having more luck than I but I figured out one issue with installation..

while trying to install OS X though the onboard usb ports on the back of the board.. it was behaving strangely.. and this board should boot the loader I am using but was not having it.. in a fit of rage I went a connected a usb header to the internal connector and it installed finally...


ugh..

so still have one nic not reporting a mac in bios but it has an enable disable option... updating everyone and even doing a recover flash of 64 did not fix that.. next would be doing the recovery flash using the jumper but the docs said that it should not make a difference.. but something still is not cleared out from whatever hacking Rackable systems did to it?

the system passively cools the 2 l5640s.. with 3 80mm fans pulling air out back.. that board is turned 180 so the connectors are on front..

if I choose custom FRU what would be my best bet to get some of the fans to spin down.. should I connect all 3 to the internal chassis headers and tell FRU that no cpus have fans.. or should I connect 1 ea 3pin fans to the cpus and 1 to the chassis.. according to intel docs, if I choose custom enclosure it will spin ALL chasis fans at 100%.. but cpu fans would be throttled... ?

will esxi 6 install straight away and have good hardware support or should I install esxi 5.5 first and upgrade without upgrading drivers?

Thanks techtoys ...

 

[dragonme]

anyone have any additional ideas on how I might recover my second NIC.. ?

 

[dragonme]

The main issue appears to be bios modifications that will not allow me to access all functions and, more severely, edited out the second NIC controller

The board is showing no MAC address for the second NIC.

When I received the system it had
BIOS 47
BMC 49 PIA 10.49 Boot .18
ME 1.12.0070
SRD null package .21

I updated the software stack to the latest intel, and no change

I did a recovery bios to the latest version, no change

According to intel documents, physically the Left NIC is NIC1 and its sticker has a MAC address that ends in 66
In bios, NIC 1 shows a mac address ending in 67 and NIC2 has no address at all but its enabled

Physically, NIC1 does not connect, it lights up left light only, but Its actually NIC2 (NIC1 in bios) that connects as MAC ending in 67

so obviously Rackable customized the BIOS / HARDWARE to lock down the board

so to recap...

Left NIC does not connect to router for full access
Right NIC connects and is fully usable

When connected to the router though NIC2 with a address of 192.168.0.100 and setting in bios the BMC to static on 192.168.0.2 I can ping the router and both .2 and .100 addresses but perhaps it was still all through the same physical NIC I will have to set an address for the other bmc controller and see if all addresses work..

Is there any way to remove these customizations and return the hardware to Intel Stock config?

thoughts guys... thanks!!

 

[dragonme]

OK

update.. so I have both NIC1 and NIC2 plugged into a router

NIC2 physically (right NIC) is acting as NIC1 in bios, and has a mac address one higher than the sticker on the motherboard

I set the BMC nic to static and gave it an address in the same DHCP subnet

If I ping the router I get activity on the right NIC, NIC2 physical NIC1 bios.. lights on both the router socket and the NIC blink

If I ping the BMC address, I get activity lights on BOTH NIC1 and NIC2 on the motherboard and the ROUTER activity lights for those ports..

so it looks like the custom bios edit removed NIC1 physical from bios, made NIC2 NIC1 in bios settings, and is allowing NIC1 to be used for management only since it does not show up in my routers list as even being connected but its pingable.

?????