H12SSL-I Stuck at "bmc initiating"

[oneplane]

I'm not seeing a part number on these. Should the voltage be the same on all 3? I'm getting 1.2v on the top one then 30mv on the other 2. Is there a way of disabling the BMC?

You can usually not do that from the BIOS, but some boards have jumpers for that. The problem is that for the firmware to talk to the BMC it needs to have modules and those are either present or absent in the firmware image. Most of them then have either a timeout, or a enable/disable GPIO (in the shape of a jumper).

For your specific case it might help to explain the problem you have. If you also have a burnt chip, that's the same, but if you don't, then it's different.

 

[Dick Lee]

" Besides that, what about the burn mark on that part? " - Since I have no idea how to fix it even if that's something wrong with it, I would put it on hold until I know whether the flashing, if any, could work.

 

[null224]

You can usually not do that from the BIOS, but some boards have jumpers for that. The problem is that for the firmware to talk to the BMC it needs to have modules and those are either present or absent in the firmware image. Most of them then have either a timeout, or a enable/disable GPIO (in the shape of a jumper).

For your specific case it might help to explain the problem you have. If you also have a burnt chip, that's the same, but if you don't, then it's different.

My chip is not burned but the BMC is also stuck initializing as well. On PSU power up it goes into initializing and doesn't stop fast blinking even after an hour+. If I power up all fans are full blast and I get no video. Ethernet ports blink but no IPMI access from any port. I did see some other boards have the option that's why I was hoping there was a jumper I was missing but it doesn't appear this board has one. The manual shows the VGA port is tied to BMC so maybe that's why. I was mostly hoping for a BMC bypass.

 

[Dick Lee]

My chip is not burned but the BMC is also stuck initializing as well. On PSU power up it goes into initializing and doesn't stop fast blinking even after an hour+. If I power up all fans are full blast and I get no video. Ethernet ports blink but no IPMI access from any port. I did see some other boards have the option that's why I was hoping there was a jumper I was missing but it doesn't appear this board has one. The manual shows the VGA port is tied to BMC so maybe that's why. I was mostly hoping for a BMC bypass.

Had you changed PCIE cards/other hardwares immediately before the problem appeared?

 

[RolloZ170]

My chip is not burned but the BMC is also stuck initializing as well. On PSU power up it goes into initializing and doesn't stop fast blinking even after an hour+

*fast blinking* your BMC does not run. check your UID switch, if stuck pressed down this can cause this.
check outside a case without CPU/RAM. if still fast blink, RMA.

 

[oneplane]

My chip is not burned but the BMC is also stuck initializing as well. On PSU power up it goes into initializing and doesn't stop fast blinking even after an hour+. If I power up all fans are full blast and I get no video. Ethernet ports blink but no IPMI access from any port. I did see some other boards have the option that's why I was hoping there was a jumper I was missing but it doesn't appear this board has one. The manual shows the VGA port is tied to BMC so maybe that's why. I was mostly hoping for a BMC bypass.

Since the BMC is the GPU on this board, you can't really skip the entire BMC. But without working flash, the BMC turns into a dumb I/O controller. If the BIOS says it is waiting for the BMC, it means that it asks the BMC for something, but the BMC doesn't respond. Usually it has like a 10 minute timeout, but regardless, if you want to find out if it is broken due to missing/corrupt flash you can read it and see what the contents are. This does require an SPI interface. If you have a raspberry pi or a beaglebone around you can use that. It doesn't require a special 'programmer', it's just an SPI flash chip. Do take care on the voltage levels, the chips all have markings on them and the data sheet for those markings will tell you if it is a 1.8V, 3V or 5V part.

If the UID button is stuck, that would save you a whole lot of work.

 

[RageBone]

I'm not seeing a part number on these. Should the voltage be the same on all 3? I'm getting 1.2v on the top one then 30mv on the other 2. Is there a way of disabling the BMC?

There is some writing on them, but i can only glimpse it under a microscope and special lighting.
I am very sure that they are not supposed to have the same voltage, or just 0.03V on the Output.
So you for sure have a problem with the voltage regulation of those two, likely preventing the BMC from being able to do its job.
1.2V sounds like it is for the Dram of the BMC.
Other voltage ranges that i would expect / suspect on those are 1.0, 0.9V, 0.8V and 1.8V.
Nothing i have measured, so reality might differ but i have no good board to check.

If you are sure that you did not hit those ICs with anything and there are no scratches visible, RMA with SM is a good idea.
In any other case, SM will likely refuse repairs, service and replacement.

" Besides that, what about the burn mark on that part? " - Since I have no idea how to fix it even if that's something wrong with it, I would put it on hold until I know whether the flashing, if any, could work.

Thing is, as long as those parts are potentially dead, you can absolutely forget about flashing.
If that Voltage-Regulator IC is busted, the board is absolutely without any doubt defective and no firmware can ever make it work.

I can only recommend you take good closeup pictures where those ICs are clearly visible and where you / we can make out if they are an issue or not.

If we find that they are not an issue, please go ahead with flashing, i wish you much success and hope it works after that.
If they are indeed really busted and burned, don't even try flashing.

Pictures first, flashing later.
At least that is what i can recommend and what makes sense to me.

Everything else can come after we have seen better pictures.
So please, post good pictures.

 

[Debtor8100]

This thread sound eerily familiar to my experiences yesterday. I ordered H12SSL-i and Epyc 7320P for my home server and could not get past BMC initializing part. I tried the basic diagnostics including:
- Running without PCIe or USB devices
- Running with only one RAM stick
- Running without RAM
- Running outside case on antistatic platform
- Removing CMOS-battery and shorting the contacts
- Shorting the motherboards BIOS reset header as per instructions on manual
- Reseating CPU

No boot, no beeps, nothing. Just a fast blinking BMC light. I even tried calling Supermicro Technical Support and they basically told me to test everything I had already done. They also suspect that the BMC is bricked. The BMC initialization should only take about one minute according to them, mine didn't initialize in 2 hours when I tried to just give it more time.

Currently I can still test with another PSU (although the one I tested with is now running the old motherboard just fine) and maybe try running without CPU (should I even try this?). I hope this is not a wide spread issue with these H12-series boards. New board is on the way at the moment and I hope this wont happen again.

 

[RageBone]

This thread sound eerily familiar to my experiences yesterday. I ordered H12SSL-i and Epyc 7320P for my home server and could not get past BMC initializing part. I tried the basic diagnostics including:
- Running without PCIe or USB devices
- Running with only one RAM stick
- Running without RAM
- Running outside case on antistatic platform
- Removing CMOS-battery and shorting the contacts
- Shorting the motherboards BIOS reset header as per instructions on manual
- Reseating CPU

No boot, no beeps, nothing. Just a fast blinking BMC light. I even tried calling Supermicro Technical Support and they basically told me to test everything I had already done. They also suspect that the BMC is bricked. The BMC initialization should only take about one minute according to them, mine didn't initialize in 2 hours when I tried to just give it more time.

Currently I can still test with another PSU (although the one I tested with is now running the old motherboard just fine) and maybe try running without CPU (should I even try this?). I hope this is not a wide spread issue with these H12-series boards. New board is on the way at the moment and I hope this wont happen again.

Please take some close up pictures of the area around the BMC.
Specifically of those small silver rectangle chips towards the edge of the board.

Did you install any PCIe cards or did screw the board into a case already?

 

[i386]

Where you do you buy these boards?

 

[Debtor8100]

Please take some close up pictures of the area around the BMC.
Specifically of those small silver rectangle chips towards the edge of the board.

Did you install any PCIe cards or did screw the board into a case already?

Here are some pictures, sorry for the quality. This is the best my phone can come up with, the distortions are quite massive.




I was feeling brave at first so I just installed everything on the board (network card, HBA, SAS expander, ASUS 4x NVME card). Everything went smooth, I did not bump or scratch anything. When the system wouldn't post I started to take things apart and ended up with the board on a cardboard box, doing the tests I wrote on my earlier message.

 

[RageBone]

http://imgur.com/a/zBuity2

Click the Link, i added pictures of my Board that are worth a look.

Questions?
Label on U4 is
OCKI 15X or 0CKI 15X

 

[Debtor8100]

http://imgur.com/a/zBuity2

Click the Link, i added pictures of my Board that are worth a look.

Questions?

Yeah I noticed the tiny scratch on the edge. I would say it's on the surface level and so far in the edge that it shouldn't kill the BMC. Motherboards have a little buffer area on the edge just for tiny impacts during install. If that really killed the board the design is beyond stupid.

The middle one of the circled IC's however has taken a hit. It doesn't look as bad as on the one on your second picture, but it certainly has taken a tiny hit. That might've done it still. If that is really all it takes to brick this board, this is just comical. I have installed too many computers to count and never have I broken a motherboard without even noticing that I might've done so. I've had a screwdriver slip and killed few memory slots on old board but this is a whole different level of fragile.

It's not like a dropped a PCIe card on top of it full force. My tiny Mellanox Connect X3 was in the bottom slot and it might have ever so slightly touched the motherboard, but not even in a way that I would deem noteworthy.

But yeah I know, it seems I broke it. I am pissed to myself but also Supermicro for this design...

 

[RageBone]

Yeah I noticed the tiny scratch on the edge. I would say it's on the surface level and so far in the edge that it shouldn't kill the BMC. Motherboards have a little buffer area on the edge just for tiny impacts during install. If that really killed the board the design is beyond stupid.

The middle one of the circled IC's however has taken a hit. It doesn't look as bad as on the one on your second picture, but it certainly has taken a tiny hit. That might've done it still. If that is really all it takes to brick this board, this is just comical. I have installed too many computers to count and never have I broken a motherboard without even noticing that I might've done so. I've had a screwdriver slip and killed few memory slots on old board but this is a whole different level of fragile.

It's not like a dropped a PCIe card on top of it full force. My tiny Mellanox Connect X3 was in the bottom slot and it might have ever so slightly touched the motherboard, but not even in a way that I would deem noteworthy.

But yeah I know, it seems I broke it. I am pissed to myself but also Supermicro for this design...

I feel the absolute same.
Luckily, i didn't personally break such board myself, but still, i got them and it is infuriating.

Noteworthy is that only the H12SSL and H12DSI seem to use those ICs.
None of the newer SM Intel boards do.

Those ICs are Bare-Silicon-Die 6 ball BGA DC-DC Voltage-Regulator that take in 3.3V-StandBy and produce lower voltages for the BMC.
Similar ICs can be found in phones and laptops for a while. Usually those are not for sale and widely available.
Usually they are also bound to an NDA, making them unobtanium.

I have a case where the damage made one fail catastrophically with 3.3V shooting into the BMC and killing it.
3.3VSB is shorted 0Ohm to GND, causing the 3.3VSB converter to not regulate, rendering the board totally inert.
No Leds, no reaction to anything, as if it is off.
Just to have mentioned another picture of symptoms.


Also worth mentioning, the guy i got that H12ssl from, asked SM for a repair and was turned down because the board was visibly scratched.
He told me that it was running and then just stopped.

 

[RolloZ170]

No boot, no beeps, nothing. Just a fast blinking BMC light

means the BMC does not boot. ID switch stuck pressed ? PSU +5VSB to weak.

 

[RolloZ170]

and maybe try running without CPU (should I even try this?).

sure, BMC heartbeat LED should blinking in second periods after a short init.time.

 

[Debtor8100]

I feel the absolute same.
Luckily, i didn't personally break such board myself, but still, i got them and it is infuriating.

Noteworthy is that only the H12SSL and H12DSI seem to use those ICs.
None of the newer SM Intel boards do.

Those ICs are Bare-Silicon-Die BGA DC-DC Voltage-Regulator that take in 3.3V-StandBy and produce lower voltages for the BMC.
I have a case where the damage made one fail catastrophically with 3.3V shooting into the BMC and killing it.
3.3VSB is shorted 0Ohm to GND, causing the 3.3VSB converter to not regulate, rendering the board totally inert.
No Leds, no reaction to anything, as if it is off.
Just to have mentioned another picture symtoms.

Also worth mentioning, the guy i got that H12ssl from, asked SM for a repair and was turned down because the board was visibly scratched.
He told me that it was running and then just stopped.

Considering the placement, fragility and non-existent protection of those ICs, i think this at least boarders on being a trap.

If I just had this info before purchasing it might've swayed my opinion to some other board. This sounds horrible.

Knowing this I shall treat the new board as if it were made of glass. Or actually something far more fragile. These sort of business practice wouldn't end well to Supermicro if it was European company. Manufacturer should have some sort of accountability if the design is this vulnerable. They should atleast give extra warnings to be extra careful installing PCIe cards. I am far from a novice and know not to use too much force installing components and did not even notice that I might've broken something. That is just wildly bad desing...

There is however a slight difference in the symptoms. I get the fast blinking light atleast, but that is about it. Maybe the IC is not damaged enough to completely brick everything, but not intact enough to allow BMC to initialize. Anyway, I feel like I need a beer or twenty...

means the BMC does not boot. ID switch stuck pressed ? PSU +5VSB to weak.

I think we diagnosed my problem. PSU has been running fine for a year and is currently running my server with the old motherboard. ID switch is not stuck.

 

[RolloZ170]

PSU has been running fine for a year and is currently running my server with the old motherboard

even a healthy PSU needs to supply at least +5VSB 3Amps. 2Amps = not work.

 

[RageBone]

There is however a slight difference in the symptoms. I get the fast blinking light atleast, but that is about it. Maybe the IC is not damaged enough to completely brick everything, but not intact enough to allow BMC to initialize.

The symptoms can vary widely because it depends on the kind of damage and the amount of damaged ICs.
Lets say, one of the three, the one for the 1.2V for its memory, is damaged enough and in a way to not produce any output voltage.
That would render the BMC core and logic functional, just the memory for everything else like the OS, management and GPU would not be working.
I find it likely that the BMC would indicate such an issue with fast blinking.


Possibly similar in case just the Core-voltage for the ARM-Cores inside is not being produced but the rest is there.

With more damaged ICs, it gets more likely that the BMC would not even be able to blink a LED.

If the damage is more catastrophic in nature, it could permanently kill the BMC and even further impact the board.

You can measure the output-voltages with a multi-meter and make sure if and what the case is.

 

[RolloZ170]

These sort of business practice wouldn't end well to Supermicro if it was European company. Manufacturer should have some sort of accountability if the design is this vulnerable. They should atleast give extra warnings to be extra careful installing PCIe cards. I am far from a novice and know not to use too much force installing components and did not even notice that I might've broken something. That is just wildly bad desing...

very exaggerated. NO manufacturer mounts useless tiny smd elements on they're motherboards. even if you brick off a little resistor a motherboard becomes fully/partialy unfunctional regardless of the manufacturer.