Haunted? SC847 - won't power up in garage rack, fine on dining table

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
Honestly I don't even know where to begin with this one, so if it's a bit disjointed I apologise.

I have a server built in a SuperMicro SC847 chassis - X9SCM-F, E3-1230, 32GB, 2x X520, LSI RAID, 36x 3TB and 8x 300GB SSD, so it's heavy! It's worked fine for ... six months at a guess? I just instrumented it (new grafana deployment) and wondered why the fans were showing 0 RPM, so I shut it down to check (turns out all the fans are on the backplanes, not the mainboard). Oh well, that's easy.

Now it won't power up.

Well - to be specific. It won't power up in the rack. All I did was disconnect everything and slide it out so I could remove the top and check the fans, then reassemble and slide it in. The orange "has power" LED on each PSU is lit (goes green on server powerup).

I swapped the power cables, and the rack PDU to which the server is connected, no change. I swapped power supplies from the "spare" chassis. Eventually I decided I couldn't fix it in situ so I pulled it and placed it on the dining table (because where else). Powered up first go.

Crap. OK that's fine, I'm a masochist. Disconnected it all. Moved back to the rack in the garage. Installed.

Nope. Not a freaking thing.

Swapped power again. Nope. Pulled it out of the rack and tried running it in the garage but outside the rack. No go. Ran the power from the dining room out to the garage, used the same cords and power board that work in the dining room. Not a damn peep.

Brought it inside again and connected it on the dining table. Powered up perfectly and it's humming away (literally, I forgot how loud it can be) on the dining table. Which sucks when you don't have the 10Gb network inside the house, just the rack/garage, and it is the storage for 47 VMs and the media for Plex etc. (I could run some house ports at 10Gb, but I don't have 10GBaseT NICs - I'm all SFP+). 47 VMs and media streaming over 1Gbps is a bit of a stretch.

So ... ideas? Anything? I'm completely flummoxed as are several people I've discussed it with.

Edit: There's a stack of things running just fine in that same rack off the same base PDU - a pair of C6100 chassis, LB4M, a pair of PowerConnect 8100's, two whitebox rack servers (DCs). So it's not JUST the one thing in the rack.
 
Last edited:

xnoodle

Active Member
Jan 4, 2011
258
48
28
Can you try powering it on via IPMI or shorting the pins for power on?

Weird that the power button is somehow magically disabled in the garage...
 

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
Can you try powering it on via IPMI or shorting the pins for power on?

Weird that the power button is somehow magically disabled in the garage...
Oh right - so the board doesn't even give me a link light in the garage. Normally the IPMI port would get a 100Mbps link, the two X520's I think have LEDs or there's something green on the X9SCM in that area. Nothing. It's like it goes into the garage and loses all ability to provide DC power.

And the board is set to power on automatically after an outage, too.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Try taking it to the house again and starting it up, then back to garage and try again and repeat until chassis is confused about its location and it works o_Oo_O

This is a perplexing one...
 

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
Try taking it to the house again and starting it up, then back to garage and try again and repeat until chassis is confused about its location and it works o_Oo_O

This is a perplexing one...
Yeah. Yeah it is. I wish this were a joke or a prank, but you just can't make this sort of crap up!

I think I'll sleep on it since it's after 1am.
 

xnoodle

Active Member
Jan 4, 2011
258
48
28
So there's no DC power getting to the motherboard from the PDB it sounds like?
Can you try pulling out one of the PSUs and powering up from the other?
Pull out PSU0, try powering up via PSU1, and vice versa.

Maybe the PDB is on the fritz and the house event was just a fluke.

Do you have a spare PDB?

I bought two for my work bench, since I had a bunch of extra SM PSUs.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
Sounds to me like there is enough flex in the chassis when it's on its rails to move the mainboard a smidge on it's mounting posts. Could it be a screw somewhere that has vibrated loose? Of course, it could also be a poor solder joint on the board, but that's worst case. I would re-seat everything, including the CPU and then loosen and re-tighten the mainboard screws so that nothing is being put under any strain. It might also be worthwhile checking that all the mainboard mounting posts actually have a screw in them and that you don't have a rogue one hiding under the board without a screw hole :)
 

marcoi

Well-Known Member
Apr 6, 2013
1,532
288
83
Gotha Florida
Did you try running a power line from a different socket? Avoiding the rack power supply? Also where in the rack is it? Did you try putting it on the lowest spot maybe like pricklypunter mentioned, the chassis is stressed causing issues?
 

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
All the connections have been pulled and replugged (I had to, as I wanted to slide the motherboard tray back after the first time it failed to start). Also, I tried the cables I had successfully used in the dining room on the end of an otherwise known working extension lead and the same powerboard, with the server located in the garage (both in the rack and on an insulating and isolated plastic tub). Each time no DC delivered.
 

pricklypunter

Well-Known Member
Nov 10, 2015
1,708
515
113
Canada
If you have re-seated everything, loosened and re-tightened the Mainboard mounting screws and double checked all power connections, I think it's time to break out the multi-meter...

Check your standby 5V rail at the Mainboard, check for power good signal back to the PDB when you turn it on etc, see if the Mainboard is at least receiving power and telling the power supply to start. Obviously it's getting power as far as the mains is concerned, and I think you can rule out the power supplies being the issue, that leaves the PDB, Mainboard and Wiring :)
 

fractal

Active Member
Jun 7, 2016
309
69
28
33
What do you have connected to the system when you run it on the dining room table? Video? Network?

This has some of the symptoms of a ground issue, but that is easy to eliminate if nothing else is connected.
 

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
On the dining table - 2x power, 2x Cat6A LAN, 1x Cat6A IPMI. The LAN cabling goes back to the LB4M in the rack.

In the rack - 2x power, 2x SFP+ LAN, 1x Cat6A IPMI, USB KVM POD (Cat5E). The IPMI cabling goes back to the LB4M in the rack and the SFP+ (DAC, not fiber) go back to the two Dell 8024Fs.

On the plastic tub ("stand") - 2x rack power or 2x house power (i.e. "do I have some weird grounding thing" mode). No other cabling.
 

TeeJayHoward

Active Member
Feb 12, 2013
376
112
43
I had a similar issue with an old Dell system a while back. The motherboard was shorting against the case when it was in the rack due to the rack rails being off by 1/3rd of a U. Does it power up if the chassis is fully extended on the rails?
 

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
I had a similar issue with an old Dell system a while back. The motherboard was shorting against the case when it was in the rack due to the rack rails being off by 1/3rd of a U. Does it power up if the chassis is fully extended on the rails?
Sadly no - as I mentioned, it doesn't even power up on an electrically isolated plastic storage tub "table". When in the rack it's also in the right position (whole RU top and bottom)
 

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
Check the mains voltage of the power points, have seen a bad connection give low enough voltage to cause failure to power up. Possibly not getting power good signal from motherboard.
 

sfbayzfs

Active Member
May 6, 2015
259
143
43
SF Bay area
Did you ever track this down?

I have also had problems with slightly low AC voltage causing some PSUs to not turn on in the past, moving to another outlet with another volt or two fixed the problem for me - these are supposed to be full range active PFC PSUs meant for 110V, 208V, and 220V though.

Are the PSU's LEDs by the power cord showing the orange, but not going green when you try to power on, or no light on at all, or a different color? (I assume they go red when they fail)

Another thing you could look for is go all around the chassis and tighten all of the little screws holding it together, if they are not tight enough, there could be just enough flex when supported by rails instead of the bottom of the chassis to separate a broken trace or cold solder joint on your motherboard just enough enough for it to not work.
 

DavidRa

Infrastructure Architect
Aug 3, 2015
329
152
43
Central Coast of NSW
www.pdconsec.net
Did you ever track this down?

I have also had problems with slightly low AC voltage causing some PSUs to not turn on in the past, moving to another outlet with another volt or two fixed the problem for me - these are supposed to be full range active PFC PSUs meant for 110V, 208V, and 220V though.

Are the PSU's LEDs by the power cord showing the orange, but not going green when you try to power on, or no light on at all, or a different color? (I assume they go red when they fail)

Another thing you could look for is go all around the chassis and tighten all of the little screws holding it together, if they are not tight enough, there could be just enough flex when supported by rails instead of the bottom of the chassis to separate a broken trace or cold solder joint on your motherboard just enough enough for it to not work.
Not satisfactorily. I pulled it back to the table, tried again a couple of weeks later and it worked.

I did have the "won't turn green"scenario for the PSUs. I can't come up wiht a scenario where the AC would be sufficiently low to not work (Australia so 230V nominal, 240V actual) and it's residential so it's not like we get the industrial sag and overshoot that some of my customers see (industrial areas can hit 265V sometimes).

Fact is it's been up 24x7 for months, and other than the odd random crash (this board has always done that but replacing is too expensive right now) it's otherwise been reasonably stable.