Haunted? SC847 - won't power up in garage rack, fine on dining table

Discussion in 'General Chat' started by DavidRa, Feb 18, 2017.

  1. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    Honestly I don't even know where to begin with this one, so if it's a bit disjointed I apologise.

    I have a server built in a SuperMicro SC847 chassis - X9SCM-F, E3-1230, 32GB, 2x X520, LSI RAID, 36x 3TB and 8x 300GB SSD, so it's heavy! It's worked fine for ... six months at a guess? I just instrumented it (new grafana deployment) and wondered why the fans were showing 0 RPM, so I shut it down to check (turns out all the fans are on the backplanes, not the mainboard). Oh well, that's easy.

    Now it won't power up.

    Well - to be specific. It won't power up in the rack. All I did was disconnect everything and slide it out so I could remove the top and check the fans, then reassemble and slide it in. The orange "has power" LED on each PSU is lit (goes green on server powerup).

    I swapped the power cables, and the rack PDU to which the server is connected, no change. I swapped power supplies from the "spare" chassis. Eventually I decided I couldn't fix it in situ so I pulled it and placed it on the dining table (because where else). Powered up first go.

    Crap. OK that's fine, I'm a masochist. Disconnected it all. Moved back to the rack in the garage. Installed.

    Nope. Not a freaking thing.

    Swapped power again. Nope. Pulled it out of the rack and tried running it in the garage but outside the rack. No go. Ran the power from the dining room out to the garage, used the same cords and power board that work in the dining room. Not a damn peep.

    Brought it inside again and connected it on the dining table. Powered up perfectly and it's humming away (literally, I forgot how loud it can be) on the dining table. Which sucks when you don't have the 10Gb network inside the house, just the rack/garage, and it is the storage for 47 VMs and the media for Plex etc. (I could run some house ports at 10Gb, but I don't have 10GBaseT NICs - I'm all SFP+). 47 VMs and media streaming over 1Gbps is a bit of a stretch.

    So ... ideas? Anything? I'm completely flummoxed as are several people I've discussed it with.

    Edit: There's a stack of things running just fine in that same rack off the same base PDU - a pair of C6100 chassis, LB4M, a pair of PowerConnect 8100's, two whitebox rack servers (DCs). So it's not JUST the one thing in the rack.
     
    #1
    Last edited: Feb 18, 2017
  2. xnoodle

    xnoodle Active Member

    Joined:
    Jan 4, 2011
    Messages:
    259
    Likes Received:
    48
    Can you try powering it on via IPMI or shorting the pins for power on?

    Weird that the power button is somehow magically disabled in the garage...
     
    #2
  3. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    Oh right - so the board doesn't even give me a link light in the garage. Normally the IPMI port would get a 100Mbps link, the two X520's I think have LEDs or there's something green on the X9SCM in that area. Nothing. It's like it goes into the garage and loses all ability to provide DC power.

    And the board is set to power on automatically after an outage, too.
     
    #3
  4. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,831
    Likes Received:
    1,484
    Try taking it to the house again and starting it up, then back to garage and try again and repeat until chassis is confused about its location and it works o_Oo_O

    This is a perplexing one...
     
    #4
  5. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    Yeah. Yeah it is. I wish this were a joke or a prank, but you just can't make this sort of crap up!

    I think I'll sleep on it since it's after 1am.
     
    #5
  6. xnoodle

    xnoodle Active Member

    Joined:
    Jan 4, 2011
    Messages:
    259
    Likes Received:
    48
    So there's no DC power getting to the motherboard from the PDB it sounds like?
    Can you try pulling out one of the PSUs and powering up from the other?
    Pull out PSU0, try powering up via PSU1, and vice versa.

    Maybe the PDB is on the fritz and the house event was just a fluke.

    Do you have a spare PDB?

    I bought two for my work bench, since I had a bunch of extra SM PSUs.
     
    #6
  7. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    Yeah, tried one at a time, each. Both power supplies "fail" the same way in the garage. I do have a complete spare chassis, I'm about to build the backup server into it though.
     
    #7
  8. pricklypunter

    pricklypunter Well-Known Member

    Joined:
    Nov 10, 2015
    Messages:
    1,524
    Likes Received:
    434
    Sounds to me like there is enough flex in the chassis when it's on its rails to move the mainboard a smidge on it's mounting posts. Could it be a screw somewhere that has vibrated loose? Of course, it could also be a poor solder joint on the board, but that's worst case. I would re-seat everything, including the CPU and then loosen and re-tighten the mainboard screws so that nothing is being put under any strain. It might also be worthwhile checking that all the mainboard mounting posts actually have a screw in them and that you don't have a rogue one hiding under the board without a screw hole :)
     
    #8
  9. Red_Devil_24

    Red_Devil_24 New Member

    Joined:
    Jan 2, 2017
    Messages:
    16
    Likes Received:
    3
    Have you checked all the connections, you could have a loose one.
     
    #9
  10. marcoi

    marcoi Well-Known Member

    Joined:
    Apr 6, 2013
    Messages:
    1,334
    Likes Received:
    205
    Did you try running a power line from a different socket? Avoiding the rack power supply? Also where in the rack is it? Did you try putting it on the lowest spot maybe like pricklypunter mentioned, the chassis is stressed causing issues?
     
    #10
  11. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    All the connections have been pulled and replugged (I had to, as I wanted to slide the motherboard tray back after the first time it failed to start). Also, I tried the cables I had successfully used in the dining room on the end of an otherwise known working extension lead and the same powerboard, with the server located in the garage (both in the rack and on an insulating and isolated plastic tub). Each time no DC delivered.
     
    #11
  12. pricklypunter

    pricklypunter Well-Known Member

    Joined:
    Nov 10, 2015
    Messages:
    1,524
    Likes Received:
    434
    If you have re-seated everything, loosened and re-tightened the Mainboard mounting screws and double checked all power connections, I think it's time to break out the multi-meter...

    Check your standby 5V rail at the Mainboard, check for power good signal back to the PDB when you turn it on etc, see if the Mainboard is at least receiving power and telling the power supply to start. Obviously it's getting power as far as the mains is concerned, and I think you can rule out the power supplies being the issue, that leaves the PDB, Mainboard and Wiring :)
     
    #12
  13. fractal

    fractal Active Member

    Joined:
    Jun 7, 2016
    Messages:
    309
    Likes Received:
    67
    What do you have connected to the system when you run it on the dining room table? Video? Network?

    This has some of the symptoms of a ground issue, but that is easy to eliminate if nothing else is connected.
     
    #13
  14. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    On the dining table - 2x power, 2x Cat6A LAN, 1x Cat6A IPMI. The LAN cabling goes back to the LB4M in the rack.

    In the rack - 2x power, 2x SFP+ LAN, 1x Cat6A IPMI, USB KVM POD (Cat5E). The IPMI cabling goes back to the LB4M in the rack and the SFP+ (DAC, not fiber) go back to the two Dell 8024Fs.

    On the plastic tub ("stand") - 2x rack power or 2x house power (i.e. "do I have some weird grounding thing" mode). No other cabling.
     
    #14
  15. TeeJayHoward

    TeeJayHoward Active Member

    Joined:
    Feb 12, 2013
    Messages:
    374
    Likes Received:
    106
    I had a similar issue with an old Dell system a while back. The motherboard was shorting against the case when it was in the rack due to the rack rails being off by 1/3rd of a U. Does it power up if the chassis is fully extended on the rails?
     
    #15
  16. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    Sadly no - as I mentioned, it doesn't even power up on an electrically isolated plastic storage tub "table". When in the rack it's also in the right position (whole RU top and bottom)
     
    #16
  17. Tom5051

    Tom5051 Active Member

    Joined:
    Jan 18, 2017
    Messages:
    230
    Likes Received:
    28
    Check the mains voltage of the power points, have seen a bad connection give low enough voltage to cause failure to power up. Possibly not getting power good signal from motherboard.
     
    #17
  18. sfbayzfs

    sfbayzfs Active Member

    Joined:
    May 6, 2015
    Messages:
    245
    Likes Received:
    102
    Did you ever track this down?

    I have also had problems with slightly low AC voltage causing some PSUs to not turn on in the past, moving to another outlet with another volt or two fixed the problem for me - these are supposed to be full range active PFC PSUs meant for 110V, 208V, and 220V though.

    Are the PSU's LEDs by the power cord showing the orange, but not going green when you try to power on, or no light on at all, or a different color? (I assume they go red when they fail)

    Another thing you could look for is go all around the chassis and tighten all of the little screws holding it together, if they are not tight enough, there could be just enough flex when supported by rails instead of the bottom of the chassis to separate a broken trace or cold solder joint on your motherboard just enough enough for it to not work.
     
    #18
  19. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    Not satisfactorily. I pulled it back to the table, tried again a couple of weeks later and it worked.

    I did have the "won't turn green"scenario for the PSUs. I can't come up wiht a scenario where the AC would be sufficiently low to not work (Australia so 230V nominal, 240V actual) and it's residential so it's not like we get the industrial sag and overshoot that some of my customers see (industrial areas can hit 265V sometimes).

    Fact is it's been up 24x7 for months, and other than the odd random crash (this board has always done that but replacing is too expensive right now) it's otherwise been reasonably stable.
     
    #19

Share This Page