Unable to power disks on Supermicro 846A backplane

[fossxplorer]

What used to work fine, stopped working. I have 2 SSD in my SM 846 bays that are seen by the OS, but any new SSDs or HDDs don't seem to get powered up by the backplane. It all happened after me testing 3 HDDs and pulling the last one out.
The blue light is on when i insert an SSD or HDD, but nothing is being powered up.

What could cause such behavior? I see there are 3 reset jumpers for the chip on 846A from https://www.supermicro.com/manuals/other/BPN-SAS-846A.pdf. Is that something i should try?

EDIT#1: have taken down server and the BMC/IPMI by disconnecting the power completely. For some reason, it detects 3 (as USB, weird) devices which get the same device same (/dev/sdg) as the last device i connected earlier. It shows up in the boot log and then after some time disappears

 

[DavidWJohnston]

Hmm that's a strange one. It sounds like an unfortunate edge case with the i2c/BMC microcode was hit when that drive got unplugged. Did you try a second complete dark power cycle? I'll assume booting to a Live Linux key won't be different, as the boot log in the IPMI is messed-up.

I don't think the backplane chips have any on-board memory, so setting the reset jumper shouldn't do anything that a dark poweroff doesn't do - But I'm not positive about this. My source is reading the data sheet for the MG9072 chip - There is nothing in there about non-volatile memory.

Some data in the BMC's memory could have been corrupted. Maybe there is a way to reset that. A firmware update that covers the BMC would be a good idea as well. This really sounds like a firmware bug.

I wish I could help more, hopefully others will have some ideas. Good luck!

 

[fossxplorer]

First of all, thanks for taking time to answer!

I really missed some important details:
- i am using ASRock D1541D4U-2O8R board with Intel Xeon D-1541
- The onboard LSI3000's is connected to the backplance using SFF8643 to SFF8087
- I am using an ATX PSU in the 846 chassis and not Supermicro's.

So the BMC isn't controlling the backplace in this setup. The backplane only has the3 leftmost molex connected for powering some of the bays (yes, i figured out this also means no LEDs without powering the other 3s)

What's strange, the 2 SSDs that i haven't touched are still working fine. Remains to see what will happen when they are pulled out and inserted in again.

It was lot of dust in the non-occupied bays and i wonder if that could've contributed to the issue in any way?

I'll try to fully power down again tomorrow to see if that makes any difference.

 

[Sogndal94]

What does these lights(marked in yellow) do when you plug in the disks? if D53 or D54 lights up, i think you might have powerissues. Have you tried any other ports? Are you using the Raidcard as a HBA or Hardware raid, and if you are using it as HW raid, what does the Raid software say when you connect the new disks?

 

[fossxplorer]

The lights on the right side are on, 2 LEDs with green light constantly on. None of the LEDs from that group you marked in yellow are on.
Yes, i've tried multiple other others, e.g the bays on the far right side when you stand in front of the server.

I am using ASRock D1541D4U-2T8R with integrated LSI3008, i don't see it's menu during boot so it might be flashedfor IT mode without BIOS.

Now i did try to reset the MG9072 chip using the jumpers and saw red light on many bays, but not all which i find strange. It was during a boot with jumpers in reset position. But it did not help.

I should be able to have some downtime in the weekend for more troubleshooting.

IIRC, the disks should be powered once the backplane has power right? It doesn't need to be connected to an HBA in order for the disks to be powered?

What i find strange is that i see 3 USB/SCSI devices being found on boot by the OS which i suspect comes from the BMC/IPMI. After a while, they are being seen as "disconnected" and removed.
That happens when i get up the remote console redirection through BMC, so it's nothing to do with the LSI3008 controller.


Perhaps i should boot with a live USB ISO after all.

 

[DavidWJohnston]

After reading through your details again I'm wondering if this is something more simple like a bad connection. Many backplanes have multiple power connections - Try swapping them to see if the problem moves, and make sure they're well-seated, clean, and all power ports are populated.

Make sure the SAS connector(s) and backplane ports are clean and well-seated too.

You could also pull out all the drives, boot to a live Linux, then start plugging in drives in different slots to see if they come up. Try different slots/drives to try and eliminate possibilities.

Maybe one of the power rails in the PSU is toast or getting overloaded and causing dropouts. Some PSUs are a single-rail design, some multi.

 

[fossxplorer]

Will certainly do it this weekend when i can have downtime. Will report back here.
Thanks so far!