Xeon D-1541 Quanta storage servers on ebay

[YuanyangX]

I'm seeing a weird issue with my server. That is: right after the server is shutdown, no matter is a graceful shutdown from OS or power button shutdown, the two PSU fans start screaming.
I tried to leave them as-is and after hours those PSU fans are still up and running happily, at around 9K+ RPM. I also tried different versions of BMC and BIOS, all got same result. Maybe I should ask seller about it.
Does anyone have experience of handling this funny problem? Thanks in advance.

 

[sparx]

PSU fans are still up and running

If noone has a good answer Ill put a guess in here. There might be a PLD component on the board to handle misc-signals. Power handover between PSUs, easy upgrades of BMC and a variety of signals you didnt find any good way to control from anywhere else. And since this is a very specialized platform maybe noone has reported errors and making sure that the code is updated in the PLD. Or perhaps they dont even have remote upgrade of this component. ( Or even there might not be one. Hehe ).

 

[Yarik Dot]

Could anyone please make a tour of BMC? Thanks in advance.

 

[scobar]

Could anyone please make a tour of BMC? Thanks in advance.

What would you like to see? I can slap together a gif and go through each menu, etc. The first 30 seconds or so I whipped through it seemed rather OK. Sensors, console redirection, power control.

Currently all the sas slots on back plane are filled with misc 2.5" sata drives. I put then into various storage spaces and now going to put them through their paces.

At console, I note the mouse skips a lot but via an RDP session it seems OK. *shrug*

 

[gigatexal]

1U delta fans = fail for me

 

[Yarik Dot]

What would you like to see? I can slap together a gif and go through each menu, etc. The first 30 seconds or so I whipped through it seemed rather OK. Sensors, console redirection, power control.

Currently all the sas slots on back plane are filled with misc 2.5" sata drives. I put then into various storage spaces and now going to put them through their paces.

At console, I note the mouse skips a lot but via an RDP session it seems OK. *shrug*

Just a few screens - homepage, sensors, power page, console page is enought I think.

Thanks!

 

[tjk]

I finally got sometime to play with it. After running storcli.efi /c0 /eXX download src=S1Q_LSI_FW_v1.0.8.fw, I got a success reported.
I think the tricky part would be:
1. Run upgrade without any disk connected.
2. Run upgrade in EFI shell instead of any OS. I think it will limit device mounting or something.

I only put single 1TB SATA (ST1000DM003) to test the disk cage from cold start. And the disk shows up correctly. I'm considering to move the SAS3108 from the other server to this one to get better RAID performance.
Attached a screenshot to show the FW version numbers.

Wish your upgrade will be same smooth as mine.

Not sure what magic you were able to use, but I'm getting the same errors as trying via a loaded OS. I disconnected ALL HDD and SSD's, and getting the same error, tried using Enc 12 and Enc 252, which are the one two that are listed.

Any other ideas? Did you screen cap your output? If so, can you share the exact steps?

 

[YuanyangX]

Not sure what magic you were able to use, but I'm getting the same errors as trying via a loaded OS. I disconnected ALL HDD and SSD's, and getting the same error, tried using Enc 12 and Enc 252, which are the one two that are listed.

Any other ideas? Did you screen cap your output? If so, can you share the exact steps?

That’s really weird... How about let me find a time and capture a short video for you. I need one day or two since weekend is over.
BTW, may I ask do your server PSU fans still running loudly after shutdown?

 

[scobar]

I am going to reach out to the seller. One of the spots for PSU is bad, if I move the 'bad' PSU, it works. CMOS battery is also toast.

 

[YuanyangX]

I am going to reach out to the seller. One of the spots for PSU is bad, if I move the 'bad' PSU, it works. CMOS battery is also toast.

My CMOS battery is also dead. I cut the shrink wrap open then used few electrical tape to wrap a new battery with pins.
I should talk to seller about the PSU instead of digging around myself.

 

[scobar]

1U delta fans = fail for me

That's why all the loud stuff is outside in the garage.

My CMOS battery is also dead. I cut the shrink wrap open then used few electrical tape to wrap a new battery with pins.
I should talk to seller about the PSU instead of digging around myself.

Did you also loose a PSU? Was it the unit located on the outside? My fallback was to solder the leads onto a spare battery, really I just want a working box. If this didn't have the PSU issue, I'd just solder the leads to a new batt, heat shrink it and keep truckin'.

The seller indicated they would get another out today, I had asked for them to provide a return label for this unit.

I did pop the heat sink off, that paste is pretty crusty and may want a vac to ensure nothing gets onto the MOBO.

 

[scobar]

Just a few screens - homepage, sensors, power page, console page is enought I think.

Thanks!

Imgur

 

[tjk]

So, I ordered 8 of these...4 arrived damaged and looked like they just shipped damaged units since the shipping boxes were fine. They shipped replacements, and 2 of the replacements were also bad...so they shipped 2 more.

The CMOS bat is dead in 7 of them so far, not sure if this is something I can just replace or not/find the part.

The expander won't upgrade on any of them, so not sure how the earlier poster was able to get his updated. I am using the 3108 Mezz card, which I swapped out for the 3008 card.

 

[YuanyangX]

Did you also loose a PSU? Was it the unit located on the outside?

The PSUs can work. I had swapped them in different combinations, single on slot 1, single on slot 2, dual, etc. All combinations can boot together with 10x3.5 SATA disks without any probloem. It's just weird that PSU(s) keep on after shutdown...
Waiting for seller's response.

The expander won't upgrade on any of them, so not sure how the earlier poster was able to get his updated.

I used "007.1316.0000.0000" storcli (download from Broadcom Inc. | Connecting Everything) and I only got one enclosure listed when calling storcli /c0 /eall show. I'm capturing some screens while re-running the firmware download for expander. But I don't want to downgrade the FW first then upgrade FW as it feels risky. I will update the screenshots later in this thread.

Update:
I've uploaded a short video captured by console. Hope you don't mind watching it on YouTube.

 

[Yarik Dot]

So, I ordered 8 of these...4 arrived damaged and looked like they just shipped damaged units since the shipping boxes were fine. They shipped replacements, and 2 of the replacements were also bad...so they shipped 2 more.

Pictures of damage, please.

We are considering taking full rack (45 pieces) of these.

 

[Yarik Dot]

Is there an option to boot via UEFI IPv6? It might not be enabled by default, but there is a switch for that in bios.

^^ I can see UEFI IPv4 in the video above.

 

[tjk]

The PSUs can work. I had swapped them in different combinations, single on slot 1, single on slot 2, dual, etc. All combinations can boot together with 10x3.5 SATA disks without any probloem. It's just weird that PSU(s) keep on after shutdown...
Waiting for seller's response.


I used "007.1316.0000.0000" storcli (download from Broadcom Inc. | Connecting Everything) and I only got one enclosure listed when calling storcli /c0 /eall show. I'm capturing some screens while re-running the firmware download for expander. But I don't want to downgrade the FW first then upgrade FW as it feels risky. I will update the screenshots later in this thread.

Update:
I've uploaded a short video captured by console. Hope you don't mind watching it on YouTube.

Thanks for the video!! Still not working for me, see attached pics. I swapped the 3008 for the 3108 card, not sure if that has anything to do with it or not. I tried with and without drives in the system, same error. The debug log is also useless, just says error=45.

This is across 8 systems.

 

[tekmunki]

Thanks for the video!! Still not working for me, see attached pics. I swapped the 3008 for the 3108 card, not sure if that has anything to do with it or not. I tried with and without drives in the system, same error. The debug log is also useless, just says error=45.

This is across 8 systems.

Weird your screens are showing the d51ph-1ulh expander that's a different unit from what we have. On the qct.io site it lists the expander firmware as a different file "12G-EXPANDER_FW-Broadcom_1.0.8.zip" vs. "12G-EXPANDER_1.08.zip" for the sd1q-1ulh. Maybe give the broadcom fw a try and see if you get the same error.

I got 2 more SD1Q-1ULH's in and one doesn't see the expander at all so the seller is sending a replacement. The other unit seem fine with no issues and is serving 60TB fine. The status indicators on trays light up but neither the OS or IPMI tool can communicate with the expander on the one unit. Just shows a N/A for firmware version and expander temp sensors. I've checked all the cables and can't find anything physically wrong with it. I can only imagine the type things you'd see with a full 48U rack of these could probably start your own wiki on how to fix various issues.

 

[YuanyangX]

Thanks for the video!! Still not working for me, see attached pics. I swapped the 3008 for the 3108 card, not sure if that has anything to do with it or not. I tried with and without drives in the system, same error. The debug log is also useless, just says error=45.

This is across 8 systems.

Mine doesn't have SGPIO device listed there so all the HDD LEDs are always off. Maybe my server is a broken one which by chance can take new firmware....

This morning I just found out the two SFP ports from motherboard are not working. Tried 3 different modules (2 different models of Intel E10GSFPSR and a 1Gb copper). None of them worked on those ports. But all of them work well on OCP mezz card. So I believe I have to get a replacement from them...

 

[YuanyangX]

Weird your screens are showing the d51ph-1ulh expander that's a different unit from what we have.

Maybe the warehouse was just swapping broken components which only physically fit to the case and can power on...