Xeon D-1541 Quanta storage servers on ebay

[fiftynine]

flash SAS Expander first. then BMC, then BIOS last.
as far as the one that's not booting, try connecting the rj45 to network and see if it picks up an IP address and then connect to it via the web interface to that IP address. default username is admin/admin
the event log may give you an indication of any issues.

It didn't sadly pick up an ip via ipmi. I'll try to troubleshoot more in the coming days if I find time. Thanks for the help!

Thank you @semidetached and @lnxpro. Really appreciate the help!

Well, that's a bummer with regards to the firmware no being available anymore, but thank you for the heads up. That sort of kills it for me . Hate to ask, but anyone happen to have local copies of the latest files and willing to generously share them?

I zipped up everything I found: two revs of the BIOS and BMC, the expander FW, and both IT and IR FW for the SAS controller. My expander identified itself as the "S2PH" model, so I used the "Broadcom" file.

Quanta_SD1Q.zip

drive.google.com

Thanks for the zip! Firmware seems to be available via qct.io site but lacking the bios files for some reason

Edit: is there a user manual available anywhere? I'd like to check a few things from there
Edit2: I see now, apparently the firmware for BMC and BIOS were pulled from qct.io, not for expander etc

 

[sleeper404]

After a bit more toiling with the system I managed to finally get an upgraded SAS expander, IT mode SAS3008 controller, BMC and I will attempt BIOS shortly. It took a mix of specific versions of storcli, the broadcom expander firmware package (lots appear to have shipped with the D51PH-1ULH and not the standard). The thing feels fairly un-brickable so long as you can get BMC access, but it was a lot of trial and error to get the right versions of sas3flash and storcli. IT mode flashing requires having your SAS ID so make sure you grab it from the output before you do the firmware erasure or else you'll have to pull it apart to find it on the side of the card not exposed even after you pull the mezz card out of the chassis...

Side Note: I was attempting to use some 10GTEK 3-meter DAC cables with this unit and an Arista switch and it was having very odd link flapping issues. I swapped out for 10G-SR optics and the issues went away.

 

[fiftynine]

After a bit more toiling with the system I managed to finally get an upgraded SAS expander, IT mode SAS3008 controller, BMC and I will attempt BIOS shortly. It took a mix of specific versions of storcli, the broadcom expander firmware package (lots appear to have shipped with the D51PH-1ULH and not the standard). The thing feels fairly un-brickable so long as you can get BMC access, but it was a lot of trial and error to get the right versions of sas3flash and storcli. IT mode flashing requires having your SAS ID so make sure you grab it from the output before you do the firmware erasure or else you'll have to pull it apart to find it on the side of the card not exposed even after you pull the mezz card out of the chassis...

Could you write up the versions of sas3flash/storcli and methods used? I've tried integrated EFI shell, no luck, I tried on linux as well, no controller found.

And as I have no bmc access, either, I feel like I'm running in circles.

 

[sleeper404]

So for storcli.efi, I had to have the right version and the controller had to be fully operational before you can flash the expander. The most annoying part is that when running storcli, if the controller wasn't showing up in the BIOS I wasn't able to find anything, period. If it was operational and I booted into the built in UEFI shell then i could run "storcli /c0 /e1 show" and that would show the details regarding the SAS expander, if that doesn't find anything you can try replacing the /e1 portion with /e0 or /e2 and see if it's just labelled something else. If you're unsure about the controller state you can run the sas3flash.efi utility with "sas3flash -list" to list all of the current details regarding the SAS3008 mezz controller (you'll want that SAS address for later if you plan to flash from IR to IT mode). I'll zip up the entire contents of my usb drive and throw a link up here once I get back to my workstation.

 

[fiftynine]

So for storcli.efi, I had to have the right version and the controller had to be fully operational before you can flash the expander. The most annoying part is that when running storcli, if the controller wasn't showing up in the BIOS I wasn't able to find anything, period. If it was operational and I booted into the built in UEFI shell then i could run "storcli /c0 /e1 show" and that would show the details regarding the SAS expander, if that doesn't find anything you can try replacing the /e1 portion with /e0 or /e2 and see if it's just labelled something else. If you're unsure about the controller state you can run the sas3flash.efi utility with "sas3flash -list" to list all of the current details regarding the SAS3008 mezz controller (you'll want that SAS address for later if you plan to flash from IR to IT mode). I'll zip up the entire contents of my usb drive and throw a link up here once I get back to my workstation.

I finally found the working version for the storcli. The one off the broadcom site didn't recognize the controller even though sas3flash listed it. I tried the broadcom storcli on embedded efi shell, on linux and on windows server. The one downloaded from here https://www.thomas-krenn.com/en/download.html?product=12190tps worked on a first try.

So, to flash the expander, I used aforementioned storcli and broadcom firmware linked earlier in this thread.

 

[sleeper404]

Glad you got it all playing nicely. If you're still having issues with the BMC it looks like there's a UEFI utility where you can reset it to factory defaults.

Here's a quick thread I found that might have the commands needed (haven't had to do it myself): POPUP

 

[fiftynine]

Glad you got it all playing nicely. If you're still having issues with the BMC it looks like there's a UEFI utility where you can reset it to factory defaults.

Here's a quick thread I found that might have the commands needed (haven't had to do it myself): POPUP

Thanks for the help.

I got one running fine out of two I ordered after some hardware and software tinkering.

Other one was somewhat beaten during transport and has physical damage. Waiting for a response from seller so I haven't really spent time with this one, apart from one evening. As the shipment box was open from one corner when it was thrown at my doorstep, I guess additional cushions were sadly lost during travels.

 

[avluis86]

Hey all; I've been eyeing these units as I have a Supermicro X10SDV-TLN4F running the same Xeon D-1541 rock solid;
FedEx as usual loves destroying things so out of the two purchased was able to get one usable frame (already talking to seller to have the other replaced).
I was still able to go thru all the needed steps and upgrade FW/BIOS to latest on both systems -- both did not want to pick up an IP for BMC and one did not want to output video;
I'll be posting a follow up with detailed instructions/steps to take but if you are in the same situation where there is no video out (and can't access BMC) you'll need to power up the system -- hold down the power button for 30 seconds, it should turn off a few seconds as you hold it down but hold it until 30 seconds have passed; power it back up and you should now have video output (note: make sure your RAM is seated properly, plug in a keyboard, preferably RGB; it should light up, turn on then off and back on again as the bootup sequence runs -- another way to tell bootup is running is the fans as they come online then slow down, spin up and then slow down once more).

Spoiler: For those interested, here are my notes:

-- Assuming you have no BMC access (not picking up IP) most likely BMC set to LOM-NIC and not dedicated NIC (or static IP); you can try plugging into each NIC until you see it grab IP via DHCP but not guaranteed -- you will need a KVM or VGA to HDMI adapter or similar (I used this as I still need to invest in a KVM solution) for troubleshooting.
If you get a video out, access BIOS by mashing <DEL> or <F2> -- otherwise follow instructions above to reset BIOS settings.

Once in BIOS navigate to Server Mgmt > BMC network configuration:

Make changes as needed (assign BMC LAN Port to Dedicated NIC, set static IP address and gateway respectively):

Strike <ESC> and Save Changes and Reset -- you should be able to access BMC at this point via the IP set above:

Default username/password is admin/admin (seeing an SSL warning is fine).
Hopefully you are now looking at something similar to below (screenshot taken after BMC update so yours may differ):

<wip> <more to come, gotta rest up for work>

 

[avluis86]

Quick break from the above WIP post:
While messing about with updating FRU info, I believe I've found the correct raw IPMI values to address fans on the SD1Q-1ULH;
Here are the respective raw values for the system fans (will be looking into the PSU fans as well):

raw 0x30 0x39 0x01 0x04 0x0<fan_num:1-5> 0x<fan_pwm_hex>

Because these systems have an empty fan slot where Fan_SYS0_0/Fan_SYS0_1 we can skip over that.
So if you want max PWM on fan 1 then: ipmitool raw 0x30 0x39 0x01 0x04 0x01 0x3f

Can someone else test this on theirs to confirm the above?

 

[fiftynine]

How the CMOS battery being dead manifests?

I have a "bmc failure" on boot. I can ssh into the bmc, it has been flashed (twice) to the latest firmware. Every boot takes 5-10mins so troubleshooting is a wee bit time consuming. After a lot of fiddling I got bmc access after a flash (earned a beer for this achievement), but the error still happens on every boot.

Glad you got it all playing nicely. If you're still having issues with the BMC it looks like there's a UEFI utility where you can reset it to factory defaults.

Here's a quick thread I found that might have the commands needed (haven't had to do it myself): POPUP

These commands don't work on my machines. There were some commands earlier in this thread which allowed to reset the bmc though

 

[Chielos]

Quick break from the above WIP post:
While messing about with updating FRU info, I believe I've found the correct raw IPMI values to address fans on the SD1Q-1ULH;
Here are the respective raw values for the system fans (will be looking into the PSU fans as well):

raw 0x30 0x39 0x01 0x04 0x0<fan_num:1-5> 0x<fan_pwm_hex>

Because these systems have an empty fan slot where Fan_SYS0_0/Fan_SYS0_1 we can skip over that.
So if you want max PWM on fan 1 then: ipmitool raw 0x30 0x39 0x01 0x04 0x01 0x3f

Can someone else test this on theirs to confirm the above?

This works! Quite brilliant!

edit:
found out that 0x06 with value 0x3f turns them all on.

 

[avluis86]

This works! Quite brilliant!

edit:
found out that 0x06 with value 0x3f turns them all on.

Lovely -- good to know; now all that is left is PSU fan control (low priority for me).
I'll be working on a script similar (a fork) to nmaggioni/r710-fan-controller to wrap this all nicely as time permits.
I currently make use of this on my Dell r7910 with some slight tweaks and working a treat; certain it can be done with this platform as well.

Update/sidenote:
raw 0x30 0x39 0x02 0x04 0x06 0x00 returns fans to auto (last two values [0x06 0x00] do not matter but they need to be present for the command to succeed)

Another brief update:
These fans can ramp up much higher than initially thought; try running ipmitool raw 0x30 0x39 0x01 0x04 0x06 0x63 -- this is their max value, enjoy.

 

[fiftynine]

How the CMOS battery being dead manifests?

I have a "bmc failure" on boot. I can ssh into the bmc, it has been flashed (twice) to the latest firmware. Every boot takes 5-10mins so troubleshooting is a wee bit time consuming. After a lot of fiddling I got bmc access after a flash (earned a beer for this achievement), but the error still happens on every boot.

I swapped the CMOS battery, didn't resolve the bmc self test failure. Maybe I'll try to flash an older version of the bmc firmware next (everything is up to date currently).

Bmc settings page in bios is inaccessible so I can't see or change anything bmc related there.

 

[oldfett]

I think a few folks bought these primarily for the compute rather than the storage. If that's the case anyone willing to sell me the internal 3.5" drive trays? I would like to pick up 1-2 more of these for backup systems but it sounds like they are now coming without trays/caddys. Otherwise anyone know a source for those trays? It looks like they are part number 1HY9ZZZ049E but I haven't had any luck finding them.

 

[avluis86]

I think a few folks bought these primarily for the compute rather than the storage. If that's the case anyone willing to sell me the internal 3.5" drive trays? I would like to pick up 1-2 more of these for backup systems but it sounds like they are now coming without trays/caddys. Otherwise anyone know a source for those trays? It looks like they are part number 1HY9ZZZ049E but I haven't had any luck finding them.

I'm purchasing these myself for ceph/storage as well so if you end up sourcing the trays please keep us informed.
Speaking of informed, my contact at QCT provided me the latest BIOS for these unit(s): 2020WW21.1_S1Q_3A15.zip

 

[pypypy]

anyone able to get freenas/truenas running on one of these?

 

[fiftynine]

anyone able to get freenas/truenas running on one of these?

I have one running truenas. I run it off a cheap intel optane 16gb ssd.

 

[megalo]

I'm purchasing these myself for ceph/storage as well so if you end up sourcing the trays please keep us informed.
Speaking of informed, my contact at QCT provided me the latest BIOS for these unit(s): 2020WW21.1_S1Q_3A15.zip

Has anyone been able to source 3.5" trays/caddies?

Edit: Alternatively, are trays totally necessary to work properly? 3d printing an option?

 

[cookiesowns]

I'm purchasing these myself for ceph/storage as well so if you end up sourcing the trays please keep us informed.
Speaking of informed, my contact at QCT provided me the latest BIOS for these unit(s): 2020WW21.1_S1Q_3A15.zip

can you PM contact of your quanta rep. Looking for firmware updates for some of their other systems.

 

[Jaket]

Has anyone been able to source 3.5" trays/caddies?

Edit: Alternatively, are trays totally necessary to work properly? 3d printing an option?

I would say that they are necessary to work properly. Otherwise you are just putting drives in without being attached to anything and with spinning HDD's that creates a lot of movement even more with 12 drives spinning around.