A cost-effective Intel W680 ECC server: repurposing an HP Z2 G9 motherboard

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

cromo

Member
Jun 6, 2019
98
29
18
I asked a while ago whether anyone tried to repurpose an HP Z2 G9 Tower/Lenovo ThinkStation P3 W680 Tower/Dell Precision 3660 W680-based workstation motherboard and got no response, so I decided to go ahead and do it myself.

I managed to buy a second-hand HP Z2 G9 W680 motherboard for less than €100 and within the last 3 months had gained enough experience with it to share here with the community. Considering all other W680 motherboards cost upwards USD 500 at this point, this turns out to be a fairly attractive solution to get a modern Intel system with low idle power consumption, PCIe Gen 5, and, most importantly, ECC memory support. Interestingly, I saw some recent eBay listings that sold those motherboards for as little as 20-40 USD!

The HP Z2 G9 system spec is here, motherboard details on page 17: https://h20195.www2.hp.com/v2/getpdf.aspx/c08109687.pdf
The spec of the motherboard itself can be looked up in the Service Manual here, page 51: https://kaas.hpcloud.hp.com/pdf-public/pdf_5451169_en-US-1.pdf

I originally had it running with 12600k, I since upgraded to 13600k (more on that later).

There are a few challenges running it, compared to your run-of-the-mill products:

1. PSU
HP, Dell, and Lenovo all use proprietary PSUs, which at least in case HP are actually conformant to ATX12VO. That means the PSUs themselves only deliver 12V to the board, which then handles the 5V supply by itself. This also means that using off-the-self PSU is very limited, especially because the board expects +12VSB (stand-by voltage), as opposed to the standard 5V. There are workarounds available (google it), but I resorted to using an HP power supply.​
For the record, while the HP Z2 G9 comes can be configured with 3 different PSUs, you can use PSUs from other systems. Notably, the PSU must have a 7-pin PWRCMD connector, as opposed to a 6-pin That is, however, not enough, as not all PSUs have enough cables connected to it, and my conclusion at this point is that any PSU with *black cabling* and a 7-pin PWRCMD connector will work with this system. I am myself using it with an HP G5 450W PSU, which has Gold certification (Platinum ones are also available).​
Also worth noting is that there isn't anything proprietary about those PSUs outside of the connectors: all of the cabling is standard ATX. In fact, before they switched to using all-black cabling, all of the cable colors used matched the ATX standard.​
2. FANs
Another proprietary solution is HP fan connectors. They are compatible with regular PWM fans, but the HP CPU fan comes with 5 pins, instead of regular 4. Pins 1-4 are the same as in the standard, pin number 5 is crucial to get the system to NOT report fan incompatibility upon POST.​
Their fans come in at 65W and 125W TDP versions. So far, what I found, is that to emulate 125W, one needs to connect pins 3 and 5. Shorting pins 3, 4, and 5 will allow the system to not report when the fan is missing entirely. More on that later.​
The system will also report a missing chassis fan, CHFAN2 (P9 in manual). This one is a regular, 4-pin one. The solution here is to short pins 3 and 4.​
The problem, unfortunately, is that HP fan implementation is not exposed to the system, so lm-sensors is not able to control them. That wouldn't be that bad if not for the fact that they tie the fan speed to the CPU usage, not the temperature. And this in itself wouldn't be a problem if they didn't tie them to a *single core* usage, which results in fans blasting as soon as the system sees even modest utilization, around 3-5%.​
With that in mind, you need to resort to an external fan controller to take the control back. You'll also need to short pins 3, 4, and 5, as explained above, to avoid the POST complaints. Some recommended options are Corsair and NZXT, which luckily come with Linux kernel support, together with liquidctl to control them. More here: liquidctl
3. Motherboard and its non-standard size
Screenshot 2024-04-11 at 9.16.20 PM Medium.png
All of these HP/DELL/Lenovo motherboards come in non-standard sizes. They are Micro-ATX by lengths, but exceptionally wide. This has both pros and cons:​
  1. You can't use them in regular ATX cases, most likely. I don't have a problem with it, since I have it installed in a custom IKEA shelving
  2. Excellent IOMMU grouping. Pretty much every single device has its own group. No ACS overrides needed. No issues in passing GPU, Mellanox SRIOV VFs, or SATA controller to the VMs.
  3. 3 x NVMe PCIe4 x4 slots + 1 x m2 PCIe3 x1
  4. HP Flex IOv2 connector. That means you can get an extra 10Gbit NIC or install a relatively inexpensive HP 3UU05AA Thunderbolt 3 extension card (which takes up PCI slot 4 as well).
  5. I219-LM Gigabit Ethernet with vPro support.
  6. Extra USB connectors on the other end of the board
  7. Only 4 PCI slots. Slot 1 is PCIe5 x16. The rest are PCIe3 x1, x4, x4. A 2.5-width GPU card will leave you with a single PCI slot 4 free — unless you resort to some creative PCI ribbon risers.
  8. 4 SATA ports. You need a "P160 HP cable" to power the disks, as the PSU being ATX12VO does not come with SATA power cabling.
  9. There's an onboard USB header, used to connect the Media Card reader, but it is non-standard, so you can't use it to connect the Corsair/NZXT fan controller.
4. Issues running it

13th gen "PCA not fully compatible" error

After updating to the 13th gen, I am seeing a "PCA not fully compatible" error. This is actually discussed here on the forums, too, affecting the HP Z2 G9 Mini system. I have no clue what's causing this: the fan is rather unlikely since you can disconnect it altogether and the SKU error still shows, followed by the "missing fan" error. The PSU is also unlikely, since AFIK there's no way for the motherboard to know the model of the PSU connected. There is another revision of the motherboard, which is said was released to fix some 13/14th gen compatibility issues, but HP themselves have not confirmed that, and all of the posts on their forums that ask for help with the very same issue were left with no response from HP.

Unfortunately, the system will not post past this error unless you acknowledge it with a keyboard press. I did not notice any other compatibility issues with the 13th gen on top of the 12th gen. Goes without saying the firmware is up to date.

Power consumption

With no external devices connected and the system idling, 64GB (2 x 32 GB) ECC RAM, and 2 x Lexar NM 710 1 TB SSDs, the power consumption measured off the wall is 4.5-6W, which is astonishing (with only 1W difference between 12600k and 13600k). It could potentially get even a bit better with a Platinum PSU. AMD clearly cannot compete here at all. For the record, the system would reach C10 states with 12600k and C8 states with 13600k. I am not sure if the latter is expected — rather not.

Unfortunately, some serious problems start as soon as I connect PCI devices. Specifically, with the AMD 6600 XT GPU connected to slot 1, the system will no longer enter deeper C-states, stopping at C2. I believe this is because the PCIe5 slot 1 is connected directly to the CPU, which apparently limits the C states to 2, as explained here. So while the GPU is very efficient, idling at 4 W only, the system's consumption spikes up to 40-55W. Combined with my Mellanox 4 LX and some USB devices + 4 x 2.5" idling HDDs, the total idle consumption is 55W-70W. This is, unfortunately, on par with similarly configured, idling AM5, with regular, off-the-shelf components.

It's worth noting that all of the devices have ASPM enabled, as reported by the lspci. I am left without ideas here, I asked on HP forums, but I doubt I will be met with any help: Z2 G9: unable to reach C-state above 2 with GPU installed in PCIe5 slot

Noteworthy, HP specs suggest a similarly specced system, i9 12900 with NVIDIA GPU, should idle at 22-24W in Windows. This is considering NVIDIA GPUs are actually worse than AMDs in idle. I have not tested the system with Windows yet, it would be interesting to see whether I can confirm those numbers — especially in light of what is reported on C2 levels enforced by occupying CPU-bound PCI slots.

RAM issues with Intel x710 NIC

Another weird issue is happening with an Intel x710 installed in PCI slot 4 in place of the Mellanox. I figured I would want to reduce the overall power consumption by using the Intel NIC, which is reported to be more power efficient. Unfortunately, with Intel NIC installed, the system POSTs with half of the memory, i.e. 32GB in my case. Super odd issue, I would need to have someone try to replicate it, otherwise it may sound like the motherboard is somehow broken. Although it's interesting that I am not getting such an issue with the Mellanox installed.

RAM speed limited to 4400/4000 MT/s

As explained below by [B][SIZE=5]zir_blazer[/SIZE][/B], this works as intended on this platform.

There's some off limitation of memory speed imposed by this system. With two modules installed, my 48000 MT/s memory is effectively limited to 4400 MT/s, which I have confirmed using lshw

Screenshot 2024-04-11 at 9.11.02 PM.png

Turbo speeds

I am convinced the motherboard limits the socket to 125W and, as a result, the CPU will hardly ever get close to the Turbo speeds, even if the thermals permit it. This is with the High-Performance Mode enabled in BIOS.

Undervolting/Overvolting

This being a proprietary system means you cannot underplot the CPU, which has the potential of reducing the consumption significantly with Intel CPUs.

Summary

All things considered, whether this is a viable platform depends on whether the issues above are fixable. Having to manually press "Enter" to boot with a 13th gen CPU can be worked around by resorting to a 12th gen CPU. However, if you want to use it with extra PCI devices, you need to consider the idle power consumption issues, which then make this system lose its advantages over the AMD AM5 platform. Other than that, the system has been very stable, not having crashed once in the lasts 3 months.

All things considered, if you can get ahold of an off-the-shelf W680 motherboard at a reasonable price, I wouldn’t bother getting your hands dirty with this approach.
 
Last edited:

zir_blazer

Active Member
Dec 5, 2016
361
131
43
RAM speed limited to 4400/4000 MT/s

There's some off limitation of memory speed imposed by this system. With two modules installed, my 48000 MT/s memory is effectively limited to 4400 MT/s, which I have confirmed using lshw

View attachment 36015
Working as intended. Alder Lake supports DDR5 4800 MHz and Raptor Lake 5600 ONLY IF THE MOTHERBOARD HAS ONE SLOT PER CHANNEL. Motherboard has 4 Slots? 4400 max for best case, which is 1 DPC installed.


Turbo speeds

I am convinced the motherboard limits the socket to 125W and, as a result, the CPU will hardly ever get close the Turbo speeds, even if the thermals permit it. This is with the High Performance Mode enabled in BIOS.
You could check with ThrottleStop or Intel XTU in Windows what the PL1/PL2 values are.
 

cromo

Member
Jun 6, 2019
98
29
18
Working as intended. Alder Lake supports DDR5 4800 MHz and Raptor Lake 5600 ONLY IF THE MOTHERBOARD HAS ONE SLOT PER CHANNEL. Motherboard has 4 Slots? 4400 max for best case, which is 1 DPC installed.
Thanks, at least this is sorted.

You could check with ThrottleStop or Intel XTU in Windows what the PL1/PL2 values are.
turbostat lists that too:

Code:
cpu0: MSR_RAPL_POWER_UNIT: 0x000a0e03 (0.125000 Watts, 0.000061 Joules, 0.000977 sec.)
cpu0: MSR_PKG_POWER_INFO: 0x000003e8 (125 W TDP, RAPL 0 - 0 W, 0.000000 sec.)
cpu0: MSR_PKG_POWER_LIMIT: 0x4285a800e383e8 (UNlocked)
cpu0: PKG Limit #1: ENabled (125.000 Watts, 224.000000 sec, clamp ENabled)
cpu0: PKG Limit #2: ENabled (181.000 Watts, 0.002441* sec, clamp DISabled)
So I guess it is not limited, after all. That is assuming these values take motherboard limitations into consideration?
 
Last edited:

RolloZ170

Well-Known Member
Apr 24, 2016
7,125
2,219
113
Motherboard has 4 Slots? 4400 max for best case, which is 1 DPC installed.
search and find Enforce POR in the BIOS, set to disabled and the RAM keeps 4400.
after that run memtest to be sure the cpu/board/ram combo can do this without errors.
 

cromo

Member
Jun 6, 2019
98
29
18
search and find Enforce POR in the BIOS, set to disabled and the RAM keeps 4400.
after that run memtest to be sure the cpu/board/ram combo can do this without errors.
You won't find such an option in proprietary systems like this one, which value stability over functionality/performance.
 

NickKX

Member
Oct 26, 2023
48
9
8
Working as intended. Alder Lake supports DDR5 4800 MHz and Raptor Lake 5600 ONLY IF THE MOTHERBOARD HAS ONE SLOT PER CHANNEL. Motherboard has 4 Slots? 4400 max for best case, which is 1 DPC installed.

You could check with ThrottleStop or Intel XTU in Windows what the PL1/PL2 values are.
I’ve got a similar problem, with the z2 mini g9 (same chipset), but that system only has two SO DIMM ram slots

Using this ram: Micron 32GB DDR5-5600 ECC SODIMM 2Rx8 CL46 | MTC20C2085S1TC56BR | Crucial.com..

With 12500, memory ran at 4800 (expected)

Upgraded to 14th Gen 14700k, and now runs at 5200 instead of 5600

Weird…? No option to enable XMP either in BIOS, naturally, as I don’t think HP uses XMP in any of their computers, even the gaming ones

Also, Intel XTU in Windows 11 reports “this platform doesn’t support overclocking” and voltage control and basically every other setting is locked out

What is even the point of HP selling these with the K sku unlocked CPUs?

Do they actually behave any differently with the non K variants? (Ie 14700 vs 14700k in same system)

Or is it just HP locks down all the values in BIOS on the K chips?

I was really looking forward to undervolting. Huge sad face.

I will also note I did not get any kind of POST error upon upgrading my CPU from 12th to 14th gen, however I recently had my motherboard replaced (the original board that it came with it bricked itself!), so idk if that has anything to do with it. The onsite tech took the old bricked board and sent it back to HP to be recycled, but I took at look at the revisions and components, and they were identical. Both revision -603, which according to HP’s partsurfer, is the latest for the Z2 Mini G9.

I also have both the 65w fan and the 125w fan, and it doesn’t care which is installed, either. No error message even with the 65w fan. I also upgraded the heatsink myself.

@cromo even on latest 2024 bios version you’re still getting the PCA error?

Re: the memory issues and NIC issues, try disabling the option roms (assuming you don’t need iSCSI boot or similar). That usually works to fix those kinds of problems on the Z series. HP has insanely buggy bios especially when it comes to “third party” PCIe cards.
 
Last edited:
  • Like
Reactions: cromo

zir_blazer

Active Member
Dec 5, 2016
361
131
43
I’ve got a similar problem, with the z2 mini g9 (same chipset), but that system only has two SO DIMM ram slots

Using this ram: Micron 32GB DDR5-5600 ECC SODIMM 2Rx8 CL46 | MTC20C2085S1TC56BR | Crucial.com..

With 12500, memory ran at 4800 (expected)

Upgraded to 14th Gen 14700k, and now runs at 5200 instead of 5600
Raptor Lake supports 1 SPC with a Single Rank DIMM @ 5600 MHz, or Dual Rank DIMM @ 5200 MHz (Yours)
However, Raptor Lake Refresh supports 1 SPC with BOTH a Single Rank DIMM or a Dual Rank DIMM @ 5600 MHz. And a 14700K is Raptor Lake Refresh, so...
 
  • Like
Reactions: NickKX

Bear

New Member
Jun 19, 2024
3
1
3
A few things to add:

13th gen "PCA not fully compatible" error

This is an error with some Z2 motherboards, and it can be Mini, SFF, or Tower. There are actually new motherboard revs that support 13th/14th gen CPUs. Even with the newer BIOSes, these boards will always get this error. The solution is to contact HP directly and inform them of the error after a CPU upgrade. As of time of writing (6/24) all of these Z2 G9s are in-warranty. HP does not care if you've upgraded the CPU, you can call them and get a tech sent on-site to do a motherboard replacement. Newer boards officially support 13th/14th Gen CPUs.

RAM issues with Intel x710 NIC

I had this problem with a generic Intel -and- a Dell-branded X550-T2. After purchasing and installing an HP/HPE-branded X550-T2, I had zero issues with the RAM being halved. I imagine this NIC may work fine - 727055-B21 HPE ETHERNET 10GB 2-PORT 562SFP+ ADAPTER 790316-001 784304-001 | eBay

Turbo speeds

Tower
As far as the cooler is concerned, yeah, it kinda sucks. However, you can buy a Noctua U9S, use the BLUE spacers, and purchase some metric screws (I believe 1", M4, but take your stock HSF into Home Despot or Lowes to be 100% sure) and you can, with a bit of force, screw those into the Noctua Intel frame, and successfully mount it. Afterwards, you can follow the fan guide above, or as I did, since I had a 125w cooler, I just moved the factory 80mm fan over rather than mess with wiring.

SFF/Mini
I haven't found a better cooler than HP's own 125w. Would recommend that.

All
Linux:
With powercap-utils under Linux/Proxmox, you can change the max CPU wattage - For example, with an i7-14700 (non-t, non-k/ks) I use this in a script run at reboot after installing the powercap-utils package to up the long-term power limit from 65w to 95w, which the 65w CPU cooler has no problems with keeping cool under stress tests:

powercap-set intel-rapl -z 0 -c 0 -l 95000000

You could do the same with a 125w limited CPU under Linux to run it at ~140w (max I've seen the stock 125w cooler take) or 180-185w with a Noctua cooler. (See ProTip below)

Windows: I've also had luck under Windows using ThrottleStop to change the maximum wattage from 125w to 180w, and the Noctua cooler, with OCCT in Windows, I was seeing nominal temps of 82C after a full 1 hour extreme load run.

ProTip: You can do a Thermaltake LGA 1700 frame on these for a few C better cooling. Just make sure you're careful when removing the screws and do it on a level surface, because nothing is holding the bottom mount, and if you move it, you'll have to remove the logic board. If you're careful, you can get the new screws in (diagonal is best) without any issues. EXCEPT ON THE MINI where the retention bracket falls a bit too far for the stock screws to reach. You can get a slightly longer screw, pull it up, then screw in the others...just something to be aware of. I screw these in until there's resistance, then stop.

Do these make good W680 Servers/Workstations?

In a word, depends. It depends on your needs. The towers are not going to be the best choice for running a RTX 4080/90 as cooling, even with the front fans and upgraded exhaust fan isn't superb. But otherwise, I've picked Z2 G9 SFFs without CPU and RAM but complete otherwise for $40 on Fleabay with a "dead motherboard" but as most all of these have 3/3/3 warranties, that unit was 2 years in warranty, and HP shipped me a mobo no questions asked. I went to Partsurfer, typed in the serial, and was able to input all of the BID, Serial, etc. data myself, and lock down the motherboard. $40 shipped for a W680 system? Yes please. But that's a fluke! - Maybe, but I also picked up 2 Z2 G9 Minis that were barebones (no PSU, CPU cooler, CPU, RAM, Fan) for $90 shipped. One had a faulty motherboard, called HP, they sent a tech and gave me a brand new motherboard, and programmed it for me. I'm running 3 Z2 G9 SFFs with 128GB ECC, booting to a Micron 7450 Pro 800GB M.2, with Micron 7400 Pro 7.68TB U.2s for ZFS pool with a PCIe->U.3 adapter and HP Intel X550-t 2x 10GBE as a ProxMox Cluster. It's extremely cost effective, I have ~2 years of warranty left as of time of writing, and the PSUs are cheap enough on Fleabay.

Anything else? Other weirdness?

I've found a weird compatibility issue with Micron 7450 Pro 7.68TB U.3 drives, where the drive will FAIL after being formatted with ZFS, and a write is attempted subsequently. NO CLUE why, or if it's limited to this system. Warranty covered it, but I moved to 7400 Pro U.3s with no issues.

Hope this helps someone. :)
 

NickKX

Member
Oct 26, 2023
48
9
8
Interesting info about the motherboards. Any idea specifically what it is about those old boards that won't support the new CPUs? I would love to know, just for my own knowledge.
 

Bear

New Member
Jun 19, 2024
3
1
3
Any idea specifically what it is about those old boards that won't support the new CPUs? I would love to know, just for my own knowledge.
Oddly enough, even HP doesn't know. The TWR model does have a motherboard in Partsurfer that says it's no longer sold, but there are no specifics on what changed. All both techs I spoke to said, was a motherboard replacement was necessary, and it was authorized. The SFF is the only one that is user-replaceable, the other 2 require a tech to go onsite free of charge. I just gave them the POST error message

I had both an SFF and TWR that exhibited this behavior when I moved to 13th and 14th gen CPUs, and HP mailed me a replacement for the former, with a FedEx sticker to return the original mobo, and had an onsite tech replace the mobo in the latter. Both no longer have this error at POST.
 
  • Like
Reactions: NickKX

NickKX

Member
Oct 26, 2023
48
9
8
Oddly enough, even HP doesn't know. The TWR model does have a motherboard in Partsurfer that says it's no longer sold, but there are no specifics on what changed. All both techs I spoke to said, was a motherboard replacement was necessary, and it was authorized. The SFF is the only one that is user-replaceable, the other 2 require a tech to go onsite free of charge. I just gave them the POST error message

I had both an SFF and TWR that exhibited this behavior when I moved to 13th and 14th gen CPUs, and HP mailed me a replacement for the former, with a FedEx sticker to return the original mobo, and had an onsite tech replace the mobo in the latter. Both no longer have this error at POST.
I had a different issue with mine, it bricked itself basically — they replaced the board because it was in warranty. It took a long time to get the new board though, I think it was around two months. And they had to have an onsite tech do it, they wouldn’t let me do it myself (z2 mini g9). But yeah the new board just works with the newer CPUs. No problems.
 

Bear

New Member
Jun 19, 2024
3
1
3
It took a long time to get the new board though, I think it was around two months. And they had to have an onsite tech do it, they wouldn’t let me do it myself (z2 mini g9).
It took them under 3 days for me, as the board was in stock. Point is, anyone who has this error is likely still under warranty and can get a new board that will remedy it. :)
 

mack

New Member
Jul 17, 2022
20
6
3
@cromo Have there been any new BIOS fixes since you wrote? Has anything changed with ASPM support for the graphics card?
 

cromo

Member
Jun 6, 2019
98
29
18
@cromo Have there been any new BIOS fixes since you wrote? Has anything changed with ASPM support for the graphics card?
I moved on to another platform since, I won’t be able to answer that for you, but hopefully others can?
 

mack

New Member
Jul 17, 2022
20
6
3
@cromo will you write what you chose? I am looking for a motherboard that will have the ability to insert 3 nvme drives, will not consume more than about 6-8W in idle and will support higher states than C2 with a graphics card.
 

cromo

Member
Jun 6, 2019
98
29
18
@cromo will you write what you chose? I am looking for a motherboard that will have the ability to insert 3 nvme drives, will not consume more than about 6-8W in idle and will support higher states than C2 with a graphics card.
Sure, but you won't like it ;) I switched to AMD, which consumes 60W with all my peripherals in idle. Not great, but does the job for now.