16 Core 32 Thread HP z820 dual socket workstation build/performance upgrade

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
I recently picked up two HP z820 workstations via eBay for parts only ($125 +150). These are fully functional rigs that shipped with everything but a hard drive, lucky for me. The plan is to maximize system performance across the board.

This includes CPU, GPU, SSD and memory upgrades. We are also upgrading to HP's factory liquid cooling system, this high end option is designed to handle up to two 150w processors such as the power hungry E5 2687w, so yeah, pretty much a killer CPU cooler and cooling system that would have been insane for overclocking if you could actually do that. Very robust cooling systems on the z820, including active cooling on all 16 DIMM slots and double redundancy on airflow through the case, front to back. I'm actually amazed at how well this computer was put together, and tool less design; coming from poor quality consumer based throw away retail machines for most of my life, it's refreshing to see some quality products and extensive documentation from HP. Now if HP could turn some of this talent over to the retail space, I would be a much bigger fan.

A little context:
This workstation offered best in class performance back in the day and even in 2020 appears to have pretty good bang for the buck if you enjoy tinkering with high performance equipment from yesteryear. Plus, the CPUs are relatively cheap, but you may have to go to China to get them. That's where I had to go for my E5 2673s, absolutely nothing on the US market because it's an OEM chip and was never released for public sale. It's actually quite a rare CPU and they can be difficult to find.

The HP Z820 has 16 DIMM Slots supporting DDR3 ECC Memory.
The first generation Z820 using the Xeon Sandy-Bridge processor supports up to 128GB of 1333MHz memory.
The second generation Z820 using the Xeon Ivy-Bridge processor supports up to 512GB of 1866MHz memory. <-------- I have this one


*I also have the high end 1125W HP Gold rated PSU option. The HP Z820 Workstation power supply is also capable of 1450 watts of output power when the input voltage is greater than 180V.

*I will be running 2 SSDs in a raid 0 configuration (for a total of 1TB storage) for the boot drive/data and a 64GB octal channel 1866MHz memory kit (populating all 16 slots to achieve a theoretical maximum of 100GB/s r/w/c bandwidth, and this is only possible with all eight channels across both CPUs are fully populated.

*I did my best to find better ram with tighter timings, but unfortunately, my search did not reveal any compatible memory in this case, nothing out there with better timings that will work with this machine. At the end of the day, I am netting around 70 ns latency which is really not that far off what AMDs current Ryzen chips will get you.

On the GPU side, I'm going with the MSI Radeon RX 5700 XT GAMING X 8GB. This GPU is a powerhouse and so I should be futureproofed for quite some time with this beast under the hood. The MSI Gaming X model in particular, has one of, if not the best, cooling system out of any 5700 XT on the market. It also offers the lowest noise at idle and is tied for being the quietest GPU under load as well. That's quite a challenging feat that MSI seemed to pull off here. Even when the fan is on, I dont even notice. One look at the heatsink and you'll get my drift. This is a great marriage, at least in my book, to bring the 5700 XT and an HP z820 together, united forever as a peer powerhouse. This is going to offer excellent GPU performance for this rig for many years to come. It's also a cool running card with a sick backplate that minimizes and directs heat away, so as not to deep fry your CPU from below. Case temps are also dramatically lower (on the other test rig I have) when you look at internal temp numbers from the RX 5700 XT and the RX 580 with the side panel on.

The Xeon E5-2673 v2 8 core 16 thread 3.3GHz CPU was chosen for a few reasons.
First, I could have gone higher core count, but the trade off is clock speed, and since I'm intending to use this for benchmarking and gaming, I favored the higher clocked chips (plus I naturally favor per core performance over core count anyway). Second, the E5 - 2673 v2 turbos to the magic number of 4GHz. Another thing, the E5 - 2673 v2 has a TDP of 110W, whereas virtually every CPU similar to it, or above it is either 115w, 130w or 150w, (and these require the larger PSU and liquid cooling option). I have all the needed hardware to run the high end chips, but I can do without the heat for the same clock speed, thank you very much. So a good blend of clock speed, single core performance and pretty good power efficiency overall, as you can see outlined in the snip.

1598389431195.png

*The CPU has a base speed of 3.3GHz, however, the all core turbo is 3.6GHz, effectively, giving the processor a real base clock of 3.6GHz, identical to the much hotter 2687W or its retail equivalent, the 2667 v2. In theory, all three 8 core CPUs have an all core turbo speed of 3.6GHz and a single core turbo speed of 4.0GHz. The conclusion we can draw from this is that the OEM 2673 v2 offers identical performance to it's retail siblings at a substantially lower TDP of 110w. There is no doubt, OEM chips tend to have some advantages over their retail counterparts, as evidenced here in TDP.





These are the upgraded, high end liquid cooling system. Small radiator with pump built it - per CPU. It's amazing. To give you some perspective on performance, I can blaze through Cinebench R20 without even spooling up the fans! Holds temperature like nothing else I've ever seen.



Additional pictures of the build:






Upside down cooler (this locks into place with each 80mm fan above each CPU/radiator) This is essentially a hood. You can actually lose a fan here and the system is still able to keep operating normally.


Setup right now


RX 5700 XT Installed, great airflow through this case, runs super cool!


Typical, three display port and one HDMI


That's a serious cooler if you ask me.




.
 
Last edited:
  • Like
Reactions: hlhjedsfg

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
This CPU is superior to the flagship 2687w v2 or the 2667 v2 because it boosts to identical clock speeds at a substantially lower TDP. As I said, OEM CPUs tend to have some advantages over their retail counterparts. Take a look for yourself:

 

TRACKER

Active Member
Jan 14, 2019
169
48
28
I have two Z820 and they are first gen (sandy bridge only). Both have dual E5-2687W (v1) and i run them with 512GB (16x32GB LRDIMM) without any issues. Just memory speed is 1333MHz (limited by CPUs memory controller).
 
  • Like
Reactions: Storm-Chaser

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
I have two Z820 and they are first gen (sandy bridge only). Both have dual E5-2687W (v1) and i run them with 512GB (16x32GB LRDIMM) without any issues. Just memory speed is 1333MHz (limited by CPUs memory controller).
Okay thanks for that correction. I will update the post accordingly.

Another question: Do you know the difference between motherboard revisions?
One of my z820s is 1.0.2 and the other is 1.0.3

Just curious if you knew anything more about this. Thanks
 

TRACKER

Active Member
Jan 14, 2019
169
48
28
Mine is rev 1.0.0 and i know 1.0.2 and higher are supporting ivy bridge generation CPUs.
Not sure about 1.0.1
 

Whaaat

Active Member
Jan 31, 2020
301
157
43
This CPU is superior to the flagship 2687w v2 or the 2667 v2 because it boosts to identical clock speeds at a substantially lower TDP
It's fun, but some people don't understand that low TDP is a limitation, not a virtue. Under heavy load CPU will always drop turbo multiplier only to fit into own TDP. My first Sandy was rare 2689 with 115w TDP and turbo frequency 3.3 GHz for all cores. I tested it for a couple of weeks, sold it and switched for the twice as expensive 2687w with turbo frequency 3.4 GHz for all cores. Not worth the hassle for additional 100 MHz, right? Wrong. I use computationally intensive solvers in daily routine, but you can use anything like Linx to check that: while 2687w easily maintain 3.4 GHz under heavy load consuming 140-150W, 2689 had struggled and dropped down the multiplier to 28 only to fit into tight 115W TDP budget.
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
From my experience, the liquid coing in these HP Z820 is neither robust, nor high end.
We had a lot of premature failures with these, both inside and outside of the warranty period. And due to the size and design, cooling performance is not where you would expect when hearing 'water cooling'.
I upgraded cpu coolers to two small Noctua air coolers after the last failure. Cpu temps are better, noise is lower, and I don't expect them to break any time soon.
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
From my experience, the liquid coing in these HP Z820 is neither robust, nor high end.
We had a lot of premature failures with these, both inside and outside of the warranty period. And due to the size and design, cooling performance is not where you would expect when hearing 'water cooling'.
I upgraded cpu coolers to two small Noctua air coolers after the last failure. Cpu temps are better, noise is lower, and I don't expect them to break any time soon.
Strange, I cannot speak to reliability, as I have limited operating time in ownership... but I've had nothing but excellent results since upgrading away from the OEM HP air cooler to the OEM HP water cooling system, in terms of overall cooling performance, and I couldn't be happier.

To give you an idea of what I'm talking about, you can actually disconnect a CPU cooling fan from the "hood" that covers the CPUs from above, and still be able to operate the system normally. This was tested with a relatively highly clocked E5 2673 v2 8 core with a single turbo speed of 4.0GHz and a TDP of 110W. This chip in this configuration was also tested with Cinebench R15 extreme run. In this case the peak CPU temp for the chip during that run was 170*F, still well within acceptable limits. Obviously, this is not something I'm in the habit of doing but I was curious to see what happened and how the system would respond and my results appear to substantiate the fact that the z820 can take a pretty big hit in cooling and still get the job done. The z820 also features active cooling on all memory slots, which is a plus. I would say without a doubt, it's one of the best cooling systems in it's day.

I also have another z820 with two E5 2696 v2 processors. These are 120w tdp ivy bridge 12 core chips with an all core turbo base clock of about 3.1GHz and single core turbo of 3.5GHz. I can run prime or any other torture test and rarely see anything above 155*F, even after many hours. Keep in mind it is also holding these excellent temps with almost no additional noise or fan rpm.

In many instances, I can run Cinebench R20/R15/R 15 extreme and complete the benchmark before the fans even spool up in the first place. There are not many rigs out there that can run these types of benchmarks and successfully complete them with a zero % increase in fan speed at the end. That is a key indicator of a robust cooling system, as in going from zero load to 100% load instantly across both CPUs does not even spike temps in the least.

Keep in mind these tiny 80mm rads HP used here are substantially smaller than your average consumer based radiator, like a 120mm rad for example, yet perform much much better than most 120mm rads on the market today.
 

alex_stief

Well-Known Member
May 31, 2016
884
312
63
38
I am not trying to spoil how happy you are with the solution you have there. Compared to the CPU air coolers in those HP Z820 workstation, the liquid cooler is an improvement. As long as it lasts. Just sharing some long-term experience with quite a few of those workstations. And my thoughts on it from an engineers point of view.
In the same spirit, I am going to bite my tongue now ;)
 
Last edited:
  • Like
Reactions: Storm-Chaser

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
I am not trying to spoil how happy you are with the solution you have there. Compared to the CPU air coolers in those HP Z820 workstation, the liquid cooler is an improvement. As long as it lasts. Just sharing some long-term experience with quite a few of those workstations. And my thoughts on it from an engineers point of view.
In the same spirit, I am going to bite my tongue now ;)
On the contrary, thank you for posting your own opinions on the matter. You've peaked my curiosity.

I understand your experience is opposite of what I've seen. That's okay, it does not diminish anything! Lets disregard the reliability for now, but please don't bite your tongue, I'm interested in your own personal results as well and why you've come to that conclusion, especially since the numbers I am getting appear to contradict the notion that the z820 cooling system is sub par.

Also keep in mind the OEM HP liquid cooling system is a requirement for running the E5 2687w processor, a 4.0GHz 8 core chip with a TDP of 150 watts. In fact it is the most demanding chip (power wise) out of the entire E5 2600 v2 family, and the z820 has no problem keeping up. Even with a second E5 2687w installed, the liquid cooling system has no problem handing all 300 watts TDP in total.

I would consider a cooling system with that level of heat dissipation very robust indeed.

What makes you say the z820 cooling system is sub par in the first place? Again, lets stick to performance not reliability in this situation.
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
It's fun, but some people don't understand that low TDP is a limitation, not a virtue. Under heavy load CPU will always drop turbo multiplier only to fit into own TDP. My first Sandy was rare 2689 with 115w TDP and turbo frequency 3.3 GHz for all cores. I tested it for a couple of weeks, sold it and switched for the twice as expensive 2687w with turbo frequency 3.4 GHz for all cores. Not worth the hassle for additional 100 MHz, right? Wrong. I use computationally intensive solvers in daily routine, but you can use anything like Linx to check that: while 2687w easily maintain 3.4 GHz under heavy load consuming 140-150W, 2689 had struggled and dropped down the multiplier to 28 only to fit into tight 115W TDP budget.
I too, am interested in turbo core "dwell time" as I like to call it and how this varies from chip to chip in some instances.

Chips with low TDP are perfectly capable of running and hitting at their specified max turbo speed, provided they have the appropriate operating environment which is a factor of ambient temp, system cooling and of course, the specific type of load we’re talking about being relatively normal. But I do get your point that depending on power requirements and other demands, some chips will turbo more aggressively in certain instances.

But you've mentioned the E5 2689 sandy bridge processor has a 3.3GHz all core turbo? Because the Sandy Bridge E5 2689 definitely does not have an all core turbo speed of 3.3GHz as far as can tell? .So you will have to re make the comparison if you want to further illustrate your point and then contrast. I only bring up these processors in particular because I am familiar with them all and they happen to share identical clock speeds.

There are a total of THREE eight core E5 based xeons with a single boost clock of 4.0GHz in this family.

1 OEM 110w 2673 v2 base speed: 3.3GHz
2 RETAIL 150w 2687w v2 base speed: 3.4GHz
3 RETAIL 130w 2667 v2 base speed:

So in choosing the 110w OEM 2673 v2, we can get identical performance vs its retail counterparts... As in the turbos and their settings are configured nearly identically in every way possible, nothing is left to chance Think about it for second. Is intel going to release a Xeon E5 2600 processor to market that's bumping up against a predefined TDP restriction which is why its not hitting turbo?

Point being, the 2673 v2 was my processor choice. It was never a retail chip, it was strictly used for OEM environments. The lower TDP is not a disadvantageous to CPU performance.... For example, this CPU has an all core turbo of 3.6GHz and a single core max of 4.0GHz. It's retail rival, the e5 2667 v2 comes in at 130w, so it uses substantially more power at basically the same clock speed. You also need to have the 1125w PSU to feed the power hungry CPU in this particular system if you go with the other 4GHz 8 core CPU option, the 2687w. Which I have, but I can due without the heat of 150W TDP CPU.

The big daddy of the bunch, the E5 2687W v2, comes in at a staggering 150w TPD, and has a base speed of 3.4GHz, only 100MHz more than my OEM 2673s base speed of 3.3GHz. And that gets negated when you factor in that both CPUs have the exact same turbo frequency set for all cores at 3600 MHz (5 or more cores). The Xeon 2673 v2 uses less energy and has never throttled back on me yet. If there is some magical restriction

Yes, regardless of what's going on with my





Turbo core configuration:
3600 MHz (5 or more cores)
3700 MHz (4 cores)
3800 MHz (3 cores)
3900 MHz (2 cores)
4000 MHz (1 core)
 
  • Like
Reactions: anoother

Whaaat

Active Member
Jan 31, 2020
301
157
43
But you've mentioned the E5 2689 sandy bridge processor has a 3.3GHz all core turbo? Because the Sandy Bridge E5 2689 definitely does not have an all core turbo speed of 3.3GHz as far as can tell? .So you will have to re make the comparison if you want to further illustrate your point and then contrast.
lol, you are wrong again. e5 2689 has 3.3 GHz all cores turbo. You can simply believe it or you can find it out from intel datasheets or wiki, but only if you are familiar with arithmetic:
2689.PNG

As I said this multiplier is almost useless because first limitation is TDP. Under heavy load 2689 behaves exactly like 2670, which I have in two servers too. It simply hits 115w TDP. Constant 3.4GHz for all 8 cores under any heavy load requires about 155W of power (easily tested with HWinfo), so only 2687w is not limited by TDP, all other options, including 2690 will drop their multiplier.
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
lol, you are wrong again. e5 2689 has 3.3 GHz all cores turbo. You can simply believe it or you can find it out from intel datasheets or wiki, but only if you are familiar with arithmetic:
View attachment 15718

As I said this multiplier is almost useless because first limitation is TDP. Under heavy load 2689 behaves exactly like 2670, which I have in two servers too. It simply hits 115w TDP. Constant 3.4GHz for all 8 cores under any heavy load requires about 155W of power (easily tested with HWinfo), so only 2687w is not limited by TDP, all other options, including 2690 will drop their multiplier.
That's because sandy bridge had a non configurable TDP. So your logic here does not apply to Ivy bridge, which permits the CPU to push beyond the stock TDP for limited periods of time, further maximizing the effectiveness of the turbo system as a whole.

Keep in mind Ivy Bridge was the first intel CPU to introduce a configurable TDP. So any comparison drawn between turbo characteristics of a sandy bridge vs an ivy bridge should be taken with a grain of salt.

You are saying your chip cannot hit it's advertised turbo specs due to TDP limitation. Makes sense, but do keep in mind these later Ivy bridge chips will provide over and above the advertised TDP for short periods of time to maintain turbo boost. I'm interested in your thoughts on the 110W TDP E5-2673 v2. This is an OEM processor ONLY, meaning they can be difficult to source in the States and I had to go to China to get mine. I would suspect it might be perhaps even more rare than your 2689 sandy bridge... :)

This chip has some good attributes including a relativity low TDP of 110w and relatively high all core turbo speed of 3.6GHz. Attached is a list of three processors within the 2600 v2 family. These processors are very similar to one another. Including all three having a 4.0GHz single core turbo and an all core max turbo speed (effective base speed) of 3.6GHz and all three having 8 cores and 16 threads. So this should make for an interesting comparison. Applying this same logic to these chips, are you saying the 110 W TDP processor could potentially be slower despite possessing identical turbo / clock specifications as its big brothers? I'd think Intel would build the processor to measure up in this regard.

1600019952422.png

Keep in mind: I am able to run a full 3.6GHz all core turbo on this processor. Meaning 3.6GHz across all eight cores does not exceed the 110W TDP limit... The 2667 v2, the 2673 v2 and the 2687w v2 have a nearly identical all core turbo speed of 3.6GHz and single core turbo speed of 4.0GHz. If you want to pick the one with the higher TDP, more power to ya but it is not going to do anything for you but add more heat to your enclosure :)

Here is another chip of mine. This is a 12 core Ivy bridge with a TDP of 120W and all core turbo speed of 3.1GHz and have no problem hitting advertised turbo numbers because the CPU

To further define the characteristics here, lets look at individual core speeds:

1600024408756.png

In this case, my CPU is able to maintain 3.1GHz across all 12 cores without exceeding the stock TDP limit of 120 watts because that's not a hard wall -- the chip will allow boost clocks to exceed the TDP, in this case up to 150 watts for short periods of time

1600021791040.png
 
Last edited:

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
I specifically chose the OEM 2696 v2 for the same rational... It's actually the fastest twelve core variant available in the entire 2600 series family. Because in turbo mode, it has a higher all core turbo base speed of 3.1GHz vs 3.0 for the 2697 v2 retail equivalent, and the same single core max turbo of 3.5GHz. So that's equal performance and better power efficiency. Another clear example of why OEMs in this case are superior to their retail counterparts. Using your logic, we should say the 2697 is the clear winner because it has no TDP limitation, when the reality the 2696 v2 is the clear winner because it operates at a higher level of performance while using LESS voltage The specs are the specs and the performance doesn't change... in other words, in turbo form, the 2696 v2 has the fastest all core base clock of 3.1Ghz out of any 12 core ivy bridge 2600 series v2 chip on the market. Doing this at 120W TDP is indeed a blessing, as the system will run cooler and thermals will be better vs the same CPU at a higher TDP.

1600027842510.png

EDIT: And I suspect this is exactly why Intel marketed this chip in the OEM space only. As an OEM processor, it has to be something that was pretty well power efficient but also relatedly potent since OEM servers tend to wear more than one hat. Intel knows when they sit down to design a new OEM chip for vendors, they must look at each factor very carefully. This typically means better binning and lower power consumption vs same clock speeds as retail chip.

Seeing as how the performance values of the 2673 v2 and the 2667 v2 are identical we can also make the statement that the 2673 v2 is actually the better route to take. As I said, same performance but at a lower voltage.
 
Last edited:

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
From my experience, the liquid coing in these HP Z820 is neither robust, nor high end.
We had a lot of premature failures with these, both inside and outside of the warranty period. And due to the size and design, cooling performance is not where you would expect when hearing 'water cooling'.
I upgraded cpu coolers to two small Noctua air coolers after the last failure. Cpu temps are better, noise is lower, and I don't expect them to break any time soon.
This is so bizarre. The z820 has one of the best cooling systems ever designed for comparable high level workstation hardware. lol
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
To further demonstrate and prove my point, measured performance via benchmarking is needed. This will be a net assessment and benchmark results should speak for themselves. The only difference between the 2667 v2 and the 2673 is the 2667 uses more voltage. Performance characteristics are identical otherwise. If the 2673 v2 was in fact limited by its 110 watt TDP in achieving max turbo speeds, the chip may not live up to the full performance values of a retail 2667 v2, right?

Yet numerous benchmarks around the web show there is no difference in measurable performance between both chips I described. For you to be correct, we should be seeing less performance from the chip with the lowest TDP, and that's simply not the case with these Ivy Bridge chips.

Intel has baked in the ability for an Ivy bridge to boost above it's TDP to solve the very problem that you are describing.
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
From my experience, the liquid coing in these HP Z820 is neither robust, nor high end.
We had a lot of premature failures with these, both inside and outside of the warranty period. And due to the size and design, cooling performance is not where you would expect when hearing 'water cooling'.
I upgraded cpu coolers to two small Noctua air coolers after the last failure. Cpu temps are better, noise is lower, and I don't expect them to break any time soon.
Im assessing this from a performance perspective only but from what I gather HP used high quality fans in these CPU coolers. As in, asking you a question... you don't think a cooler rated for 300 W TDP is "high end"?

Or a cooler than can keep a combined total of 240 watt TDP below 160*F for the entire duration of a torture test across all 24 cores good cooling?

I am going to go out on a limb here and say your expectations of liquid cooling performance could potentially be off base here or perhaps you had your system configured wrong. I know this because I've run a 2687w v2 which is a 150 watt processor and that was still very much within the performance capabilities of the Z820s CPU cooling system.
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
@alex_stief
You wouldn't call a cooling system capable of dissipating 300 watts worth of TDP high end?
I think you must have the z820 confused with some type of economy level desktop lol

@Whaaat
Does this concept make sense now? Can you see why I chose the OEM 2673 v2 over the flagship 2687w v2?

As I said from the beginning, both CPUs offer identical performance, and I've got the benchmarks to back this up.

Very easy to see that lower TDP is actually a blessing after all.
 

Storm-Chaser

Twin Turbo
Apr 16, 2020
151
25
28
Upstate NY
The 64 GB 1866MHz memory kit has been installed and bandwidth payoffs are impressive to say the least. The kit contains a total of sixteen 4GB modules and makes use of all eight available memory channels to deliver peak memory performance to the system. This is an aging machine, yet still to this day, remains competitive in terms of overall memory performance / bandwidth performance.

Here you can see AIDA64 is incorrectly
identifying my OEM 2673 v2 chip as it's retail equivalent, the 2667 v2, which is identical but uses more voltage at the same clock speed.

Was the 64GB memory upgrade worth it? Why don't you tell me :)



imageproxy.png
 
Last edited:

BlueFox

Legendary Member Spam Hunter Extraordinaire
Oct 26, 2015
2,059
1,478
113
Except it doesn't. Modern single socket systems have considerably more even with half the CPUs. Dual socket AMD Epyc nets you ~4x the memory bandwidth.

The others are also correct about TDP and multipliers. Turbo boost is not indefinite.