My December 2013 Workstation Build

Patrick

Administrator
Staff member
Dec 21, 2010
12,168
5,234
113
Decided it is time to share a bit about the workstation I have been building for awhile now. I will update this thread in awhile.

The "Old" Workstation

The previous workstation was built almost two years ago. It was a rock solid build but needs have changed. This would have been a simple swap, but I decided to "go bigger" and moving to a larger 12x13" motherboard meant a new chassis since the previous one was pushing it:


Just barely enough room for the SATA connectors.

Old Workstation Specs
  • Intel Core i7-3930K
  • Corsair Carbide 500R
  • Corsair H100 Cooling
  • 64GB DDR3 (8x8GB)
  • AMD Radeon HD6970
  • Crucial M4 256GB OS SSD and various SSDs

Reason for the new build

  • Needed more than 64GB DDR3 - I have been using Hyper-V quite a bit recently and even with Dynamic memory, using applications like Photoshop CC alongside a host of Hyper-V machines meant I was actually using most of the 64GB
  • Wanted to right-size GPU and networking. The current GPU was too much for what I use the machine for. Networking was still provided by dual 1 gigabit network connections
  • No longer use 3.5" storage locally, even for backups
  • The parts were finally available

New Build
Certainly there were a few things I would have loved to have done with the build, but in the end, it is important to right-size workstations a bit. The NVIDIA Quadro K4000 is awesome since it uses only 80w. That somewhat offset the fact that the Intel Xeon E5-2687W chips are power hungry. One item that I will look into is moving to something more like dual E5-2630 chips for significantly lower power consumption (190w combined TDP v. 300w).

Specs
  • Dual Intel Xeon E5-2687W
  • Supermicro SC732 Chassis
  • Cooler Master Hyper 212+
  • ASUS Z9PE-D8
  • 128GB (8x16GB) DDR3 1600MHz ECC RDIMMs Samsung
  • NVIDIA Quadro K4000
  • (Coming soon) Mellanox 56gbps FDR IB/ 40GbE
  • Crucial M500 960GB SSD
  • Seasonic 650w 80Plus Gold power supply

For those wondering, the reason for the dual port Mellanox 56gbps IB card is that I am strongly considering re-wiring for IB both to the lab and to the pfsense appliance.

Thermals
In 2014 we will be officially adding thermal imaging to our testing. Decided to snap a quick image of the workstation under load. Fairly solid results as the max temperatures are in the 50-55C range at the hottest spots. Well under what we would want to see.

I did get some comments mainly saying the ASUS motherboard is running hot. As one can see, the temperatures max out at 50-55C. For comparison, a Intel Xeon E5 processor has a max operating temperature of over 80C so these ICs are easily fine. The other big note is that the single slot cooler on the NVIDIA Quadro K4000 has a cold zone near the rear of the card. The front of the single slot cooler shows it is dumping heat into the chassis. Since the GPU was under load in this picture, it is expected and cooling is working properly.

A word on the easiest workstation transition ever
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,009
920
113
NYC
Thermals
Trying out the report building feature:
DROOL how freaking cool is that? So it takes an optical and a thermal image? Low resolution but you can certainly make everything out.

.... and I see you are in the special SSD drive mounting club too.
 

PigLover

Moderator
Jan 26, 2011
3,012
1,314
113
An idea for you. If you want to lower the noise, etc.

With that case, that MB and Hyper-212+ you can actually cool both CPUs and get your case exhaust using a single fan. Take the fans off the towers and build a sealed duct connecting them. Ghetto version is cardboard and duct tape. Clean version uses this: Thermalright 120mm Fan Duct - Blue - RM Accessories

Then mount a high performance low noise 120mm fan at the case exhaust and duct it to the rear tower cooler. The rear CPU won't be as cooled as efficiently as the front one. And it might be pushing the envelope a bit for 150W CPUs if you really press the limits for long period of time. But it should work very well.

I built this configuration for a friend once and it was amazing - dual E5s, cool and quiet.
 

mixtecinc

New Member
Feb 18, 2013
29
0
1
Patrick,

Could you post the name brand and part number of the memory you used?

Any thoughts on using ES processors in this setup?

Thanks

Justin
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,168
5,234
113
Hi Justin. They are Samsung 1600mhz dimms. Will find the model number when I get back.

On ES you can so long as they are later revisions.
 

clonardo

New Member
Sep 17, 2013
6
0
1
Nice build.

With regard to ES CPUs, the Asus boards work well with C0-stepping Sandy Bridge CPUs in most BIOS revisions. With B0, they will work, but you're limited to old BIOSes and can't downflash (on the Z9PA-D8, I've had luck with the BIOS revisions released through March 2013).
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,168
5,234
113
Added a bit more to the build log today including:
  1. memory information,
  2. networking information,
  3. a bit more on the build logic
  4. discussion on why the thermals are actually fairly good
 

Aluminum

Active Member
Sep 7, 2012
431
45
28
Devils advocate here, why did you go dual socket? Now you are splitting that ram across two cpu IMCs so you really only get 64GB with proper full bandwidth for something like photoshop, VM type loads of course you can segment.

You also had to pay the rather large premium for a E5 2xxx, or a severe drop of clock speeds compared to the equivalent 1xxx. (granted there are no 10 or 12 core options yet, but the clock wall on those really sucks for anything but core-loving loads)

If you needed more cores and more ram I get it, but there are cheaper and equivalent feature single socket 2011 boards that have 8 dimms and accept registered. The wonderful variant with "almost free" 2x10GbE and LSI 2308 comes to mind ;)

The 32GB/R/ECC dimms are retardedly expensive at the moment but a 16GB/R/ECC is cheaper than an equivalent pair of 8GB/R/ECC based on my quick lazy google check. (YMMV, void where prohibited)
 
Last edited:

Patrick

Administrator
Staff member
Dec 21, 2010
12,168
5,234
113
Devils advocate here, why did you go dual socket? Now you are splitting that ram across two cpu IMCs so you really only get 64GB with proper full bandwidth for something like photoshop, VM type loads of course you can segment.

You also had to pay the rather large premium for a E5 2xxx, or a severe drop of clock speeds compared to the equivalent 1xxx. (granted there are no 10 or 12 core options yet, but the clock wall on those really sucks for anything but core-loving loads)

If you needed more cores and more ram I get it, but there are cheaper and equivalent feature single socket 2011 boards that have 8 dimms and accept registered. The wonderful variant with "almost free" 2x10GbE and LSI 2308 comes to mind ;)

The 32GB/R/ECC dimms are retardedly expensive at the moment but a 16GB/R/ECC is cheaper than an equivalent pair of 8GB/R/ECC based on my quick lazy google check. (YMMV, void where prohibited)
Great points on all counts.

Some of the decision was based on what I had on hand. Not having to buy new components (other than the Crucial M500) helped make the decision. That linked motherboard did not come with built-in audio and no USB 3.0. I also wanted IB/ 40GbE rather than 10GBaseT.

I am using 16GB RDIMMs that I had available. Still only 8.

Will add more on why dual socket next time I get a chance.
 

RimBlock

Active Member
Sep 18, 2011
824
27
28
Singapore
Your built in audio you what now ? :p.

I went for a Asus Xoner DX a few years ago to get off the ASRock Extreme 4s Realtek audio chipset and it made quite a big difference to the clarity of the music and sounds from the PC.

I recently got a decent set of PSB headphones and a FiiO X3 music player which also acts as an external DAC via USB. The sound quality has jumped up another level with it. I have now removed my Xoner card and just use the player. I will probably get a FiiO Andes E07K to have as a dedicated USB DAC for my machine. The advantage is that I do not have to buy boards with onboard sound and I can use it as a portable headphone amp if I should want to. The quality is much better than the onboard chipsets I have personally heard and I am in no way an audiophile. I have a set of Creative Labs T20s for the PC, so nothing particually special (although they are pretty good for what they are).

Get an external USB DAC, get better quality sound and free up your motherboard options.

Just a suggestion :D.

RB
 

mrkrad

Well-Known Member
Oct 13, 2012
1,244
52
48
Why don't you use the K4000 hdmi audio? You can get roms with it enabled (or not)?

I'd love to get some new hotness but I can't justify the cost of L5639 $75 6-core/60 watt. Boxes made with those are stupid cheap.

You never want on-board RAID/Networking. ESP not 10gbase-T when someone plugs it into a phone jack or something dumb and torches something.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,168
5,234
113
RimBlock said:
Get an external USB DAC, get better quality sound and free up your motherboard options.
Unfortunately that still does not solve my USB 3.0 need. And, of course, the Z9PE-D8 was ready to go and is working well.

Why don't you use the K4000 hdmi audio? You can get roms with it enabled (or not)?
I generally use a very simple speaker setup for the PC. RB's idea of the USB audio is a solid one and was certainly a thought given how easy it is. HDMI out would be easy except I am using the two DP and do not have speakers that would receive HDMI audio.

Now... next build there are certainly things I will do differently.