Dual Xeon E5 v4 workstation/server, dist. computing, render farm, server 1st build

[Rapturoso]

I was thinking:
2x E5 2630 v4 ES/QS retail stepping
64 to 128 GB Corsair Dominator kits (2 x 32/64GB quad channel 8 DIMM sockets fully populated).

Got the idea from Level1Techs on YouTube as he had some Dominator memory installed there on an ASUS board but also mentioned he couldn't get it to POST on non ECC DIMM.

Any thoughts? Searched for this specific issue but come up with nothing on here. Any other WS/server board suggestions?

There is some moderate to serious power needed with regards to multi-threaded operation, a minor one being 4K HDR 10/12BIT video work and some pretty heavy gaming and imaging/rendering at 1440p and above. That said, these are not the most taxing applications for a 40+ thread behemoth, so with all those extra horses under the hood I was also considering to branch out a bit with this build and make it a fairly comprehensive and fleshed out all-rounder. A mini lab workstation/server of sorts, with 24/7 duties as a file server, a dumping ground for every bit of media and data I produce and consume. Basically a nice set of powerhorses under the hood in one neat and tidy solution, possibly with a couple of connected remote units to access that power whilst elsewhere.

I'm hearing horror stories about ASUS dual boards, so maybe they aren't the best to go with. However, I have been reading some great success stories with Supermicro, even with ES/QS Xeon processors which should prove much less expensive and allow me to spend more on other things, like 10GBIT SFP and other items that will give this build some grunt and future proofing for a few years at least.

Feel free to share your experiences and don't hesitate to deviate a little from my needs, all will be considered. I'm ideally looking for something I can pretty much do everything on studio wise, without breaking a sweat but also be more than adequate at server duties too and maybe some distributed computing.

What I'm not looking for is a data centre industrial grade rack that sits crunching heavy data sets all day long as this is way OTT. Ideally something between a 3D rendering and audio production workstation and an i/o intense server would be the goal of this. I will also use this to relax and game once in a while... obviously! Who wouldn't?

 

[TLN]

1. Why 2630 v4? Think there're CPU's that are better for rendering.
2. Why Dominator? You won't be able to overclock CPU anyways.

I got almost the same board(D16WS), will be building with V3 cpu though, when I get the CPUs. My 1st build too.

 

[PigLover]

Don't waste money on "branded" memory for a server build. You'll pay for things you don't need (pretty heatsinks) and can't use (overclock capabilities and SPD values beyond what the CPUs will activate). Just go on eBay and buy 64GB DDR4 RDIMMs (Crucial, Hynix, etc). Going price for 64GB RDIMMS is up a bit in the last few weeks, expect ~$190/stick. For an E5 v1-v4 build always install them in sets of 4 per CPU.

I have a pretty strong preference for SM boards for a build like this, though it is a preference thing. ASUS builds good stuff too - just that they are so "consumer/overclocker" focused that I believe they add in useless baggage on most of their boards. SM boards are KISS. And once you've tasted remote management by IPMI you'll never want another board without it.

@TLN, wonder what CPUs you see as "better for rendering"? Personally I see the E5-2630 as a something of a cost/performance/power "sweet spot" for most builds like this. You can get more cores or faster cores but cost and power consumption rises pretty fast as you do,.

 

[patchate]

Where are these 64gb Rdimms for $190? I would love to know your source!

 

[TLN]

@PigLover I think 2658-v3 (12core @2.0, 2.9 max turbo for $160) would be way better. Or 2683v3 (14 core @2.0, 3.1 Max turbo for a bit more).
Here's a topic with benchmarks for all CPUs. Extra cores are very beneficial for rendering apps.
Dual xeon build for rendering and low power consumption are quite opposite things, isn't it?

 

[Rapturoso]

1. Why 2630 v4? Think there're CPU's that are better for rendering.
2. Why Dominator? You won't be able to overclock CPU anyways.

I got almost the same board(D16WS), will be building with V3 cpu though, when I get the CPUs. My 1st build too.

Hi there. Thanks for the quick response. I'm sure you have already gathered that this is my first departure from enthusiast motherboards and general purpose system builds and it has taken me 12 years to decide that I actually want to do it and do it right first time. I'm not completely in the dark about some of the more commercial and industrial computing applications where Intel CPUs are concerned. That said, I need to read about SFP more and also the different types of storage solutions as there are so many. SAS, SATA1/2/3 (the good old one I'm used to), SATA Express and a few I have never heard of too. I'm going to stick with plain old SATA3 SSDs for the OS for now but will be installing non SATA m.2 to replace the SATA SSDs shortly after the build. There will be many drives attached, plenty of network storage drives connected internally on the in-built SATA connectors once I've researched the tech above.

I would like to take the opportunity to absorb as much insight regarding my first proper server/workstation build as possible from people with years of first hand experience within server and workstation environments. I'm no stranger to gaming and general optimisation for productivity purposes and music production using multiple VST instruments and effects ( multi-threaded processors go some way to alleviate the strain this puts on regular desktop machines ). However I am a little bit of a newcomer to some of the more server, big data set and number crunching orientated builds.

I will elaborate as to why I have suggested such items for the bare bones of the system but please remember that I am still unsure which server/workstation board to use:

1. I decided to go for the E5 2630 v4 because they are cheap, have much better performance per Watt so they are many times more economic than the E5450 I have on my 12 year old ABIT IN9 32X MAX (nVidia 680i SLI with microcode added for 775 to 771 XEON mod). This processor whilst being pretty good, still doesn't better any of the equivalent i series Intel desktop processors that came after it and is limited to an unlinked memory speed of DDR2 1066MHz and an unlinked processor of 1333MHz. That said the processor is also very old and doesn't support half the optimisations and features that the Xeon E5 2630v4 supports.

PigLover put it plainly, cost/performance. I got 2 x E5 2630 v4 Xeon for a total of $330 to my door, which I believe is a pretty good deal. If you can get 2.2GHz and 3.1GHz Turbo in a single CPU with 10 or more cores for less $ at the same power/performance ratio then I would be interested. Please point me in the right direction.

There may be better processors for rendering but also music production is CPU heavy too, especially if you are monitoring at reasonable quality in real-time. Do you mean rendering in a multi-threaded sense? From what you and PigLover said this appears to be what you're implying. Of course, more fast cores at once is always a good thing! I'm not sure how much of an impact the limited 3.1GHz Turbo will have.

2. The Dominator memory was mentioned in the video. It was just a suggestion by the video creator to keep the price down as opposed to using LRDIMM or RDIMM but seeing as he stated the board wouldn't POST with his "non ECC memory" (direct quote from the video, I'd assume he meant the Dominators) it makes them not an option for this build. As far as I can tell these Dominators are non ECC gamer and enthusiast over clocking memory, right? That brings me to the other question of what will be appropriate for this or other boards like this without breaking the bank and keeping the performance adequate rather than unnecessarily causing a bottleneck.

I am totally open to the suggestion of a Supermicro board but would like more workstation and server features. The Asus Z10PE D8 WS board lacks the remote management feature and a few other features that other manufacturers seem to bundle as standard.

 

[PigLover]

Where are these 64gb Rdimms for $190? I would love to know your source!

oops - typed to fast. 32Gb RDIMM.

eBay. Used. See here for one: 805351-B21 32GB DDR4 2400MHz ECC RDIMM Memory HP Proliant ML110 ML150 ML350

 

[ServerSemi]

I have an asus board and it is giving me issues I would not recommend you buy one because they don't cross ship you a new/refurb. You gotta wait a long time to get it back. I would be getting other brand on my next upgrade.

 

[Rapturoso]

Any suggestions for memory over 8 slots? I was thinking

8 x 16GB RDIMM single rank (for pure performance) RDIMM (2400MHz to account for a possible processor upgrade beyond the E5 2630 v4 2133MHz memory limit)

or even

8 x 32GB dual rank LRDIMM 2400MHz (better performance than RDIMM dual rank) if someone knows a good source of quality tested used memory that's a bargain. I don't mind used equipment as long as it's tested thoroughly.

 

[Rapturoso]

The final build:

ASUS Z10PE D8 WS
Decided against the D16 version as 16 RAM slots is excessive for this build and the m.2 slot is full bandwidth rather than half bandwidth m.2 on the D16. Also 10Gbit can be added later with a PCIe riser/splitter to take full advantage of one 16x slot.

Xeon E5 2630 v4 ES2 (retail stepping) 2.2GHz 3.1GHz boost 2133MHz memory bus 10 core 20 thread CPU x2
These will be upgraded later when 20 core 2400MHz memory bus CPUs come down in price after the Skylake Xeons launch.

Samsung DDR4 16GB RDIMM single rank memory modules x4
These will be useful both in dual channel mode and quad channel mode when I have the need to add a further 64GB to the system. Also 2400MHz will provide a speed boost when the processors are upgraded.

MSI GeForce 970 4GB (3.5GB + 512MB) OC x3
There's absolutely no need for three of these with Davinci Resolve, Adobe or live Twitch streaming and 3D live face/body replacement rigging but having them for things like casual gaming on top of some light crypto mining and video and blender rendering work doesn't hurt. When the 1080tis come down in price they will be replaced. This is a long way off.

Cryorig R1 Ultimate x2
Nice cool CPUs, zero noise, no liquid = less to go wrong. Also very BIG!

Sandisk Extreme Pro 960GB x2
Just because I can.

Western Digital 1TB Caviar Black x2
Faster spinner and scratch disc.

Western Digital RED NAS 3TB x2
Slow spinner backup and redundant copies.

EVGA P1200 Platinum
Super efficient and reliable. Cool and quiet. Taken from my outdated ASIC crypto mining machines.

Lian Li PC V2120A case
It's large and houses everything perfectly. Nice and cool with the below fans (stock fans removed).

Black Noise NoiseBlocker eLoop B120PS x3
Black Noise NoiseBlocker eLoop B140PS x3
Completely silent, ultra-reliable position-invariant magnetic bearing NB-NanoSLI, cool running, efficient, amazing.

Razer Blackwidow X Chroma
Added o rings to dampen bottom out but still have positive tactile and audible feedback. Programmable dynamic lighting for switching between studio and video functions with visual dynamic feedback of editing and production elements (and gaming). Simple keyboard. Totally OTT.

Logitech G502 Proteus Spectrum
For accurate tracking and ergonomic.

Rocccat Taito mid-size 3mm mouse mat
For accurate tracking whilst producing any kind of design and art work (and gaming).

A few other less relevant bits and bobs such as monitor speakers, audio interfaces, 4K production cameras, monitors, control surfaces and MIDI etc.

All done