512Gb RAM Apartment Server?

[Otto]

Edit: Updated Build 3/26/2017

Thanks for all the helpful advice - this is shaping up nicely!

Build’s Name: Big RAM - Small Apartment
Operating System/ Storage Platform: Windows Server 2016 DC, Hyper-V VMs
CPU: 2x Xeon E5-2683 v3 ES/QS 2GHz 120W ($350 on ebay)
CPU Cooler: 2x Noctua NH-U14S 55.0 CFM CPU Cooler
Motherboard: ASRock EP2C612D16C-4L
Chassis: Fractal Design Define XL R2
Drives: 2x Samsung 850 EVO 2Tb SSD (boot and VMs)
2x WD Red 4Tb 5400 (mirrored, backups)
RAM: 16x Crucial 1x32Gb RDIMM 288-pin DDR4/2133
Video Card: MSI GTX 1060 GAMING X 6G
Power Supply: SeaSonic PRIME Platinum 1000W 80+ Platinum
Other Bits: Fan replacements, extra fan controller if needed
Usage Profile: SQL Server/IIS, load testing, performance tuning, high availability scenario testing

Other information…
{Expletive adjective} {expletive verb} {expletive adjective} RAM prices!!!!
I'd really prefer to go with the 64Gb LRDIMMs and get the headroom up to 1Tb, but I'm having a hard time with the 50% premium over the 32Gb sticks. Even so, I'll be buying the RAM in 128Gb stages over several months as I can afford more - I can mess with shared memory to the failover SQL Server VM to get some useful work done at lower RAM totals.

I'm somewhat concerned about getting ES/QS CPUs, but the price/performance ratio makes it worth the risk to me. Once I get closer to buying, I'll post details on the ES thread here and make sure that I improve my chances of getting a working pair.

Minor bits:
- Fractal Design support confirmed that the XL case works fine with an SSI EEB mobo, just need a couple of standoffs. I also like the look of the Phanteks Enthoo Pro, which looks like it can be just as quiet if I add some acoustifoam.
- The video card is for my wife's machine learning, will have to figure out how to pass it through to a VM

Now to save a little more money and do my research on CPU/GPU fans total CFM vs case fans CFM, verify the power supply needs, and remember that it isn't a laptop so I'll need a UPS if I care about graceful shutdowns.

Thoughts?

------------------------
Original post:
Help! This is my first build in a decade, since we moved from a large house to a small apartment and I switched to a laptop-only world. I haven't kept up with server hardware much in the interim, so I'm a bit at sea.

Build’s Name: Big RAM - Small Apartment
Operating System/ Storage Platform: Windows Server 2016 DC, Hyper-V VMs
CPU: 1 or 2 Xeon, 20-24 cores total
Motherboard: ASRock SSI EEB? (dual), X99? (single)
Chassis: ???
Drives: ???
RAM: 512Gb
Add-in Cards: None
Power Supply: ???
Other Bits:

Usage Profile: SQL Server/IIS, load testing, performance tuning, high availability scenario testing

Other information… I’ve been using my prior laptop as a home test bed for software development, but the venerable m18-x has finally given up the ghost. I live in a small NYC apartment and the laptop was been nice and quiet, but was also severely memory limited (32Gb).

I really need something that can do useful testing for a production system that runs on 4x 12-core (dual 6C Xeon E5 2640s)/128Gb RAM with 10GpE. Our DR environment runs fine on half the CPUs so I don’t need 48 cores, but there’s no getting around the memory requirements.

Noise is of some concern, since this will sit in a corner of the bedroom, but it also won’t being doing much while I’m asleep, so quiet at idle and under light workload is probably what I’m shooting for.

This is just the beginning so I have a bunch of questions:

1. Single, hi-core count CPU or dual with lower core counts each? The dual approach would give me some space for processing upgrade down the road if needed.
2. How important are the newer Xeon revs (v3/v4)?
3. 8x64Gb or 16x32Gb for the RAM? The 32’s leave room for a future upgrade, but is it useful?
4. Large case with room for big slow fans, or a denser arrangement? Which is better for noise control and cooling? I’ve got room for a full tower, but smaller is nicer.
5. Do I need NVMe or will SATA SSDs work? I’m mimicking 10GpE iSCSI to a 10K SAS spinning disk SAN so even a slow SSD has 2 orders of magnitude better iops, but I’m very interested in database tuning and don’t want to find myself bottlenecked on the I/O.
6. I don’t currently have any monitor in the apartment – is it possible to do the setup and install with the server headless?

 

[i386]

1.) If you don't many pcie lanes then go single cpu (40 pcie lanes)
2.) (usually) Newer cpus = same or better performance with less power, but higher price
3.) if you go the single socket route you will be limited by 8 slots, with lr-dimm you can still get 512 gb ram (8x 64gb)
4.) I know that supermicro has some 1u servers with 20+ cores and 512gb ram, but they are loud and you don't want to be in a room with them. Get the largest chassis you can get. Maybe some of the supermicro 4u/twoer, like the sc745 which supports rackmount/how swap psus (eg 920sq).
5.) that depends how much money you have left after getting 512gb ram
6) yes, if the mainboard/server supports ikvm (ipmi like on intel mainboards without the remote control board is not neough)

 

[Aestr]

I'm in a very similar boat about downsizing. I've ordered one of the SuperQuiet SC745's and while I'll be going single socket to start and then gauge my needs from there I think the noise and space constraints are similar. I'll let you know what I think of it.

 

[Evan]

DDR4 using 16x32gb dimm, that's regular dimm, not load reduced that would be needed for ddr3. Also the ram will still run at 2133/2400 compared to probably 1300 with that many LR DDR3 dimms in a system.

In a bedroom, keeping this sleeping quiet is going to require some careful thought ! But certainly possible.
I watch with interest

As an aside did you think to split it up into a couple of smaller systems and only keep one running 24x7 if needed and start the others when you want to load test ? Maybe a nice single e5 or Xeon-D for 24x7 and some older Gera that can run for load testing when needed but not to silent ?

 

[Jeggs101]

@Otto I think you're going to fit in well at STH. Welcome!

What about one of those SM towers?

 

[K D]

From what I understand, this is going to be a stand-alone box that is going to be tucked into a corner of a bedroom apartment. If you dont want the bells and whistles that SM chassis's offer, then look at a consumer tower chassis like Fractal Define R5 or the bigger XL. I've used the R5 with an ASUS Z8 dual CPU board with dual E3-1245 and 5-6 HDDs. The case is quiet, has good thermals and is highly configurable and the newer revisions are even better than the one I used.

I have not used an SM tower workstation but have heard good things about them. But I don't know if they can be quietened to the level a consumer case like Fractal Design can be.

 

[Diavuno]

I meant it is my older, but I have a dual X8 system in a NZXT case. The computer is almost completely inaudible. I've had it loaded up to 200 gigs of RAM give or take a few... You will need to go supermicro if you want to use an SSI board e-atx you can normally find a full tower. Cooling the tips that can be an issue, but can be done.

Sent from my 0PJA2 using Tapatalk

 

[Otto]

Holy information overload, batman - you folks rock!

v4s will be lower max TDP at the same core count & speed
big cases are easier to quiet/cool
stick to 16-slot RAM since LRDIMM is massively slower (dang, I had no idea!)
the Fractal Design cases are gorgeous, but limited to ATX mobos
the SM cases look kinda terrifying, but that that will be the route for an SSI mobo
ikvm will let me do headless setup (yea!)

I'm leaning towards dual 10-12C since the 20C & 22C draw a ton of power (135-145W!), and my instinct is that it will be easier to cool a pair of lower power chips.

Looks like I've got a bunch of research to do (the fun kind <grin>)

Huge thanks to all!
-Otto

 

[Patrick]

@Otto - I was going to put this SM one on Craigslist Q1 2017 STH Lab Cleaning

No RAM for you (sadly that is tight at the moment) but I may be able to help with CPUs as well.

 

[Logan]

Adding to the quiet case recommendations of Fractal Design and NZXT, I have been very pleased with Nanoxia.

 

[zer0sum]

Holy information overload, batman - you folks rock!

v4s will be lower max TDP at the same core count & speed
big cases are easier to quiet/cool
stick to 16-slot RAM since LRDIMM is massively slower (dang, I had no idea!)
the Fractal Design cases are gorgeous, but limited to ATX mobos
the SM cases look kinda terrifying, but that that will be the route for an SSI mobo
ikvm will let me do headless setup (yea!)

I'm leaning towards dual 10-12C since the 20C & 22C draw a ton of power (135-145W!), and my instinct is that it will be easier to cool a pair of lower power chips.

Looks like I've got a bunch of research to do (the fun kind <grin>)

Huge thanks to all!
-Otto

You can actually fit an E-ATX/SSI board into a lot smaller case than you think.
The usual issue is just that the standoffs are in the wrong places, but you can fix that with a drill and a tap in minutes
Or you could go with the Supermicro SC732 which is small and has the right standoffs

 

[Otto]

Cool, more to process!

@Patrick - That is a sweet price for the SM tower! I'm nowhere near ready to pull the trigger yet, but if you've still got it in a few weeks I'll be really tempted - where is "local pickup" for you? For the CPUs, I've been watching the ES models on ebay (those prices are crazy low), and I'd love to hear what you can hook me up with.

@Evan - I considered splitting the system into a couple of smaller boxes, but there are two issues that I've run into. First is the networking - I'd need to put 10GpE in place since 1G won't cut it between DB, App and Cache machines, and I don't really want more cabling in the corner. Second is my significant other - she's limited me to a single tower <grin>. I even flirted with doing this all in the cloud; however, MSDN licenses can't be used for cloud instances, so the hourly rate gets out of control really quickly.

@zer0sum, @K D - Fitting an SSI EEB into a big, quiet consumer case is really starting to appeal to me. They look great, and I won't be chewing up my fingers trying to make things fit in small spaces.

I'll update the build spec next weekend after I've poked about some more and see where things stand.

Thanks everyone!

 

[Klee]

Here is my first AsrockRack dual xeon SSI-EEB sized motherboard in a Phanteks Enthoo pro case.

Also you can run a single cpu in a dual cpu motherboard and that gives you the option of purchasing another identical cpu plus ram at a later time and they will be cheaper, possibly a lot cheaper and that will offset the initial price of the dual cpu motherboard over a single cpu motherboard.

 

[Otto]

@Klee That's a beautiful setup! How does the Phanteks do at noise control under load?

 

[TLN]

1. Xeon E5-v3.
2. Because they have all-core-tubro bug and that's useful
3. Dual CPU, in case you need it.
4. Bigger case, better coolers - this all will keep your system cool and silent.
5. NVMe or SSD is always good thing. No need to get the fastest though.
6. With proper motherboard you can get to BIOS remotely. No need in monitor.

 

[nk215]

Unless power is free or very low cost, you should ask yourself if you need everything to be on 24/7.

If you don't, get 2 machines. First one is a E3 variant (they are in the 40-50 watts idle and virtually silent). Second one is a dual CPU E5 that doesn't need to be on when you sleep. This cuts down the noise and power cost while you sleep.

In all practical purposes, for home server with a hand full of users, you can't tell a different between NVMe or SATA SSD. It comes down to how much you dislike cables and cooling setup. VNMe especially the M2 with PCIe adapter (950 pro for example) run hotter than SATA SSD. Many PCI NVMe cards expect some airflow.

A dual E5 can be very silent with tower cooling or water cooling system with mostly solid state SATA SSD. If you do water cool, you don't need any case fan. Radiator fan pulling air out is enough (I do use a small 40mm fan on the chipset heat sink however).

 

[Rand__]

There is also an ATX sized Dual CPU Board btw if you go the split way (Supermicro | Products | Motherboards | Xeon® Boards | X10DRL-i), it's only got 8 DIMM Slots so not really for your AIO with 512GB unless you want to spend a ton of money on Ram (64GB modules). But it would allow you use two quite small boxes, eg with diverse CPU setups (many cores/high frequency)

 

[Klee]

@Klee That's a beautiful setup! How does the Phanteks do at noise control under load?

I added an additional 200mm fan at the top rear of the case and its hardly noticeable to me, its to the right of me under my desk.

The case is very flexible when it comes to cooling options.

I have upgraded the motherboard to a newer socket 2011-3 motherboard but its also an AsrockRack SSI-EEB sized motherboard.

 

[Otto]

@nk215 @Rand__ Splitting into multiple boxes isn't really an option - partly because primary and failover DB need 128Gb each and 10GpE network to shared storage, so that takes me to 2 boxes + network and cables, and partly because it is a really small apartment and I don't have the room I probably will shut things down at night if the idle noise/power consumption is too high - other than an occasional long stress test, I don't imagine much overnight need.

@Klee Thanks! Glad to hear the noise levels are good in the Phanteks. This is definitely in consideration.

@TLN For NVMe vs SATA SSD, I'm seriously spoiled by the 950 Pro in my main laptop, but you are right that the performance difference is irrelevant to this server - EVO 850's it is...

 

[Otto]

Updated Build 3/26/2017
(also edited first post)

Thanks for all the helpful advice - this is shaping up nicely!

Build’s Name: Big RAM - Small Apartment
Operating System/ Storage Platform: Windows Server 2016 DC, Hyper-V VMs
CPU: 2x Xeon E5-2683 v3 ES/QS 2GHz 120W ($350 on ebay)
CPU Cooler: 2x Noctua NH-U14S 55.0 CFM CPU Cooler
Motherboard: ASRock EP2C612D16C-4L
Chassis: Fractal Design Define XL R2
Drives: 2x Samsung 850 EVO 2Tb SSD (boot and VMs)
2x WD Red 4Tb 5400 (mirrored, backups)
RAM: 16x Crucial 1x32Gb RDIMM 288-pin DDR4/2133
Video Card: MSI GTX 1060 GAMING X 6G
Power Supply: SeaSonic PRIME Platinum 1000W 80+ Platinum
Other Bits: Fan replacements, extra fan controller if needed
Usage Profile: SQL Server/IIS, load testing, performance tuning, high availability scenario testing

Other information…
{Expletive adjective} {expletive verb} {expletive adjective} RAM prices!!!!
I'd really prefer to go with the 64Gb LRDIMMs and get the headroom up to 1Tb, but I'm having a hard time with the 50% premium over the 32Gb sticks. Even so, I'll be buying the RAM in 128Gb stages over several months as I can afford more - I can mess with shared memory to the failover SQL Server VM to get some useful work done at lower RAM totals.

I'm somewhat concerned about getting ES/QS CPUs, but the price/performance ratio makes it worth the risk to me. Once I get closer to buying, I'll post details on the ES thread here and make sure that I improve my chances of getting a working pair.

Minor bits:
- Fractal Design support confirmed that the XL case works fine with an SSI EEB mobo, just need a couple of standoffs. I also like the look of the Phanteks Enthoo Pro, which looks like it can be just as quiet if I add some acoustifoam.
- The video card is for my wife's machine learning, will have to figure out how to pass it through to a VM

Now to save a little more money and do my research on CPU/GPU fans total CFM vs case fans CFM, verify the power supply needs, and remember that it isn't a laptop so I'll need a UPS if I care about graceful shutdowns.

Thoughts?