1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

512Gb RAM Apartment Server?

Discussion in 'DIY Server and Workstation Builds' started by Otto, Mar 19, 2017.

  1. Otto

    Otto New Member

    Joined:
    Mar 18, 2017
    Messages:
    7
    Likes Received:
    0
    Edit: Updated Build 3/26/2017

    Thanks for all the helpful advice - this is shaping up nicely!

    Build’s Name: Big RAM - Small Apartment
    Operating System/ Storage Platform: Windows Server 2016 DC, Hyper-V VMs
    CPU: 2x Xeon E5-2683 v3 ES/QS 2GHz 120W ($350 on ebay)
    CPU Cooler: 2x Noctua NH-U14S 55.0 CFM CPU Cooler
    Motherboard: ASRock EP2C612D16C-4L
    Chassis: Fractal Design Define XL R2
    Drives: 2x Samsung 850 EVO 2Tb SSD (boot and VMs)
    2x WD Red 4Tb 5400 (mirrored, backups)
    RAM: 16x Crucial 1x32Gb RDIMM 288-pin DDR4/2133
    Video Card: MSI GTX 1060 GAMING X 6G
    Power Supply: SeaSonic PRIME Platinum 1000W 80+ Platinum
    Other Bits: Fan replacements, extra fan controller if needed
    Usage Profile: SQL Server/IIS, load testing, performance tuning, high availability scenario testing

    Other information…
    {Expletive adjective} {expletive verb} {expletive adjective} RAM prices!!!! :(
    I'd really prefer to go with the 64Gb LRDIMMs and get the headroom up to 1Tb, but I'm having a hard time with the 50% premium over the 32Gb sticks. Even so, I'll be buying the RAM in 128Gb stages over several months as I can afford more - I can mess with shared memory to the failover SQL Server VM to get some useful work done at lower RAM totals.

    I'm somewhat concerned about getting ES/QS CPUs, but the price/performance ratio makes it worth the risk to me. Once I get closer to buying, I'll post details on the ES thread here and make sure that I improve my chances of getting a working pair.

    Minor bits:
    - Fractal Design support confirmed that the XL case works fine with an SSI EEB mobo, just need a couple of standoffs. I also like the look of the Phanteks Enthoo Pro, which looks like it can be just as quiet if I add some acoustifoam.
    - The video card is for my wife's machine learning, will have to figure out how to pass it through to a VM

    Now to save a little more money and do my research on CPU/GPU fans total CFM vs case fans CFM, verify the power supply needs, and remember that it isn't a laptop so I'll need a UPS if I care about graceful shutdowns.

    Thoughts?

    ------------------------
    Original post:
    Help! This is my first build in a decade, since we moved from a large house to a small apartment and I switched to a laptop-only world. I haven't kept up with server hardware much in the interim, so I'm a bit at sea.

    Build’s Name:
    Big RAM - Small Apartment
    Operating System/ Storage Platform: Windows Server 2016 DC, Hyper-V VMs
    CPU: 1 or 2 Xeon, 20-24 cores total
    Motherboard: ASRock SSI EEB? (dual), X99? (single)
    Chassis: ???
    Drives: ???
    RAM: 512Gb
    Add-in Cards: None
    Power Supply: ???
    Other Bits:

    Usage Profile: SQL Server/IIS, load testing, performance tuning, high availability scenario testing

    Other information… I’ve been using my prior laptop as a home test bed for software development, but the venerable m18-x has finally given up the ghost. I live in a small NYC apartment and the laptop was been nice and quiet, but was also severely memory limited (32Gb).

    I really need something that can do useful testing for a production system that runs on 4x 12-core (dual 6C Xeon E5 2640s)/128Gb RAM with 10GpE. Our DR environment runs fine on half the CPUs so I don’t need 48 cores, but there’s no getting around the memory requirements.

    Noise is of some concern, since this will sit in a corner of the bedroom, but it also won’t being doing much while I’m asleep, so quiet at idle and under light workload is probably what I’m shooting for.

    This is just the beginning so I have a bunch of questions:
    1. Single, hi-core count CPU or dual with lower core counts each? The dual approach would give me some space for processing upgrade down the road if needed.
    2. How important are the newer Xeon revs (v3/v4)?
    3. 8x64Gb or 16x32Gb for the RAM? The 32’s leave room for a future upgrade, but is it useful?
    4. Large case with room for big slow fans, or a denser arrangement? Which is better for noise control and cooling? I’ve got room for a full tower, but smaller is nicer.
    5. Do I need NVMe or will SATA SSDs work? I’m mimicking 10GpE iSCSI to a 10K SAS spinning disk SAN so even a slow SSD has 2 orders of magnitude better iops, but I’m very interested in database tuning and don’t want to find myself bottlenecked on the I/O.
    6. I don’t currently have any monitor in the apartment – is it possible to do the setup and install with the server headless?
     
    #1
    Last edited: Mar 26, 2017
  2. i386

    i386 Active Member

    Joined:
    Mar 18, 2016
    Messages:
    529
    Likes Received:
    94
    1.) If you don't many pcie lanes then go single cpu (40 pcie lanes)
    2.) (usually) Newer cpus = same or better performance with less power, but higher price
    3.) if you go the single socket route you will be limited by 8 slots, with lr-dimm you can still get 512 gb ram (8x 64gb)
    4.) I know that supermicro has some 1u servers with 20+ cores and 512gb ram, but they are loud and you don't want to be in a room with them. Get the largest chassis you can get. Maybe some of the supermicro 4u/twoer, like the sc745 which supports rackmount/how swap psus (eg 920sq).
    5.) that depends how much money you have left after getting 512gb ram :D
    6) yes, if the mainboard/server supports ikvm (ipmi like on intel mainboards without the remote control board is not neough)
     
    #2
  3. Aestr

    Aestr Active Member

    Joined:
    Oct 22, 2014
    Messages:
    314
    Likes Received:
    85
    I'm in a very similar boat about downsizing. I've ordered one of the SuperQuiet SC745's and while I'll be going single socket to start and then gauge my needs from there I think the noise and space constraints are similar. I'll let you know what I think of it.
     
    #3
  4. Evan

    Evan Well-Known Member

    Joined:
    Jan 6, 2016
    Messages:
    1,048
    Likes Received:
    137
    DDR4 using 16x32gb dimm, that's regular dimm, not load reduced that would be needed for ddr3. Also the ram will still run at 2133/2400 compared to probably 1300 with that many LR DDR3 dimms in a system.

    In a bedroom, keeping this sleeping quiet is going to require some careful thought ! But certainly possible.
    I watch with interest :)

    As an aside did you think to split it up into a couple of smaller systems and only keep one running 24x7 if needed and start the others when you want to load test ? Maybe a nice single e5 or Xeon-D for 24x7 and some older Gera that can run for load testing when needed but not to silent ?
     
    #4
  5. Jeggs101

    Jeggs101 Well-Known Member

    Joined:
    Dec 29, 2010
    Messages:
    1,177
    Likes Received:
    142
    @Otto I think you're going to fit in well at STH. Welcome!

    What about one of those SM towers?
     
    #5
  6. K D

    K D Active Member

    Joined:
    Dec 24, 2016
    Messages:
    455
    Likes Received:
    104
    From what I understand, this is going to be a stand-alone box that is going to be tucked into a corner of a bedroom apartment. If you dont want the bells and whistles that SM chassis's offer, then look at a consumer tower chassis like Fractal Define R5 or the bigger XL. I've used the R5 with an ASUS Z8 dual CPU board with dual E3-1245 and 5-6 HDDs. The case is quiet, has good thermals and is highly configurable and the newer revisions are even better than the one I used.

    I have not used an SM tower workstation but have heard good things about them. But I don't know if they can be quietened to the level a consumer case like Fractal Design can be.
     
    #6
  7. Diavuno

    Diavuno Active Member

    Joined:
    Jan 31, 2014
    Messages:
    808
    Likes Received:
    100
    I meant it is my older, but I have a dual X8 system in a NZXT case. The computer is almost completely inaudible. I've had it loaded up to 200 gigs of RAM give or take a few... You will need to go supermicro if you want to use an SSI board e-atx you can normally find a full tower. Cooling the tips that can be an issue, but can be done.

    Sent from my 0PJA2 using Tapatalk
     
    #7
  8. Otto

    Otto New Member

    Joined:
    Mar 18, 2017
    Messages:
    7
    Likes Received:
    0
    Holy information overload, batman - you folks rock!

    v4s will be lower max TDP at the same core count & speed
    big cases are easier to quiet/cool
    stick to 16-slot RAM since LRDIMM is massively slower (dang, I had no idea!)
    the Fractal Design cases are gorgeous, but limited to ATX mobos
    the SM cases look kinda terrifying, but that that will be the route for an SSI mobo
    ikvm will let me do headless setup (yea!)

    I'm leaning towards dual 10-12C since the 20C & 22C draw a ton of power (135-145W!), and my instinct is that it will be easier to cool a pair of lower power chips.

    Looks like I've got a bunch of research to do (the fun kind <grin>)

    Huge thanks to all!
    -Otto
     
    #8
  9. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    9,401
    Likes Received:
    2,908
    @Otto - I was going to put this SM one on Craigslist Q1 2017 STH Lab Cleaning

    No RAM for you (sadly that is tight at the moment) but I may be able to help with CPUs as well.
     
    #9
  10. Logan

    Logan Member

    Joined:
    Feb 22, 2017
    Messages:
    56
    Likes Received:
    7
    Adding to the quiet case recommendations of Fractal Design and NZXT, I have been very pleased with Nanoxia.
     
    #10
  11. zer0sum

    zer0sum Active Member

    Joined:
    Mar 8, 2013
    Messages:
    139
    Likes Received:
    27
    You can actually fit an E-ATX/SSI board into a lot smaller case than you think.
    The usual issue is just that the standoffs are in the wrong places, but you can fix that with a drill and a tap in minutes :)
    Or you could go with the Supermicro SC732 which is small and has the right standoffs
     
    #11
    Patrick likes this.
  12. Otto

    Otto New Member

    Joined:
    Mar 18, 2017
    Messages:
    7
    Likes Received:
    0
    Cool, more to process!

    @Patrick - That is a sweet price for the SM tower! I'm nowhere near ready to pull the trigger yet, but if you've still got it in a few weeks I'll be really tempted - where is "local pickup" for you? For the CPUs, I've been watching the ES models on ebay (those prices are crazy low), and I'd love to hear what you can hook me up with.

    @Evan - I considered splitting the system into a couple of smaller boxes, but there are two issues that I've run into. First is the networking - I'd need to put 10GpE in place since 1G won't cut it between DB, App and Cache machines, and I don't really want more cabling in the corner. Second is my significant other - she's limited me to a single tower <grin>. I even flirted with doing this all in the cloud; however, MSDN licenses can't be used for cloud instances, so the hourly rate gets out of control really quickly.

    @zer0sum, @K D - Fitting an SSI EEB into a big, quiet consumer case is really starting to appeal to me. They look great, and I won't be chewing up my fingers trying to make things fit in small spaces.

    I'll update the build spec next weekend after I've poked about some more and see where things stand.

    Thanks everyone!
     
    #12
    Last edited: Mar 21, 2017
  13. Klee

    Klee Active Member

    Joined:
    Jun 2, 2016
    Messages:
    439
    Likes Received:
    117
    Here is my first AsrockRack dual xeon SSI-EEB sized motherboard in a Phanteks Enthoo pro case.

    [​IMG]

    Also you can run a single cpu in a dual cpu motherboard and that gives you the option of purchasing another identical cpu plus ram at a later time and they will be cheaper, possibly a lot cheaper and that will offset the initial price of the dual cpu motherboard over a single cpu motherboard.
     
    #13
    Marsh likes this.
  14. Otto

    Otto New Member

    Joined:
    Mar 18, 2017
    Messages:
    7
    Likes Received:
    0
    @Klee That's a beautiful setup! How does the Phanteks do at noise control under load?
     
    #14
  15. TLN

    TLN Member

    Joined:
    Feb 26, 2016
    Messages:
    200
    Likes Received:
    18
    1. Xeon E5-v3.
    2. Because they have all-core-tubro bug and that's useful ;)
    3. Dual CPU, in case you need it.
    4. Bigger case, better coolers - this all will keep your system cool and silent.
    5. NVMe or SSD is always good thing. No need to get the fastest though.
    6. With proper motherboard you can get to BIOS remotely. No need in monitor.
     
    #15
  16. nk215

    nk215 Active Member

    Joined:
    Oct 6, 2015
    Messages:
    212
    Likes Received:
    52
    Unless power is free or very low cost, you should ask yourself if you need everything to be on 24/7.

    If you don't, get 2 machines. First one is a E3 variant (they are in the 40-50 watts idle and virtually silent). Second one is a dual CPU E5 that doesn't need to be on when you sleep. This cuts down the noise and power cost while you sleep.

    In all practical purposes, for home server with a hand full of users, you can't tell a different between NVMe or SATA SSD. It comes down to how much you dislike cables and cooling setup. VNMe especially the M2 with PCIe adapter (950 pro for example) run hotter than SATA SSD. Many PCI NVMe cards expect some airflow.

    A dual E5 can be very silent with tower cooling or water cooling system with mostly solid state SATA SSD. If you do water cool, you don't need any case fan. Radiator fan pulling air out is enough (I do use a small 40mm fan on the chipset heat sink however).
     
    #16
  17. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    1,050
    Likes Received:
    108
    There is also an ATX sized Dual CPU Board btw if you go the split way (Supermicro | Products | Motherboards | Xeon® Boards | X10DRL-i), it's only got 8 DIMM Slots so not really for your AIO with 512GB unless you want to spend a ton of money on Ram (64GB modules). But it would allow you use two quite small boxes, eg with diverse CPU setups (many cores/high frequency)
     
    #17
  18. Klee

    Klee Active Member

    Joined:
    Jun 2, 2016
    Messages:
    439
    Likes Received:
    117
    I added an additional 200mm fan at the top rear of the case and its hardly noticeable to me, its to the right of me under my desk.

    The case is very flexible when it comes to cooling options.

    I have upgraded the motherboard to a newer socket 2011-3 motherboard but its also an AsrockRack SSI-EEB sized motherboard.
     
    #18
  19. Otto

    Otto New Member

    Joined:
    Mar 18, 2017
    Messages:
    7
    Likes Received:
    0
    @nk215 @Rand__ Splitting into multiple boxes isn't really an option - partly because primary and failover DB need 128Gb each and 10GpE network to shared storage, so that takes me to 2 boxes + network and cables, and partly because it is a really small apartment and I don't have the room :) I probably will shut things down at night if the idle noise/power consumption is too high - other than an occasional long stress test, I don't imagine much overnight need.

    @Klee Thanks! Glad to hear the noise levels are good in the Phanteks. This is definitely in consideration.

    @TLN For NVMe vs SATA SSD, I'm seriously spoiled by the 950 Pro in my main laptop, but you are right that the performance difference is irrelevant to this server - EVO 850's it is...
     
    #19
  20. Otto

    Otto New Member

    Joined:
    Mar 18, 2017
    Messages:
    7
    Likes Received:
    0
    Updated Build 3/26/2017
    (also edited first post)

    Thanks for all the helpful advice - this is shaping up nicely!

    Build’s Name: Big RAM - Small Apartment
    Operating System/ Storage Platform: Windows Server 2016 DC, Hyper-V VMs
    CPU: 2x Xeon E5-2683 v3 ES/QS 2GHz 120W ($350 on ebay)
    CPU Cooler: 2x Noctua NH-U14S 55.0 CFM CPU Cooler
    Motherboard: ASRock EP2C612D16C-4L
    Chassis: Fractal Design Define XL R2
    Drives: 2x Samsung 850 EVO 2Tb SSD (boot and VMs)
    2x WD Red 4Tb 5400 (mirrored, backups)
    RAM: 16x Crucial 1x32Gb RDIMM 288-pin DDR4/2133
    Video Card: MSI GTX 1060 GAMING X 6G
    Power Supply: SeaSonic PRIME Platinum 1000W 80+ Platinum
    Other Bits: Fan replacements, extra fan controller if needed
    Usage Profile: SQL Server/IIS, load testing, performance tuning, high availability scenario testing

    Other information…
    {Expletive adjective} {expletive verb} {expletive adjective} RAM prices!!!! :(
    I'd really prefer to go with the 64Gb LRDIMMs and get the headroom up to 1Tb, but I'm having a hard time with the 50% premium over the 32Gb sticks. Even so, I'll be buying the RAM in 128Gb stages over several months as I can afford more - I can mess with shared memory to the failover SQL Server VM to get some useful work done at lower RAM totals.

    I'm somewhat concerned about getting ES/QS CPUs, but the price/performance ratio makes it worth the risk to me. Once I get closer to buying, I'll post details on the ES thread here and make sure that I improve my chances of getting a working pair.

    Minor bits:
    - Fractal Design support confirmed that the XL case works fine with an SSI EEB mobo, just need a couple of standoffs. I also like the look of the Phanteks Enthoo Pro, which looks like it can be just as quiet if I add some acoustifoam.
    - The video card is for my wife's machine learning, will have to figure out how to pass it through to a VM

    Now to save a little more money and do my research on CPU/GPU fans total CFM vs case fans CFM, verify the power supply needs, and remember that it isn't a laptop so I'll need a UPS if I care about graceful shutdowns.

    Thoughts?
     
    #20
Similar Threads: 512Gb Apartment
Forum Title Date
DIY Server and Workstation Builds Apartment Living Server Mar 24, 2017

Share This Page