What h/w for a VM server?

[GaryD9]

I'm a new user on this forum... after reading many posts here, I decided that I might get the best advice here.

In my home I currently have an old desktop machine (i7-2600k / 16GB RAM / 250GB SSD / 2x 1TB spinners / 1 onboard 1GbE NIC for the host and 2x 1GbE NICs teamed for use by the VM's) running Windows 2012R2 Hyper-V (with the GUI installed on the host OS.) Not part of the same machine, but part of it's working environment, I only have 1GbE switches, and a NAS that the VM's access via iSCSI.

As VM's, I have an AD machine (using server 2012R2 standard + essentials role -- does backups of all other machines, etc), a Win7 VM, a Win10 VM, and 2 linux VM's. On occassion, I'll spin up other VM's for testing various software. (VM's make software development testing easier.) The linux machines both run various daemons, and 1 of them runs the java-based controller for my ubiquiti APs. I'd like to claim that that is all I'd use the box for, but there's a very good chance that I'll find some excuse to have other VM's running for various things.

I've decided that it's time to replace this machine, and that I'd like to buy the components knowing that they'll be used for VM hosting (instead of just recycling of an old desktop machine.) However, I'm married with children, so budget is a large concern. I'd also love to get the power/heat/noise and physical size down from the existing server (which is in a full tower case.) One of those mini ITX cases looks very appealing.

My problem is that I'm not sure if I'd be better off with a Xeon D-1528 board (would that be a good Hyper-V host machine?) or if an E3-xxxxV5 based solution (with higher power usage, but also seemingly better performance per dollar) would be better. Or, perhaps I should just get another desktop processor (low power i7-6700T?)

As far as budget, it kind of depends. My original intention (with using a D-1528) was that I could take 16 GB of non-ECC DDR4 from another machine until I was able to buy proper ECC RDIMM's for it (and then put in 2x 16GB sticks.) That wouldn't be an option with the E3-xxxxV5 (meaning I'd have to purchase RAM initially.) A desktop processor would, of course, use ONLY non-ECC. As far as persistent storage, I'd have to re-use what I already have in the short term, with the intention of replacing drives over time. The same goes for the chassis and p/s. I'd reuse initially and replace eventually.

My initial budget (motherboard + processor and RAM if I can't re-use DDR4 non-ECC) is around 650-700 (USD.) I'd strongly prefer to purchase new instead of trying to recycle another machine.

All that being said... what are reasonable options for me, and what would be best for my usage scenario?

Thank you
Gary

 

[Deslok]

Do you need the gpu in the E3? if not the xeon-D is likely a better fit, VM hosts tend to idle more than not so the extra GHZ the E3 offers is worth less than having the extra cores you can pick up with a xeon-d in the same power budget. And yes hyper-V is going to run on all of that.

 

[GaryD9]

Do you need the gpu in the E3?

All the E3's have GPU's? I thought only some of them did.... E3-1220V5 (no GPU) vs E3-1225v5 (yes GPU)

One thing that I forgot to mention (that the GPU made me think about) is that I'd really like to have IPMI. I think IPMI is an option for any of the processors, though (based on the motherboard selection.)

 

[Deslok]

No they don't all have gpu's but if you were going to use it for VDI it can be a nice feature and is the big reason to look at E3-15xx chips. One thing to look at is your cpu load percentage on your 2700 you have currently, I have a 3770 at home that's mostly idle if i was building from scratch I'd go for more cores(xeon-d or avoton) over GHZ because of that. Same thing with work actually, our 55xx xeons are rarely above 30% cpu load but we need more memory not faster cores.

 

[fractal]

I did not believe "them" at first when "they" said to check the VMWare approved hardware list. That is until I ran into little tiny gotchas. Nothing serious but enough to fill log files with issues and make me question the reliability of the build. Now, windows is not quite as picky as vmware about the hardware but it is worth checking the list for it..

I have noticed that conventional wisdom doesn't seem to fit my home VM usage. The "more slower cores" works great for web servers. It kinda sucks when you are doing software development in a VM. The cores rarely wiggle off of zero but when they do, fewer but faster is a winner. Server apps are designed for multiple threads. Desktop apps not so much.

The guidance I would give is to look at your current memory usage and double it. If that fits in 32G you can go with an older E3. If that fits in 64G then you can go with a newer E3. If not, you are looking at something bigger.

I find you trade off heat / noise / space / effort. It is not easy to get a quiet, efficient mini-itx at a budget. A standard mini tower with a micro atx board is a *LOT* easier to keep quiet on a budget.

I think Deslok was asking if your workload can make use of the GPU in the E3-xxx5 processors using virtualization passthrough or something like that. It is a corner case but worth considering if you are in that corner. Conventional guidance is to avoid the xxx5 e3's when using a server board just to save the power and to reduce the rare chance of confusion.

So, for some real advice -- IF you can fit in 32GB, consider a teaser box. Dell, Lenovo, HP all have 200 dollar entry level servers / workstations that will take 32GB of ECC memory but come with 4GB. You can get a box that will take 64GB if you up your budget to 500 dollar (with 4-8). The 200 dollar box will take 2-4 HDD/SSD's and chew a whopping 20 watts idle and will be quiet enough to sleep next to. The HP tends to make a bunch of noise when you power it up but then quiets down.

 

[GaryD9]

I'm not even sure how to check the actual total processor usage on hyper-v. Task Manager (on the host OS) only shows the usage for the host OS (and not any of the VM's.)

... It kinda sucks when you are doing software development in a VM. The cores rarely wiggle off of zero but when they do, fewer but faster is a winner. Server apps are designed for multiple threads. Desktop apps not so much.

I won't be compiling, etc, on the VM. (I use my own desktop for that.) I'll create a few different "known state" VM snapshots, and then test against those. Getting back to the "known state" is then trivial (and a massive time saver.)

The guidance I would give is to look at your current memory usage and double it. If that fits in 32G you can go with an older E3. If that fits in 64G then you can go with a newer E3. If not, you are looking at something bigger.

That's easy. I'm currently using 16GB and that hyper-v machine rarely uses any swap. I plan on 32GB within a few months, but the ability to "grow" into 64GB would be nice. I honestly don't expect to ever go beyond 64GB. (That was my reasoning for 2x 16GB sticks... it'd give me 2 slots more.)

You can get a box that will take 64GB if you up your budget to 500 dollar (with 4-8). The 200 dollar box will take 2-4 HDD/SSD's and chew a whopping 20 watts idle and will be quiet enough to sleep next to. The HP tends to make a bunch of noise when you power it up but then quiets down.

Do you happen to have any links to a 4/8 for $500? I took a look at Dell's website, and the cheapest I found any of their machines with a 4/8 E3 (E3-1230v5) was around $900. hpe.com and lenovo.com don't even seem to list any prices. (I guess I'm looking in the wrong places... where should I be looking?)

Thank you
Gary

 

[Evan]

One small item as well may be memory cost (not having checked in the last weeks) the Xeon-D & E5 user rdimm, the E3 used udimm.

Rdimm is often found stripped from servers (even ddr4 as base memory is often removed for bigger modules) on eBay etc for much cheaper and more common.

 

[Evan]

So, for some real advice -- IF you can fit in 32GB, consider a teaser box. Dell, Lenovo, HP all have 200 dollar entry level servers / workstations that will take 32GB of ECC memory but come with 4GB. You can get a box that will take 64GB if you up your budget to 500 dollar (with 4-8). The 200 dollar box will take 2-4 HDD/SSD's and chew a whopping 20 watts idle and will be quiet enough to sleep next to. The HP tends to make a bunch of noise when you power it up but then quiets down.

Very sound advice as well if your needs are simple

 

[fractal]

Do you happen to have any links to a 4/8 for $500? I took a look at Dell's website, and the cheapest I found any of their machines with a 4/8 E3 (E3-1230v5) was around $900. hpe.com and lenovo.com don't even seem to list any prices. (I guess I'm looking in the wrong places... where should I be looking?)

There's the HP ML10 with 4G, 64G max for 299 at HPE ProLiant ML10 Gen9 G4400 4GB-R Non-hot Plug 4LFF SATA 300W Entry Svr . Upgrading to 64G will hit you back at least that much. The Lenovo TS150 is new so pricey but bargains can be found on the TS140 if you can get by with 32GB. You can find bargains on the Dell T20 too if you can get by with 32GB. I don't know the Dell line well enough to know what replaces it.

 

[mstone]

No they don't all have gpu's but if you were going to use it for VDI it can be a nice feature and is the big reason to look at E3-15xx chips. One thing to look at is your cpu load percentage on your 2700 you have currently, I have a 3770 at home that's mostly idle if i was building from scratch I'd go for more cores(xeon-d or avoton) over GHZ because of that.

I'm confused about why you'd want more idle cores instead of fewer idle cores that can ramp up more for infrequent cpu-intensive tasks. The use case for more cores is when you have a substantial amount of work that can be parallelized easily across many utilized cores which in aggregate perform more work than a smaller number of individually faster cores.

Case study: If you have a small number (say 2) of fast cores and you ramp one to handle a login or something, you have 50% system utilization. If you have a larger number (say 16) of slow cores and you ramp one up to handle a login, you have 6% utilization, and the cpu-intensive task takes longer. In my experience, a usually-idle system will rarely pop up a well-parallelized cpu-intensive task, but will often pop up small, no-scalable/single-threaded tasks. With a mostly idle system consisting of many slow cores you will basically never utilize much of the systems' capacity.

 

[GaryD9]

I still can't find any 4/8 processor based "entry" systems for 500 USD. The link was for a 2 core pentium... For anything 4/8, it still looks like building it myself will be cheaper.

I can see the reasoning for the 4core at higher speed vs 6core at lower speed, but I'm not sure which would be better in my particular case. 80% of the time, those VM's are all pretty much idle. The only times I can think of (off the top of my head) that I'd had "slowness" issues with them would be I/O contention (such as one of the Win7 or Win10 machines doing MS updates while the server VM was doing a backup) and when I quickly create a new VM and forget to raise the number of virtual processors from the default of one.

Thinking about it, and trying to express this in a way that makes sense: The VM's don't have to run fast, as long as they don't run slow. Said another way: I don't need a Porsche, but I don't want a moped either. Taking the analogy further - as long as I agree to only drive the speed limit, I'd like to save some gas along the way and have something that's small enough to park in the shopping mall parking spots.

(I guess I'm not really very easy to help out when I'm not even sure what my needs are. Sorry...)

 

[Deslok]

I'm confused about why you'd want more idle cores instead of fewer idle cores that can ramp up more for infrequent cpu-intensive tasks. The use case for more cores is when you have a substantial amount of work that can be parallelized easily across many utilized cores which in aggregate perform more work than a smaller number of individually faster cores.

Case study: If you have a small number (say 2) of fast cores and you ramp one to handle a login or something, you have 50% system utilization. If you have a larger number (say 16) of slow cores and you ramp one up to handle a login, you have 6% utilization, and the cpu-intensive task takes longer. In my experience, a usually-idle system will rarely pop up a well-parallelized cpu-intensive task, but will often pop up small, no-scalable/single-threaded tasks. With a mostly idle system consisting of many slow cores you will basically never utilize much of the systems' capacity.

I don't have a substantial amount of load but I do have a substantial amount of systems, per core performance is less important in this case than keeping the systems from stepping on each other's toes where per core over commitment can become an issue.

 

[fractal]

I still can't find any 4/8 processor based "entry" systems for 500 USD. The link was for a 2 core pentium... For anything 4/8, it still looks like building it myself will be cheaper.

How about splitting the difference and mod an older system? You won't get quite the power efficiency of an i3, but ..

I was running out of RAM on a ESXi server I am using to test automatic deployment of cloud instances and the price of upgrading the DDR4 ECC memory was a sticker shock. I solved the short term need by buying 128GB of DDR3 registered ECC and sticking it in an HP Z420. The Z420 cost me 150 bucks. The RAM cost 500 but that was because 16GB sticks are expensive. Your target of 32 expandable to 64 means you can stick with the more economical 8GB sticks.

The path I choose is not without its challenges. The manual for the Z420 says unbuffered ECC only but the 2.x BIOS upgrade needed to support E5 v2 processors unofficially enables the use of registered ECC. This may be a non-starter for some people.

If you are willing to consider this, you can get a Z420 with a 4 core/8t processor and a bit of memory for 250 on ebay without really trying. 32gb reg-ecc ddr3 can be had for 50-80. There's an auction for 2x16 samsung currently at 50 which is a great buy.

The barebones I bought came with a 4c/8t workstation processor. I replaced it with a 8c/16t server processor and am regretting it. The 2c/4t i3 in the other room feels "snappier" to me. I typically float around 10-20 vm's all with 4c allocated but my cpu load rarely leaves zero. They are all idle. So even with 80 cores assigned, none are used and having a single 3.5 GHz core ready and waiting for my single-core app gets the job done faster than having 15, 2.0 GHz cores lounging around.

As always, YMMV. My workload may have no resemblance to yours.

 

[GaryD9]

So... what I ended up doing was the i7 route. An i7-7700 (non-K) has a TDP of only 65 watts (which is considerably lower than the 95 watts of the i7-2600k.) It gives me 4 cores, 8 threads, 3.6GHz with 4.0GHz "turbo" (that is active with all cores going and never seems to back down.) Single core turbo is something like 4.2GH. Along with 2x16GB DDR4 memory (instead of 4x4GB DDR3 memory), I'm saving a few more watts and gaining some speed.

No, it's not a proper server board. I don't get ECC memory, and no IPMI. However, I saved about $400 overall over the cost of a new E3 board/processor/RAM...

(And now that I've admitted to using a desktop processor as a server, I imagine I'll get banned from this forum. )

Take care
Gary