New Board and CPU for ESXi Homeserver, which socket?

[m4ntic0r]

Hello community,

At the moment i have the following server which runs 24/7 with windows 10pro and vm workstation 12.5.7 running over 20 vms:

2600k
32GB RAM
Asus Z77 V Gene
LSI 9271-8i with Cachevault and CacheCade
4x4TB HGST Megascale Raid 10
2x500GB Samsung Evo 500GB Raid1

On a second mini system (G1620, DQ77KB, 8GB DDR3, 320GB) i installed ESXi 6.5U1 to play with. There i will put my important vms like dc´s and dns servers for the migration time of the main host.

Now i want and have to upgrade my main machine to server hardware with the option to go for 64-128GB RAM, but i dont know which platform i should choose. I want to use ESXi 6.5 on the new platform. Power consumption is a important factor. With my current system i see ~90w on my apc usv. with cpu load this go up to 120-150w. Daily usage is 2,3-2,4kw/h on apc so around 95-100w.

i see four options for me (board + cpu ~ 1000€)
for ram i want to start with 2x32gb dimms ddr4 regecc 2400mhz from samsung
i want to reuse my pcie3 x8 lsi 9271-8i raid controller with hdds.

socket 2011-3: Supermicro X10SRi-F + Intel Xeon E5-2630 v4, 10x 2.20GHz, boxed
socket 3647: Supermicro X11SPi-TF + Intel Xeon Silver 4110, 8x 2.10GHz
atom c3000: Supermicro A2SDV-16C-TLN5F + Atom Processor C3958 16 Core
xeon d: Supermicro X10SDV-TLN4F + Intel Xeon D-1541, 8x 2.10GHz, 12MB Cache

i dont know what i should do. how is power consumption compared to my 2600k system?. somehow i love the specs from the atom board (16 cores + 128gbram + 31w tdp), but i dont know if performance would be sufficient.

 

[Rand__]

What are you doing with the VMs? Any direct User VMs? i.e. do you run desktops on the box for daily usage?
If so then clockspeed will matter - your current box has 3.4 Ghz (with 4c/8t, so 13.6 Ghz total compute power).
My experience says that you need about 2.4 Ghz for decent desktop performance - o/c if you don't run desktops then that's less urgent, then higher speed might only be needed for single threaded or user interacting tasks.

Also check your long time utilization stats regarding CPU and memory - if you're 100% all the time you need to increase compute power, if you're at 60% you don't need to and can focus on power savings. Same for memory. O/c also take future plans into consideration

From an efficiency level it probably will be (top least efficient but still quite good):
2011-3
3647
xeon- d
c3000

Again, from my tests - the actual CPU choice matter less and less on total efficiency as most are clocking down quite good in idle.
Actual workload, secondary components (board/platform efficiency, hba's, fans, psu efficiency, psu sizing especially) play a significant role nowadays

 

[m4ntic0r]

Hi Rand_, thanks for your answer.

There are no direct user vm´s. The only direct user vm is one Windows 10 VM for my own remote desktop from outside. But this is only for manage the homelab from outside if needed.

The rest are 5x Server 2016 VM (2x dc,dns,dhcp) (file server) (downloader) (exchange 2016)
and a lot of Debian 9 and Ubuntu VMs (nginx reverse proxy, mysql, dyndns, tinyrss, nextcloud, chevereto, bookstack, plex, piwik, monitoring, webcollector, solr, archivteam warriors, two piholes etc.)

The load on my cpu looks like that: The most cpu intensive tasks are plex transcoding and sometimes the traffic up to 100mbit over a vpn connection. the full text index for my nextcloud via solr or thumb generation takes some cpu too.

Most of my linux vms have 1 vcpu (only plex has 2 and nextcloud has 2). this is more than sufficient until today.

My memory is my main limiting factor, usage always >95% so start with 64gb should be fine. But i want the option to 128gb for future vms.

Raid Controller, the HDDs/SSDs and my tower with 3x Cremax Icy Dock MB153SP-B HotSwap with new silent fans will stay. There is one M4 512GB SSD (for the host) and one 2TB 2,5" HDD (for veeam backup target) in the tower too, but i dont know what to do with these. I will run ESXI from one SanDisk Ultra Fit V2 16GB USB stick.

 

[Rand__]

So if the outer shell remains the same and you are only looking to upgrade mainboard/cpu I guess it comes down to connectivity/expandability requirements.
Quick table:


x10sri is the exotic version here without builtin 10G - depends on whether you need/want that

C3958 will have most compute power (that you don't need). Both embeded boards are smaller and thus have less pcie slots and memory slots, means you need more expensive ram modules to fill up to 128.

If you need more than one HBA at a later time then you'll be having issues with those too. Especially on the c3958 with only 2 sata ports.

The regular xeon boards are way more flexible, albeit larger and more power hungry.


So in the end as always - totally depends on what you need

 

[m4ntic0r]

I think i will decide between xeon- d and c3000.

- my raid controller with 8 ports will be enough. so pcie3x8 is a must, but the x16 on xeon d might be more future proof.
- 10g is nice to have for future, 2x 10g is more than enough, and there are 2x1g on the xeon d, too (all working with esxi 6.5 oob)
- 2xsata vs. 6x sata onboard for the xeon d
- single thread performance better on xeon d?
- smaller size

a lot of things for the xeon d board.

i saw that there is a 4-pin power connector on this board and the manual says: The 4-pin HDD power connector J6 provides power to onboard HDD devices. So if i dont use the onboard sata and m2 slot i dont need this connector?

 

[Rand__]

- single thread performance better on xeon d?

The recent review of the c3955 implies so
https://www.servethehome.com/wp-con...Atom-C3955-Linux-Kernel-Compile-Benchmark.jpg

The 3955 (higher clock than your choice) is a little better with 16 cores than the 1541 with 8c/16T.

i saw that there is a 4-pin power connector on this board and the manual says: The 4-pin HDD power connector J6 provides power to onboard HDD devices. So if i dont use the onboard sata and m2 slot i dont need this connector?

Never used it - I think its for embedded situations only

 

[m4ntic0r]

Ok i think i will go for the Supermicro X10SDV-TLN4F. The specs are good and the c3000 not in stock and nobody will know when this will happen here in germany.

Supermicro X10SDV-TLN4F + 2x32GB Samsung will cost ~1600€, but its a very nice, powerfull and efficient ESXi platform @home for the next 5+ years?

For the power supply i have a new and never used Enermax Revolution X't II 450W ATX 2.4 80+Gold. Bought this two weeks ago because i dont want these dangerous 4pin to sata adapters anymore which i am actually using with my 300w bequiet.

I will be back when hardware is here and hopefully everything is running as expected.

 

[Rand__]

I'd probably go for SFP+ and or 2 PCie Slots / onboard SAS but I think those don't come in 1541 flavour.

Have fun with the build

 

[Evan]

I anyway would probably choose 8 proper cores (Xeon-d) over 16 simple cores (c3000), per core speed and compatibility with the rest of Intel line in terms of AVX2.

 

[i386]

I would go the E5/Xeon Silver route. The other options don't have many pcie lanes. (The atom mainboard in the op can use only one of the x8 slots!)

 

[Evan]

Or choose e5/silver if you need expansion as @i386 says, or more cores/memory.

Either way the c3000 I would not use for general purpose virtulisation. Storage and FW type duty sure it's got some advantages but otherwise I don't think it fits your needs.

 

[m4ntic0r]

I dont need more than one PCIe Slot for a Raid Controller or HBA. On my LSI 9271-8i i can use every sized HDD/SSD and especially the SSDs will get bigger in future. No need for other expansion cards. 128gb with 4x32gb will be more than enough for my needs.

If i add all my vms ram and give some of them an extra ram boost, all ram allocated to vms would be 35360MB. So with 64GB@start there will be enough free ram for future and esxi.

i pulled the trigger for

Supermicro X10SDV-TLN4F retail Preisvergleich | geizhals.eu EU
Samsung DIMM 32GB, DDR4-2400 Preisvergleich | geizhals.eu EU
SanDisk Ultra Fit V2 16GB Preisvergleich | geizhals.eu EU
Noctua NF-A6x25 PWM Preisvergleich | geizhals.eu EU

 

[Rand__]

€300 for the memory ? Holy Moly

 

[m4ntic0r]

yes ram price is really high at the moment.. these samsung dimms were at 150€ in 2016.

 

[m4ntic0r]

So.. everything is up an running (hopefully stable).

For me it was not easy to change the fan. The screws are too low and the cooler did not go off easily. I fixed the new noctua with wire.

Unfortunately my power consumption did not improve and it even needs about 10w more than my old system. although i used a new gold power supply and the low TDP of the Xeon. Now the lowest i can see on my APC is 97w. With 22 running VMs and ~1,2ghz usage it jumps from 97 to 115.

Everything works great so far, bios update was easy and ipmi is a dream. never had that before

Now i have some memory left and cpu power is never a problem. Everything feels fast, so that board together with a lsi 9271-8i and some ssd or hdd raid10 seems to be a nice build.

 

[Evan]

Is that the usage also including the overhead of the UPS ?
Seems high, that board will idle 35w or a bit less, add even 15w for HBA and 5w per drive x 2 brings you to ~70w.
Anyway it certainly is capable as a system for what you need, hope it works out well!

 

[Rand__]

My Xeon D (1528) idled around 50W in an SC846 , that was with gold psu which is not efficient at that level, backplane and fans.
At low power all the small bits add up
On the other hand I have a E3v5 with gold psu (not pdb driven) that idles at 35w (cpu fan only, no backplane) - never tested the D outside of the chassis and don't have it anymore to retest.

 

[m4ntic0r]

well there are some other devices on the apc:

Server:
Supermicro X10SDV-TLN4F + Noctua NF-A6x25 PWM (Specs 0,96W)
2x Samsung DIMM 32GB, DDR4-2400, CL17-17-17, reg ECC
SanDisk Ultra Fit V2 16GB, USB-A 3.0
LSI 9271-8i with CacheVault Module (~12W) + 40mm Revoltec Air Guard Fan (~0,5w)
3x Cremax Icy Dock MB153SP-B + 3x Noiseblocker NB-BlackSilentFan X1 (Specs 3x0,48W)
4x HGST 4TB Megascale (4x6.2W)
2x Samsung Evo 850 500GB (2x1,78W)
1x Seagate/Samsung Spinpoint M9T 2TB (2,3W)
1x Noiseblocker NB-BlackSilentFan XL1 120mm (0,96W)
Enermax Revolution X't II 450W ATX 2.4

Network:
Speedport W724V (~10W)
D Link DGS 1005D Switch (~2W)
Siemens Gigaset A260 (?)
-> 15W

-> all Fans (without the Noctua) + HDD/SSD = ~33,56W + LSI = ~45,56W + Network = 60,56W
-> Supermicro + Noctua + RAM = ~40-45W (with 22 VMs)

So this all seems to be ok?
But nevertheless, I am wondering about the efficiency of my old 2600k system + 32GB DDR3 in idle and low load.

EDIT:
What I noticed now very positively: If there is really some CPU load (50%), the consumption hardly increases. If i stressed my 2600k system i could easily see 150w an higher.