New Proxmox Server Build Supermicro X11SPA / X12SPL

[tcpluess]

Good day,
I am new to the forum. As I have, during the past, found many useful hints in this forum, I decided I want to discuss my new server build here.
I have so far used, for my home network storage, VPN, and so on, an old HPE server. I don't even remember the exact product number. But it is too dated now and also has no longer enough power to run all my Proxmox stuff, so I decided I want to build something new, which will again last for the next couple years to come.

In my work place, we use a lot of Supermicro devices, and therefore, I decided I will use one for my own server as well. Also I have found here and also in the TrueNas/FreeNas community that lots of people do this. So it seems legit for me. I have currently thought about the following two options:

a) Supermicro X11SPA-T board and Xeon Silver 4210R processor. The cool thing about this board is that it has already 4 slots for M.2 SSDs onboard, 12 slots for RAM and many PCIe slots. It is a workstation board, though, and not a server board. Not sure how much this matters. But for instance it includes an audio codec, which will be useless.
b) for a very similar price, I could get an X12SPL-F board and a Xeon Silver 4310 processor. This is a "true" server board, but has only one M.2 slot. On the other hand, I will probably use hard disks for the storage, and let only the OS run from M.2, so this is probably not a criterion.

As I can get used 10GbE NICs and SFP+ modules from my employer (we are changing to 25GbE and therefore throwing out some "older" equipment), I will anyways not use the onboard 1GB ports on both main boards, but instead install a 10GbE NIC with SFP+. Further, I will start with both boards with approx. 64GB RAM, and later upgrade to 128 if needed.
Use case will be File Server, Proxmox VMs (Windows as well), VPN Server to access the home network from externally, some Plex, and general homelab use. Will use ZFS of course. What do you think, which of the two is the nicer one? the X12SPL seems to be a bit newer, but otherwise, they seem to me pretty much equal from the performance point of view, it is just that the workstation board X11SPA has more M.2 slots, which could be nice (on the other hand, more M.2 slots can be added to the X12SPL with a PCIe to M.2 card).

I will attach 8 hard disks for the beginning, but I have already included a HBA for later expansion

Any better ideas? I believe both variants could make a nice home lab server.

 

[nexox]

I would say first, the workstation board lacks IPMI, so you'll need to run a discreet GPU and you won't be able to control it over the network, plus you won't be able to set fan thresholds, so large, slow fans might cause it to cycle the fan speed up and down constantly.

Second, I don't know if that 4210R is really worth the extra price over a 4114, which can be had for under $10, which probably makes the X12 setup cost somewhat more than the X11.

 

[NPS]

Are you shure you want to use such a big platform with such low power CPUs? Looks like wasting much energy for nothing to me.

 

[SmashedSqwurl]

I've been using an X11SPA-TF with a Xeon Silver 4114T, 128GB RAM, 1 NVMe SSD, 6 SATA SSDs, and 6 SAS HDDs as my main storage + container host for a few years now. No major complaints aside from it idling in the 160-170W range. My estimate is that the drives account for about 60W of that.

 

[tcpluess]

Good day,

@NPS
what CPU would you recommend?

I also just remember that I have a couple HPE 640SFP28 network cards, I will use them with some SFP+ modules I have!

@SmashedSqwurl
thanks for your post, what OS do you use? do you run Proxmox?

 

[NPS]

It's not that the CPUs are bad, it's just that you did not talk about any needs that would break let's say a LGA1700 or AM4/5 platform. A 10core Cascadelake CPU is somehow slow compared to a modern 6-8 core CPU in nearly all metrics. Even a modern 4C could be faster for many homelab use cases because of single core performance.
There are of course reasons for the bigger plattforms: PCIe lanes, max core count, max memory, memory bandwidth
Do you need any of these?

 

[SmashedSqwurl]

thanks for your post, what OS do you use? do you run Proxmox?

Yes, I run Proxmox.

To echo @NPS I'll probably end up replacing it with an Intel 12th/13th gen system at some point for power efficiency unless there's a big enough drop in NVMe prices that going all-flash becomes reasonable and I would need the PCIe lanes.

 

[tcpluess]

The reason why I would prefer a board with lots of PCIe slots is that I want to add a couple of PCIe cards:

a) one PCIe to 4x NVME card which then works as cache for ZFS;
b) one HPE 640SFP28 network card which I will use to interface to my home network;
c) one SATA HBA to add more disks later on.
d) will probably also at some point add a graphics card for some VMs.

So my storage will be, currently, disks, and the NVME I use only for the OS and for cache (ZFS L2ARC / SLOG).

On the other hand, of course I don't desperately *need* all this, because this is a homelab server, but I want to run at least a proper file server that provides lots of speed; a Nextcloud for my files; a VPN Server to access all this from abroad; and I want to use ZFS. :-D

 

[SmashedSqwurl]

One thing to bear in mind is that you at least get the GPU for free if you go with an LGA1700 system. Onboard SATA would probably be sufficient for up to 6 disks. The NVMe card might be an issue since Intel can only do x8x4 bifurcation, but you might be able to find a board with 4 NVMe slots. Whether all of that also allows an x8 NIC is a good question, but I think it might be possible.

 

[SmashedSqwurl]

At the end of the day if your main priority is PCIe lanes, then Xeon Scalable is probably the best perf/power/price combo. It's just that you're paying a penalty on the single thread perf and power efficiency compared to something more modern.

 

[tcpluess]

I see.
Would the X12SPL-F board be acceptable, but maybe with another CPU, or is this a bad idea cause it is too old?

 

[SmashedSqwurl]

It depends on how much CPU performance you need. The 4310 is about 20% faster than the 4210R and should have about the same or better idle power consumption. The X12SPL-F is probably the more power efficient of the two motherboards (ATX instead of EATX, fewer bells and whistles).

I'd also make sure the NIC and HBA you use support ASPM so that the CPU can enter lower power states, and get a high efficiency power supply.

 

[nexox]

I would say the X12 platform is not as cost effective as X11, you don't get much more capability and the CPUs cost a whole lot more. I'd do X11 with a 4114 or 6132 if you can't get away with an LGA1700 system.

 

[NPS]

I see.
Would the X12SPL-F board be acceptable, but maybe with another CPU, or is this a bad idea cause it is too old?

It's more than acceptable for shure. But I think energy consumption will be even higher than Cascadelake while CPU performance in that generation was a little disappointing.

How much storage do you need and what performance in which use cases do you want to achieve?
I run a E2356G on an X12STL-F (to use the iGPU you'd need a C256 based board) with 2x SATA for Proxmox and VMs plus 2x gen4 NVMe and a 25GbE NIC passed through to a NAS VM. I also have a P4801X for SLOG but I may remove it on the long run. I still have a gen3x2 slot and 4x SATA free. Using a LGA1700 platform you will have even more (and faster I/O) so building a fast NAS is totally possible. You also don't need many NVMe for all the cache types in ZFS. Use partitions!