Next Homelab

[nitrobass24]

With so many CPU releases lately (TR, Epyc, Bronze Silver, Gold, Krpytonite ) it has me thinking about what my next move is since it has been about 5 years since I upgraded the compute portion of my lab. Nitro's - Condo=Home Data Center Replacement

Yes I am still running two identical boxes with Intel E3-1265L v2 and 16GB of RAM

Problem is 32GB is a bit too tight for my liking and now that I have loads of 4K content my plex can't keep up.

I have a quite a bit more space now, so ITX is not a hard requirement, but I do like small form factors.
However, Heat & Noise are even more important since I am not as isolated from the noise...all of this will be in my home office.

My plan is to stay with 2 nodes and move the S3710's I have from my NAS to the nodes for VSAN over 10G direct connection. I would also like to add an M.2 Optane drive for VSAN Caching.

Ideally, my future setup would provide a significant increase in transcoding capability over my existing setup. HEVC hardware transcoding would be awesome.
Need at least 32GB RAM per node
Needs to have Intel iGPU so I can pass thru to ESX
Need at least 2 10GBe (VSAN & HA) and 1 GBe interface (VM connections and mgmt).
Ideally ITX but would be willing to move up to a mATX board if it offered me a CPU option that I couldn't pass up.

I guess we should start with the CPU. Based on some reading I have done it seems the E3-15x5 series is targeted for this type of workload, but it's about a year old now and I was not sure how this stacks up now.

So STH what do you think?

 

[Netwerkz101]

My plan is to stay with 2 nodes and move the S3710's I have from my NAS to the nodes for VSAN over 10G direct connection

It has been a while since I even considered running a physical vSAN setup ... but I thought it required a switch.

I think "L" processors are not worth it over the standard CPUs of same model, especially if you are no longer space/cooling constrained.
Newer CPUs idle pretty efficiently and when you need it, the power is there.

The cost of anything new ...motherboard, CPU, and especially RAM is quite prohibitive right now - at least for my taste.

This CPU/Mobo combo seems ideal:
X11SSH-GF-1585 | Motherboards | Products - Super Micro Computer, Inc.

 

[T_Minus]

It has been a while since I even considered running a physical vSAN setup ... but I thought it required a switch.

I think "L" processors are not worth it over the standard CPUs of same model, especially if you are no longer space/cooling constrained.
Newer CPUs idle pretty efficiently and when you need it, the power is there.

The cost of anything new ...motherboard, CPU, and especially RAM is quite prohibitive right now - at least for my taste.

This CPU/Mobo combo seems ideal:
X11SSH-GF-1585 | Motherboards | Products - Super Micro Computer, Inc.

What am I missing here... why would you go with that $1000+ SOC setup vs an E3-1240\31\etc V3 + SM mobo which I've been able to source (separately, numerous times) for under $300, total. You're lacking the other 32GB capacity RAM so if that's a requirement step it up to v5/6 E3-12xx and you're still 1/2 the SOC cost!

What am I missing that's worth the extra $500 for the 1585 SOC?

 

[Netwerkz101]

What am I missing that's worth the extra $500 for the 1585 SOC?

LOL ... absolutely nothing.
I said anything new is cost prohibitive ... but it would fit his needs.
I know of the older SM boards that support VHD (X10SLH) but price (+CPU) is as much as newer combo.

Of course buying used is YMMV ... for my "next lab" I seriously considered going back to Dell Poweredge T30's (x3) for about $1000.
Doesn't help for Intel iGPU though.

EDIT - just re-read that ...

Seems like Intel iGPU and >32GB RAM capable are desired (required???).

While I definitely consider SoC for special use cases vs. being expandable, once you
start doing passthrough on _any_ platform...you just made it special use case - at least for now.

 

[nitrobass24]

Yea let me clarify a few things
1. Not space constrained but am cooling constrained
2. 32gb would be sufficient and I was looking at those SM E3-15xx boards until I realized how much they were. I'm not exactly on a budget here but I feel like I could do better than than 2K plus memory.


Sent from my iPhone using Tapatalk

 

[MiniKnight]

You've gotta go EPYC. That would be E....

 

[Patrick]

We are going to start publishing much more on EPYC this week. mATX is an issue since the new platforms are much bigger.

Instead of direct attach what about an inexpensive switch with 2x 10GbE?

 

[Rand__]

Vsan will require a 3rd node (witness at least).
Also Vsan is *not fast* with few nodes so if you're looking at fast (upper half of 10G) speeds you need a bunch of drives on a dedicated HBA - couldnt find the # of 3710s you have on your old build thread but i'd expect you'd need 4+, better more + cache drives to get anything usable.
Also o/c depends on your target storage size...

 

[Evan]

You say mATX is an issue due to size but supermicro managed to even add 2 extra dimm's to their mATX boards. And 2 x x16 slots instead of 1. The cpu and chipset are larger but it all seems to fit ok.

Supermicro | Products | Motherboards | Xeon® Boards | X10SRM-TF
vs
X11SPM-TF | Motherboards | Products - Super Micro Computer, Inc.

 

[nitrobass24]

We are going to start publishing much more on EPYC this week. mATX is an issue since the new platforms are much bigger.

Instead of direct attach what about an inexpensive switch with 2x 10GbE?

I have a switch, but if I am moving my SSD Datastore off the NAS it would seem unnecessary I would think.

Vsan will require a 3rd node (witness at least).
Also Vsan is *not fast* with few nodes so if you're looking at fast (upper half of 10G) speeds you need a bunch of drives on a dedicated HBA - couldnt find the # of 3710s you have on your old build thread but i'd expect you'd need 4+, better more + cache drives to get anything usable.
Also o/c depends on your target storage size...

Right now my needs are met by a 4x S3710 Raid10 over NFS. Are you suggesting that same setup with Optane M.2 Cache on VSAN would actually perform worse than what I have now?

 

[nitrobass24]

I think Patrick means mITX, I doubt any EPYC ITX boards will be available.

 

[Rand__]

Depends on your needs - writes will be governed by a single optane m.2 device then, reads by 2 S3710.
Also recommendation is to have cache device at 10% of data pool.

You need to be aware that vSan will not provide 100% pool read/write capability to a single instance either, it's more about providing consistent performance for a larger amount. I don't know how many vms is the goal, but its at least 10-20.

I have a bunch of threads (complaints) here trying to get proper performance out of vsan, eg
Alternatives to vmWare vSan for hyperconverged environment (Home)
I can tell you I throw a ton of HW at it and its still not where I'd like it to be.
If I wanted to start over I'd probably go with many (many) disk groups instead of nvme though

 

[Patrick]

@nitrobass24 I meant mATX (and therefore smaller like mITX.) You basically need 8x DIMMs minimum per EPYC socket and the EPYC socket is huge. That leaves very little room on mATX even. mITX the sockets for 3647 and SP3 are too big.

 

[nitrobass24]

Wow I did not realize it was that big of a socket. Hard to tell from pictures.


Sent from my iPhone using Tapatalk