ESXi 6.7 on Ryzen x399 for VMs + GPU passthrough desktop

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
40
Near Seattle
averyfreeman.com
Hey,

tl;dr,
I'm thinking of migrating my homelab from a handful of Sandy and Ivy Bridge E3s and Core i5/i7s to a single Ryzen ESXi Windows desktop (GPU passthrough) / FreeBSD file server (since Ryzen supports ECC) and a trio of old Thinkpads T520s to save power but still have high availability

I know I could go with Epyc, but I'm more attracted to the Ryzen 1) because I'm cheap and 2) because its single-theaded speed is considerably faster than any Epyc processor I could conceivably afford.

Right now my machines idle around 100w each so I'm consuming up to 400w power just having them sit on, add another 200w or so when they're being utilized - I noticed max power draw at the wall for a x399 w/ 1950x according to Anandtech.com was under 400w and idles around 75w, so that'd be a real power savings.

I've tested the aging Thinkpads I can get for $next-to-nothing and they're range is 17w to 45w each, work well with ESXi 6.7, have Intel 82579LM NICs. I ONLY want them for HA fault tolerance so the less they cost and the less power they use the better.

Anyway, I guess the question I'm wanting to ask is:

1) Do you use Ryzen + x399 for your ESXi homelab? Which motherboard? Which version of ESXi?

2) Do you use GPU passthrough (for a desktop type system)? Specifically, which GPUs have you tried, what did you have to do to get it to work?

3) What problems can I foresee running into if I go this route?

Any help would be much appreciated. Thanks!
 

MiniKnight

Well-Known Member
Mar 30, 2012
3,014
922
113
NYC
Hey, Ryzen 2700 or Threadripper?

Ryzen ECC is only on some motherboards. If you're really set on proper ECC, you'd want RDIMMs which Ryzen, Ryzen TR, and even Xeon E3's don't support.

What's the desktop for? What kind of GPU are you going to pass-through?

I like EPYC, but there's some great deals on E5 V2's out there.
 

AveryFreeman

consummate homelabber
Mar 17, 2017
254
23
18
40
Near Seattle
averyfreeman.com
Hey, Ryzen 2700 or Threadripper?

Ryzen ECC is only on some motherboards. If you're really set on proper ECC, you'd want RDIMMs which Ryzen, Ryzen TR, and even Xeon E3's don't support.

What's the desktop for? What kind of GPU are you going to pass-through?

I like EPYC, but there's some great deals on E5 V2's out there.
Hey, thanks for the 2c

Definitely threadripper

I have looked at a lot of E5 v2s (right now have mainly E3 v2s) but the power consumption at idle has gone down so much since then I am really looking to get something newer with a 14nm process and some of the newfangled bells and whistles that have been implemented to mitigate unnecessary power use (like demand based switching, etc.)

Also, ACS and assignable PCIe lanes are fairly new and generally restricted to very high-end systems, which is necessary for the GPU passthrough to work properly (I've tried VFIO with a SM C7Q67 + i7-2600 and AMD S9000, the technology just wasn't quite there).

I don't mind using UDIMMs for ECC, it wouldn't be any different than what I've been doing.

I'd be using whatever will pass through for GPU and doesn't cost too much money or use too much energy, like a 1030 GT or my old 740 GT - I don't care because I don't game or need to drive 4K monitors. I'm just using Windows 10 on which I do spreadsheets and watch the HDHomeRun TV app.