Need Advice for Xeon D Build with GPU Pass-through

[Fodmidoid]

Hi Everyone,

I'd like to have a Xeon D-15xx home server with 64 GB RAM, GPU pasthrough, with ESXi 6/6.5 (perhaps KVM down the road) running VMs: pfSense, PLEX, Asterisk, and an HTPC that I can use with a BenQ HT2050 projector for watching Kodi, streaming movies from the PLEX VM and I was hoping for some gaming as well.

Can I pull this off, or do I have to nuild like three different systems to achieve all this. I'd like to believe I could do this, and do it well.

I will need recommendations for:

• Which Xeon D board to use (or possibly something else, like an AMD Ryzen?)
• Memory type (see RAM below)
• Storage (SSD/NVRAM, etc.)
• Video Card
• CASE (I'm hoping for rack-mount, but if I can't make it work with the GPU, I already have a Thermtake Core V1 just sitting in the box.

I have Fios 1 Gigabit Internet and will be connecting this build to a Cisco 3750 Gigabit managed switch.

RAM:
Also, what type of RAM do I get for this type of application? I see these boards support 4x 288-pin DDR4 DIMM; Supports up to 128GB DDR4 ECC RDIMM, Supports up to 64GB DDR4 ECC/non-ECC UDIMM, DIMM Sizes: 32GB, 16GB, 8GB, 4GB.

As always, advice is greatly appreciated. Thanks very much for your help.

 

[OBasel]

16GB or 32GB RDIMM only.

 

[Fodmidoid]

16GB or 32GB RDIMM only.

Thanks. Are you saying, only use 16 or 32 GB RDIMM modules for a total of 64 GB max?

Any advice for the rest of the build?

Sent from my SM-G955U using Tapatalk

 

[Patrick]

Ryzen still has issues with ESXi although EPYC is launching next week so...

Sounds like you are looking at a D-1540/D-1541. You will want 4x DIMM slots. I go 2x32GB if I know I need at least 64GB RAM.

Video card/ case are related and up to you on how big you want to go.

SSD how big and budget? People here can help using those parameters.

 

[Silvio]

For GPU, if you want to passthrough stick to Radeon, the Nvidia GeForces aren't supported. I had an old GT610 that would lock the system if you shutdown the VM and read reports that modern cards are hit and miss. I have a Radeon now and it is flawless...


Sent from my iPhone using Tapatalk

 

[Fodmidoid]

Ryzen still has issues with ESXi although EPYC is launching next week so...

Sounds like you are looking at a D-1540/D-1541. You will want 4x DIMM slots. I go 2x32GB if I know I need at least 64GB RAM.

Video card/ case are related and up to you on how big you want to go.

SSD how big and budget? People here can help using those parameters.

Thanks, Patrick.

Sent from my SM-G955U using Tapatalk

 

[Fodmidoid]

For GPU, if you want to passthrough stick to Radeon, the Nvidia GeForces aren't supported. I had an old GT610 that would lock the system if you shutdown the VM and read reports that modern cards are hit and miss. I have a Radeon now and it is flawless...


Sent from my iPhone using Tapatalk

Thanks very much. Any specific Radeon card recommendations?

So, is my "One rig to rule them all" idea feasible? Especially with running a HTPC/Gaming VM?

Sent from my SM-G955U using Tapatalk

 

[Silvio]

Thanks very much. Any specific Radeon card recommendations?

So, is my "One rig to rule them all" idea feasible? Especially with running a HTPC/Gaming VM?

Sent from my SM-G955U using Tapatalk

I think it will be good as long as the different services are not competing for the same kind of resource. NAS is IO-bound, Kodi is GPU-bond, games are CPU/GPU intensive.

My build is similar to your idea with a NAS, my workstation and my infra (DNS, etc.). I use Kodi on a FireTV and map the NAS via NFS, works even at 4k resolution plus I can add more FireTVs and do the same on other TVs.

As for GPU, I started with a Radeon RX570, worked great but was too big, I replaced it with a RX550 since I don't game much. With the GPU passthrough I recommend you pass one USB controller (motherboards usually have more than one with different PCIe IDs) and connect your keyboard/mouse on it, otherwise your CPU will emulate an USB controller that will take more processing (not much but something), will be slower and introduce lags to your games.

Let us know how you progress!


Sent from my iPhone using Tapatalk

 

[jmccoy555]

For GPU, if you want to passthrough stick to Radeon, the Nvidia GeForces aren't supported. I had an old GT610 that would lock the system if you shutdown the VM and read reports that modern cards are hit and miss. I have a Radeon now and it is flawless...


Sent from my iPhone using Tapatalk

I have this issue with my RX580.

Sent from my SM-G920F using Tapatalk

 

[Silvio]

I have this issue with my RX580.

Sent from my SM-G920F using Tapatalk

Trying not to hijack the thread but then again might be useful information for the OP.

Can you give more details on the issue? My problem with Nvidia was different depending on the OS. With ESXi I just got either a Purple screen or simply hangs when trying to start the VM. With Linux KVM it would start and work however after shutdown it would give some errors and trying to start again would always crash the machine.

I didn't try ESXi again after I put the Radeons in place, KVM has been rock solid. My system is a 2xE5-2660v1, Intel C602 chipset.

 

[jmccoy555]

No hijack intended, just a heads up that all may not be fine and dandy!

E3-1230v3 ESXI 6.5, I did read that 5.5 may work better. This is my main server and I was just trying to add a mining VM. I tried both Windows 10 and Ubuntu, same result as you, lock up or purple screen. Apparently you can edit the passthrough configuration file but after so many reboots I got fed up and gave up for a while. I'm building up another server at the moment so will try again as constant reboots and offline time will not be such an issue. One thing I did notice was that ESXi reported the card as a RX480, not 580.

There's a bit of info out there but not too much, and I guess this is pretty much a home user issue. Would be nice if it worked properly though.

If you don't need to reboot the VM, or can live with a full reboot then it looks like it would work quite nicely.

Sent from my SM-G920F using Tapatalk

 

[Silvio]

No hijack intended, just a heads up that all may not be fine and dandy!

And helps a lot, since I guess he did not buy the hardware yet. The hijack comment was about me

E3-1230v3 ESXI 6.5, I did read that 5.5 may work better. This is my main server and I was just trying to add a mining VM. I tried both Windows 10 and Ubuntu, same result as you, lock up or purple screen. Apparently you can edit the passthrough configuration file but after so many reboots I got fed up and gave up for a while. I'm building up another server at the moment so will try again as constant reboots and offline time will not be such an issue.

I recall there is a known issue with E3-12xxv3 and IOMMU that doesn't support IRQ remapping or something along these lines.

One thing I did notice was that ESXi reported the card as a RX480, not 580.

Same for RX570, lspci would show RX470. RX550 shows ok as there was no RX450.

There's a bit of info out there but not too much, and I guess this is pretty much a home user issue. Would be nice if it worked properly though.

If you don't need to reboot the VM, or can live with a full reboot then it looks like it would work quite nicely.

Yes I struggled a lot to find information, had to pickup bits and pieces to make sense of it. Can't complain now running Debian 9 with KVM.



Sent from my iPhone using Tapatalk

 

[Fodmidoid]

And helps a lot, since I guess he did not buy the hardware yet. The hijack comment was about me



I recall there is a known issue with E3-12xxv3 and IOMMU that doesn't support IRQ remapping or something along these lines.



Same for RX570, lspci would show RX470. RX550 shows ok as there was no RX450.



Yes I struggled a lot to find information, had to pickup bits and pieces to make sense of it. Can't complain now running Debian 9 with KVM.



Sent from my iPhone using Tapatalk

Hi there. Thanks for all the comments. Please excuse the delay. I've been away. No, I haven't bought the hardware yet, though I'd like to start putting a parts list together and order everything as soon as I can.

I'm real confused now with all the recent discussion about cards, though. Perhaps you can enlighten me a bit and tell me exactly how this would apply to me and possibly make a solid video card reference for the build?

Thanks!

Sent from my SM-G955U using Tapatalk

 

[Jeggs101]

GTX 1060 or 1050 Ti for the price.

AMD is better for virtualization on consumer but they cost too much because of miners buying up all the supply

 

[Silvio]

GTX 1060 or 1050 Ti for the price.

AMD is better for virtualization on consumer but they cost too much because of miners buying up all the supply

I got mine as "open box" at Fry's a bit cheaper than online. From what I read at least a few people have being using GeForces 1060 in passthrough.

My recommendation is to do the scientific way: experiment

Choose the type of GPU you are more comfortable with regardless of the VM and put it to work. It might take some effort until it is running clean. Worst case scenario you will have to return and choose another. If you go that route, pay attention on who charges restock fees (e.g. Newegg) and the period you can return (usually 30 days but YMMV).


Sent from my iPhone using Tapatalk

 

[Fodmidoid]

Thanks for the advice so far.

I started out streaming kodi on a fire TV on my BenQ HT2050 projector, but during scenes that pan from side to side, it was a little jittery, so next I tried out an Nvidia shield, which is great as a streaming device. Much more powerful, but still slightly jittery (like when they sweep across a landscape or something), so that's what brought me to the conclusion that I need a system with a decent video card in order to have a flawless experience when using the projector.

Based on what I'm hearing here, I'm wondering if I should be considering splitting the gaming/htpc and always-on VM server build to separate machines, after all.

Or is it still a good idea to try and accomplish everything in a single build and virtualize the htpc and gaming?

What are your thoughts on all of the above?

Thanks a lot. I'm excited to get started with this project.



Sent from my SM-G955U using Tapatalk

 

[EffrafaxOfWug]

during scenes that pan from side to side, it was a little jittery, so next I tried out an Nvidia shield, which is great as a streaming device. Much more powerful, but still slightly jittery (like when they sweep across a landscape or something), so that's what brought me to the conclusion that I need a system with a decent video card in order to have a flawless experience when using the projector.

Hmm, that sounds like a classic screen tearing / framerate mismatch issue rather than the hardware being able to keep up. Assuming the drivers you're using can support it, XBMC's got a bunch of options in the video settings section to attempt to work around this - turning on vsync should probably be your first port of call, then try disabling hardware decoding (some video decoders don't play nice) and experimenting with the display sync option. Essentially there's a million and one ways this issue can manifest itself and a corresponding number of ways to fix it.

 

[Fodmidoid]

Hmm, that sounds like a classic screen tearing / framerate mismatch issue rather than the hardware being able to keep up. Assuming the drivers you're using can support it, XBMC's got a bunch of options in the video settings section to attempt to work around this - turning on vsync should probably be your first port of call, then try disabling hardware decoding (some video decoders don't play nice) and experimenting with the display sync option. Essentially there's a million and one ways this issue can manifest itself and a corresponding number of ways to fix it.

That's great. Thanks. I will check this out. If it ends up working, maybe I should just focus on building an always-on VM server and possibly have one of those VMs used just to browse web pages on the projector, etc, instead of a full blo name htpc and gaming VM. I'll let you know how it goes.


Sent from my SM-G955U using Tapatalk

 

[Fodmidoid]

That's great. Thanks. I will check this out and let you know how it goes.


Sent from my SM-G955U using Tapatalk

Sent from my SM-G955U using Tapatalk

 

[Fodmidoid]

But, I'm still wanting to know if I can have one good server, running ESXi, Xen, or KVM/Proxmox/Ovirt, etc. That I can run my always on VMs, including a VM as an HTPC and gaming VM. I have a Synology NAS I'll be using for network media storage.

Sent from my SM-G955U using Tapatalk