For those that have built plex\storage servers that game on them as well!

Which way would you go?

  • Repurpose a tower server (T620, TD350, ML350 etc)

    Votes: 1 14.3%
  • Build a SuperMicro workstation

    Votes: 2 28.6%
  • Build a separate plex server

    Votes: 4 57.1%

  • Total voters
    7
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Chandla

New Member
Aug 21, 2017
9
0
1
40
"Well if we let them use consumer grade graphics cards and operating systems on server chipsets, then why would they buy workstations? And if they bought half as many machines, then there would be half as many graphics cards and processors out in the market."

-Person I do not like (Obviously tongue in cheek)


Ultimately what I'm asking here is if I can avoid making a dedicated storage & plex server by repurposing my server as a workstation. Not the first, won't be the last, but I hope this thread can be a help for people in the future with the same esoteric builds.

Usage for this machine is as follows


Actual benchmarks are sparse, so anyone with experience, it would be helpful. A lot of the presumptions here could be wrong, but I'd rather err on the side of overpowered than underpowered. I have very little free time, tinkering is not something that I get to enjoy. And frankly, my family doesn't appreciate it either. That being said, I'm not going to straight throw money into the trash or start making multiple build outs if I don't have to.

In order of usage
  1. General web,office and creative use (Windows 10\Office 2016\Adobe PS & LR)
  2. Data storage (up to 200gb per day, roughly 1-2TB per month)
  3. Plex (1 4k stream with w/ 6 concurrent FLAC chromecast streams) (Not all can be hardware decoded)
  4. Gaming
#1 dictates 16gb of ram, if I can alternate between an Quadro NV800 and GTX, I can save one rig. Currently I need a separate rig for 10 bit color pipeline (Quadro only). There is evidence I can consolidate these rigs with KVM. That would necessitate the only other item to increase on this build would be ram to 64gb (huge pano stitching in close to real time).
#2 dictates either a high thread count CPU with the below requirements or dual sockets, dual sockets preferred. When high thread counts are needed (plex with desktop or gaming use concurrently) single thread performance is not an issue (Which is what would make this machine okay using mid market Xeons).
#3 dictates a ~18,000 passmark (worst case 265 w/o HW decoding, 264 only 12K)
#4 dictates a single thread ~2000 passmark (combined 5000 passmark) with at least two cores (RE: analysis that Doom 2016 will max out processor usage around an i3-4300)
(#3&4 dictates a combined passmark of around 20,000 with around 12 threads)

Other requirements
  1. A build that is highly expandable through the future (For instance dual socket LGA2011 has a number of years before I could throw top of the line processors at it for any reasonable price. The RAM limits are outrageous and some chipsets allow DDR3 and DDR4)
  2. Front hot swap access w/ At least 2 other open 5.25 bays (I hate external stuff laying around) and need consistent access to drives for backups and upgrades of storage. The machine's primary pragmatic bottleneck for me is drive management. Hotswap is #1 behind hardware requirements.
#1 Again, not a lot of time to be swapping out systems and hardware here. So ideally, I'm looking for a system a couple years from now I can throw in some DDR4, new processors and a new graphics card in about 5 minutes and truck on for another couple years. I'm replacing an X58 machine that has served well but I just can't push this thing any further. That's where I'd like to be in 5 years on this build. That workstation will be a transitioned to a dedicated XP machine. Feels nice to retire something because you've brought it to its absolute limit. Not just because someone says "look at this new processor."
#2 Leaves it to 600 series chipsets
#3 Leaves SuperMicro style cases (bottom hot swap, top 3 5.25), Full size eatx "super tower" cases and niche servers like the T620 and Lenovo TD350 and HP ML350 which have 8SFF lower bays. These all seem to have issues with using a GTX (since they have workstation equivalents in their umbrella). The ML350 does not have an issue with a W10 install, still looking to see if it will take a GTX, then it might be the ideal platform.

And given the nature of the requirements, I do seem to be coming back time and time again to a SuperMicro x9 series DP LGA2011 Socket R board. This seems to be 2600 v1 & v2. These boards from what I can tell do not have issues running consumer graphics cards, but there doesn't seem to be a lot of information out there. As a last case resort I could buy a SLI certified board like the x10dax.

Ideally I'm trying to get as close to a 2000 passmark as possible for single thread performance with the highest passmark per dollar for the actual composite score. I can tell that they're really really trying to prevent this multi faceted type of build, because of course, I should be buying a xeon and an i7. I should be buying a Quadro and a GTX (though I'll have to buy an NVS for the 10 bit Adobe pipeline).

Options
  • Dell T620, disable onboard video, install KVM, run guest as hidden and remove virtualized info from the VM, run W10 as a guest, pass the GTX. This setup has the best storage management, SuperMicro's stuff is nice, but Dell's true server grade stuff is unbeat. Firmware level RAID and super cheap SAS drives are nice. I move enough data to be able to follow storage cost curves as they are phased out for the next round in the TB wars. So this is a very appealing setup, outside of the fact that I really don't want to spend that much time setting up the server and also knowing its a small dice roll that somehow Nvidia has thrown up additional road blocks or will in the future, locking the card out of KVM.
  • Custom SuperMicro workstation build. Honestly seems to be what I'm going to have to end up doing as all the mainstream vendors are locking out the GTX and consumer OS builds. SuperMicro doesn't seem to have this issue, though I have to check if the c600 chipset has a roadblock for GTX. I believe the Asus Z9PE-D8 WS to be a rebranded SuperMicro.
  • Resign in defeat and build a separate plex\storage server.
Gut is telling me resign in defeat. Want to hear from others who have done the other options!
 

TType85

Active Member
Dec 22, 2014
630
193
43
Garden Grove, CA
It is going to be a bit of work to do all you want in one machine, seeing as you have the Quadro and the GTX (2 separate VM's i'd guess)

On the 2011 V2 side, for dual CPU you have a couple choices above 2K passmark. E5-2667 V2 & E5-2867W V2 both meet the mark and are dual socket capable. At that point though you are at the end of the socket. Nothing is really going to be faster/better but ram is cheap.

V3/V4 I am not sure other than the E5-2697/8/9 V3 get there but you are looking 1K+ per CPU (non-QS/ES) plus having to go to more expensive DDR4 ram. You are still going to be at the end of the socket.

2K+/- passmark seems like you need at least 3.6Ghz turbo clocks to get. I had a E5-2676 V3 QS processor (3.0ghz turbo) running KVM and 2 VM's (each with 16GB ram, 4c/8t, GTX 970 and GTX 1070) and for general usage it was fine. Playing World of Warcraft wasn't great. That game runs best with high core speeds on a couple cores. I got 50% better frame rates on a i5-6600K.

You could look at Threadripper, one 16 core chip meets both passmark scores, has plenty of pci lanes for expansion but right now there does not seem to be a lot of info for using it on KVM doing passthrough (this is causing me to balk at getting one right now). Leve1Techs looked at the IOMMU grouping on one board and said it was good, but it put a m.2 slot in with a video card so it's not perfect.

Separate machines is by far the easiest. You could build a overclocked Ryzen box real cheap and it would be a streaming monster. I prefer the challenging route myself :p
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
So I am a little under-educated on the productivity side of things but are you sure 10 series GTX can't handle what you need for 10bit color?

As far as plex goes, hardware transcode is in Beta right now and is working fairly well on non-4k and non-10bit video right now (should be coming). I would probably think about a 7th gen Core series to be able to use Intel Quick Sync with Plex and get h.265 10-bit de/encode. A 10 series GTX will handle it too just limited to 2 streams. Unless you go to a Quadro P2000 or higher.

I have run a GTX 1050ti using PCI passthrough on a supermicro X9DRi running ESXi. I am currently running a Quadro P400 on it specifically for Plex transcode. Both require you to trick the Nvidia drivers to work from inside the Windows VM (Turns out you have to get up to the p2000 before nvidia allows vt-d by default).

If you take plex hardware transcode into account that may affect your CPU choice. Or maybe not.
 

Chandla

New Member
Aug 21, 2017
9
0
1
40
So I am a little under-educated on the productivity side of things but are you sure 10 series GTX can't handle what you need for 10bit color?

As far as plex goes, hardware transcode is in Beta right now and is working fairly well on non-4k and non-10bit video right now (should be coming). I would probably think about a 7th gen Core series to be able to use Intel Quick Sync with Plex and get h.265 10-bit de/encode. A 10 series GTX will handle it too just limited to 2 streams. Unless you go to a Quadro P2000 or higher.

I have run a GTX 1050ti using PCI passthrough on a supermicro X9DRi running ESXi. I am currently running a Quadro P400 on it specifically for Plex transcode. Both require you to trick the Nvidia drivers to work from inside the Windows VM (Turns out you have to get up to the p2000 before nvidia allows vt-d by default).

If you take plex hardware transcode into account that may affect your CPU choice. Or maybe not.
A GTX can not, which is why I mentioned the NVS. That's Nvidia's productivity Quadro, 2D only. It's technically designed for "signage" machines, like booths and ad signs etc. But it has the Quadro license and 10 bit pipeline. A 3D Quadro is a ton of cash really to get something that you can use with 4K resolution, which I use daily just to browse.

I have hardware decoding and am going to be testing it out myself as well. From what I understand though, I thought it only worked on HVEC\246\265 content only. So my concern is when I'm transcoding small files and upconverting them for 4K that it's hitting the processor just as hard, just without the transcoding on the fly. Maybe I'm not thinking of this right, but that would simplify thinks a little bit. If I can relax my plex needs, then I just need a much more modest CPU.

That's a really nice setup you've got there, looks like you're doing what I want to do. Can you fire up Doom16 or something in the same range and transcode while you're gaming? So you're saying you couldn't use the GTX alone in ESXI. That I've read already. From what I've seen, KVM is the one that will let you play around with it to go outside of the box. All the other vendors are colluding.

Did you choose esxi for your build for any other reason that esxi is just the normal choice? Or were you constrained? Just because KVM seems to be the one to go to with passthrough. Just too many licensing restrictions otherwise, unless you're paying up big time for the Quadro stuff. It's sad they didn't further delineate the VM choices so that VM workstation power users were seperated from large online VM vendors.
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
Plex shouldn't be upconverting at the server. The end device will scale it. Plex will only transcode and downconvert if needed. As far as codec support that should be based on whatever hardware you have i.e. HVEC/265 only in newer stuff. Typically if it isn't one of those formats though it is probably lower bitrate stuff that isn't CPU intensive to decode and it will still use hardware to encode the outbound stream.

You can use GTX in ESXi you just have to change one thing.

ESXi webclient, right click your VM > Edit Settings > VM Options > Advanced > Edit Configuration > Add Parameter: hypervisor.cpuid.v0 = "FALSE"

It is an nvidia driver issue that detects it is running in a virtual environment and prevents drivers from loading.
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
See why you chose that card now ;)

Video Encode and Decode GPU Support Matrix

That's a nice little gem you got there. Not surprised they put an artificial floor there for the P2000. That might be a way better choice than the NVS if it's possible to do 4K daily browsing on it.
It will absolutely handle 4k productivity 2d stuff no problem. Certainly not going to be gaming on it. I got it cause I thought I could get away without the virtualization hack in ESXi but they limit that in drivers for the low end quadros just like the GTX.
 

Chandla

New Member
Aug 21, 2017
9
0
1
40
It will absolutely handle 4k productivity 2d stuff no problem. Certainly not going to be gaming on it. I got it cause I thought I could get away without the virtualization hack in ESXi but they limit that in drivers for the low end quadros just like the GTX.
So what's the trick with the P400? Because that's a couple ticks lower than the P2000. You can still pass the GTX right?

As far as the driver limitation, that's what KVM gets around. The card doesn't even understand it's guest because KVM let's you spoof that. Since they're not really beholden like Xen, VMware or Microsoft. They're not part of the kabal.

There's a reddit sub I just saw, but I'm not deep in it yet. Can't remember or find it right now, like VMio or something, it's all about passing hardware.
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
So what's the trick with the P400? Because that's a couple ticks lower than the P2000. You can still pass the GTX right?

As far as the driver limitation, that's what KVM gets around. The card doesn't even understand it's guest because KVM let's you spoof that. Since they're not really beholden like Xen, VMware or Microsoft. They're not part of the kabal.

There's a reddit sub I just saw, but I'm not deep in it yet. Can't remember or find it right now, like VMio or something, it's all about passing hardware.
P400 uses the same trick as GTX that I posted above. It just changes the setting that the Drivers look for to tell if it is virtualized or not. Using that setting does cause one little weirdness and that is task manager doesn't show CPU utilization but everything works fine and people have been doing this for a very long time.
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
So one other thing. Do you need to Virtualize at all? I would think about running Windows bare metal and if you need to spin up a vm you have VMware Workstation/Virtualbox/Hyper-V (Win Pro/Server). It seems people do this for extra CUDA cores but I don't know about gaming. Just a thought.
 
Last edited:

Chandla

New Member
Aug 21, 2017
9
0
1
40
I had a total change of heart today. The machine I was going to replace, I had bought an identical workstation rather than troubleshoot the problem as mobo, processor etc. As a bonus, it's a workstation I've grown attached to (Sun) because I'm a fan of vintage enterprise and other notable computing hardware. This is the future I want for when I run into problems again. I was back up and running in 10 minutes flat. This is why companies build out the way they do. It was a religious experience in that sense, old enough to be done with chasing down issues just so I can have my own little custom build. I'll stay in the box.

So the question now, is if I throw a T620 in the basement, and I use the integrated video, since Dell does not support third party video cards. Will the Quadro P400 still transcode? I'm just going to roll out a server in the basement for plex and storage. It will cramp my style to a certain degree, but it's just a much much better setup, flat out.
 

K D

Well-Known Member
Dec 24, 2016
1,439
320
83
30041
I thought the plex beta is intel quick sync only and doesn't support gpu transcodong yet.
 

StammesOpfer

Active Member
Mar 15, 2016
383
136
43
I thought the plex beta is intel quick sync only and doesn't support gpu transcodong yet.
When they originally droped the first preview they were targeting Intel first. I believe they are trying to do it in a vendor agnostic way. I can say for sure that in preview 3 (1.6) and newer it worked on nVidia hardware. The down side is nvidia limits you to 2 encode streams unless you spend $500+ on a Quadro P2000. Intel doesn't have the same limitations and there hasn't been much discussion about AMD products working. I didn't need more that 2 streams and I'm running it on Xeon's so the cheapest nvidia Pascal based GPU that supports NVENC is either a Quadro P400 or GTX1050. The GT1030 does not have the encoder.
 
  • Like
Reactions: K D