ESXi/PLEX/Storage server upgrade... WWYD

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
I've been racking my brain over cost vs performance for my planned server upgrade... the box will have to deal with all my VM's + storage, etc. The heaviest resource of course is PLEX, which can range anywhere from 6-8 users on at a time currently, mixture of transcoding + DS. with more transcoding users (obviously) so I would like to add head room for that.

My current box seems to deal with around 4-5 transcodes depending on the quality of the file before I see constant 80-90% CPU loads on the PLEX VM (8 cpus currently assigned). All my video files are of high quality .mkv's so and will remain this way (TV episodes 1.5-4 GB, 1080p movies 25-45 GB).

My current server:

2x X5670's = a (dual) passmark of 12277
64 GB RAM

I"m trying to decide between the following CPU's, but am open to suggestions of alternatives


All $$ figures are in CAD.

The 2695 v2 at this current time is the cheapest CPU, followed by the 2667 v2 at about $70 more, and then the 2690 v2 being about $120 more per CPU over the 2695 v2.

Passmark for dual CPU setups:
2x 2690 v2 = 23728
2x 2667 v2 = 22368
2x 2695 v2 = 20811

I would also like to have at least 128 GB of RAM this time around (currently have 8x8GB modules on my server), and obviously require a new mainboard (seems to be somewhere around $420-600).

So my question is, WWYD (what would you do) if this was your choice trying to stay around a budget of $1500 CAD (which is around $1150 USD right now)? And do you have any suggestions (even for a slight increase in cost) that would be better suited?
 
Last edited:

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
have you considering to use GPU transcoding instead of CPU only? i did some research and the quadro p2000 can take a lot of transcoding workload off CPU and can get upto like 20x HD streaming.

Hardware Acceleration using Quadro P2000 for Unlimited Transcoding?
HW Transcoding with Nvidia Quadro p2000
That is very interesting! I will look into that as well. That said I believe I have read somewhere that GPU passthrough can be touchy and also GPU decoding isn't working in linux for PLEX...
 

ajs

Active Member
Mar 27, 2018
101
36
28
Minnesota
That is very interesting! I will look into that as well. That said I believe I have read somewhere that GPU passthrough can be touchy and also GPU decoding isn't working in linux for PLEX...
Also note the consumer gtx 1050/1060/1070/1080 GPU's can also transcode. They are "limited" to two streams, but there are driver workarounds that remove this limitation.
 

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
Also note the consumer gtx 1050/1060/1070/1080 GPU's can also transcode. They are "limited" to two streams, but there are driver workarounds that remove this limitation.
The more I look into this, the more I think that in addition to the cpu/mobo upgrade, I should invest in a graphics card for hardware transcoding.... I have seen the hack for the normal cards (vs. a P2000), and the hack seems to include linux. But I believe PLEX still has some sort of limitation with decoding under linux. Not sure if the hack for the NVIDIA driver address that (at this point I don't think it does).
 
  • Like
Reactions: itronin

mmo

Well-Known Member
Sep 17, 2016
558
357
63
44
Also note the consumer gtx 1050/1060/1070/1080 GPU's can also transcode. They are "limited" to two streams, but there are driver workarounds that remove this limitation.
Might want to compare the power consumption as well, since Quadro uses much less power than GTX.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,515
650
113
IMO, if you have the need for HW transcoding you most likely have a lot of activity on your server every day. In that case, downtime and constant troubleshooting is not a good thing (I know this is the case for me).

What I've learned with regard to Plex is not to introduce any beta or "partially" supported features into your main everyday server otherwise you'll pull your hair out with the downtime from bugs or Plex server updates that break your "personal fixes."

I'm currently in the middle of upgrading my own Plex server and i'm focusing on the CPU. I run Plex in a docker on Linux. If in the next year ot 2 support for add-on GPU transcoding becomes widely supported I can easily add that to my system.
 

mmo

Well-Known Member
Sep 17, 2016
558
357
63
44
The more I look into this, the more I think that in addition to the cpu/mobo upgrade, I should invest in a graphics card for hardware transcoding.... I have seen the hack for the normal cards (vs. a P2000), and the hack seems to include linux. But I believe PLEX still has some sort of limitation with decoding under linux. Not sure if the hack for the NVIDIA driver address that (at this point I don't think it does).
Plex said it's due to the oudated ffmpeg and they are working on it without ETA.

Hardware Accelerated Decode (Nvidia) for Linux
 

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
Plex said it's due to the oudated ffmpeg and they are working on it without ETA.

Hardware Accelerated Decode (Nvidia) for Linux
Yes... more due to the complexity of their specific and custom use case of ffmpeg.

With all this consideration of using a dedicated card for transcoding in PLEX it's made my decision quite a bit clear. Get the 2695 V2's, use the saved money towards the RAM and P2000, deal with PLEX server running in a Windows VM for now (or forever?)... and enjoy a powerful new server while not having to dedicate the energy and heat output to just serving up PLEX streams.

Now I'm going to open up my new 4U Rosewill case to see what I will have to remove/cut to put the EATX mobo in there :)
 
Last edited:

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
Well might as well do a small update as I finalized my decision...

I went a head and purchased the following:

  • 2x E5-2695 V2
  • 8x 16Gb DDR3 EEC Samsung ram (specified by Supermicro)
  • Supermicro X9DRH-7TF
I feel that should give me more than enough horsepower for what I want to do and leave me with overhead once I purchase a P2000.

Next step is to decide on new HDDs for media storage to populate my ThinkServer SA120.

And for anyone that is interested in the Rosewill case... it's actually very nice for the money and anyone looking for a short depth rack mount case should consider it IMO. The internal drive holders and support bar are all easily removable just by unscrewing the modules all individually (support bar, and drive holders are all separately removable).


 

maze

Active Member
Apr 27, 2013
576
100
43
I’m in the process of doing a rework of my plex server. Adding a p2000 to my D-1521 and doing passthrough to a Windows VM handeling the plex. I have to say though.. it seems like plex devs are either looking more at their new player (current plex pass beta for appletv) or just put the gpu transcoding (both decode and encode) Down the priority list.. afaik, only the encode is supported in gpu as things are now..

Am i misunderstanding, or is emby actually quite a bit further along in this process?
 

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
According tot he posts I've seen, it's a matter of how complicated the change is for ffmpeg and the handlers that call it within PLEX for all associated formats. It's not as easy as replacing a library/build command and flipping a switch. Their use of ffmpeg is very custom, and the implementation inside PLEX for other things that relate to it require quite some testing to make sure it's stable.

From what I've gathered, they're currently waiting on ffmpeg to recognize/integrate the changes upstream before anything can happen on the PLEX side. So it seems it will still be some time before we see this working 100% on Linux PLEX applications.

PLEX can be quite frustrating when you have issues as while it's a product you CAN pay for you would think that they have support for issues. I've come to the realization that this is simply not the case. Most of the time larger issues will get attention eventually, and you hope that someone else has had the same issue(s) you have and there are people smart enough to dig into the issue with you and you might get some answers.

But really it doesn't work that way most of the time.

That said, I have to give it to PLEX. It's a fantastic program, and what you get out of it (if you pay or not) is quite astounding. Emby seems to have it's own set of problems from what I have read (over the course of PLEX users screaming they will leave for EMBY or another media solution) so the grass is certainly not greener.

If I have to move to Windows for a while to fully utilize a P2000 or similar then that is just what I will have to do for now. I hope there is a way to keep the DB as that is really my only concern. Once they get around to fixing the Linux side I will move back.
 

caplam

Member
Dec 12, 2018
58
10
8
I have more or less the same concerns. Till recently i was not a plex user. I have to replace my proxmox nodes. My storage is on a synology ds1815.
plex recommend an intel qsync compatible cpu for hardware acceleration. None of xeon E5 family are qsync compatible (some E3 are but in this case no dual cpu system is possible)
But if you choose gpu acceleration qsync is deactivated. So my guess is a big E5 v2 can software transcode (2K passmark for one 1080p stream) and adding a gpu will help.
For now i have setup my plex server in a lxc container. Cpu is an i3 4330 (integrated gpu). Hardware acceleration took 5min to be setup. I can't go any further as i lack ram.
For my case i think i will go with a xeon workstation (Z620 or equivalent) as i don't need big storage space in it.
 

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
Small update in case anyone is interested... yesterday I spent quite a few hours making the EATX mobo fit in the case.... ripping out incompatible case bosses (for std. standoffs) I was only left with 2 that are compatible. I purchased black nylon standoffs and drilled the case to suit. Unfortunately the nylon standoffs are just too cheap to hold up so I will be changing out for brass units instead.

Also had to modify the internal drive unit support/cases. While I didn't really think it through 100% as I was doing the modding elsewhere I should have done a few things different but for what I need the 5.25" bay to hold (icy dock SSD dock/adapter) it will still work out just fine.

Everything will have to come apart again to install the rear IO shield (still waiting on that) and new brass standoffs and complete array of Noctua fans (2x 80mm and 1x 120mm)

All in all, I'm pretty happy to have a full 2x CPU + EATX system in this very short (15") 4U case.

Anyway on to some pics...

Drive holder modded:


New MB standoffs in place and old/original ones removed:


Mobo test fit:


NOCTUA coolers in:


PSU in, and everything connected ready for testing!


Case sitting on the SA120 (waiting on NOCTUA fans for that thing too because it's WAY too loud for condo use), which is all sitting on a LACK table of course for testing purposes!


After having to reset the IMPI password, and flashing the on-board controller to IT mode, and installing ESXi 6.7U1 it was almost 1AM, but the machine is alive and I'm excited!

Next up is deciding on what hard drives and how many to fill up the SA120 with and then building the custom rack mount solution for the "closet" to hold everything. Then plan the migration from the old server + storage system. So still lots of work to do.
 

NOTORIOUS VR

Member
Nov 24, 2015
78
7
8
43
So what do you do when you want to make rack mounted case to hold HDD's but no one makes a decent/affordable case?

You cut and weld a case of course!

Purchased a RSV-L4500 case with the intention of filling it with Supermicro ME35 HDD caddys to create a 15 drive JBOD case. Already have 2 of the ME35's. I was restricted in length to a max of 20" so the case would need to be cut/shortened, took it down to 18" and tack welded the rear panel in place. Looks pretty much factory new, just shorter!

Factory new:


Cutting out the fat:


Shortened with the stock length cover on top to see the difference:


Sorry didn't take any final pics, but I can snap some when I do the assembly of the drives/PSU etc.