Bulk Optimize Media for Plex Direct Play

Discussion in 'NAS Systems and Networked Home and SMB Software' started by IamSpartacus, Jul 14, 2016.

  1. IamSpartacus

    IamSpartacus Well-Known Member

    Joined:
    Mar 14, 2016
    Messages:
    1,421
    Likes Received:
    279
    I'm looking for suggestions for the best way to go about converting the bulk of my media libraries (2000+ movies, 14,000+ TV episodes) into 4Mbps 720p mp4 files with AAC audio. I've found this seems to be the best format for files to be in for direct play compatibility amongst all Plex client apps/devices. I would then use these files to create new separate media libraries that my remote users would only be able to access while I'd still use all the original files for my own local/remote streaming uses.

    I've considered using the built in Plex Media Optimizer but from what I've seen the it doesn't take full advantage of my hardware. No matter what transcoder settings I configure I never see the CPU usage go above 1000% (barely 60% usage) when running top/htop in Linux. I'd love to be able to make full use of my dual Xeon D-1537 servers which both have full access to my media libraries.

    I'm wondering what some others may have done to bulk convert their large media libraries. Would love to discuss.
     
    #1
    Chuntzu likes this.
  2. Deslok

    Deslok Active Member

    Joined:
    Jul 15, 2015
    Messages:
    823
    Likes Received:
    83
    I've used formatfactory to good result, it will handle up to 4 transcodes at once, one thing i've run into is running out of Disk I/O before CPU in some instances.
     
    #2
    gigatexal likes this.
  3. cptbjorn

    cptbjorn Member

    Joined:
    Aug 16, 2013
    Messages:
    100
    Likes Received:
    19
    To what end? My libraries are around that size and I do 500GB-1TB per month to a half dozen active remote users and I've found on-demand streaming to be stable for a few years now.
     
    #3
  4. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    840
    Likes Received:
    225
    I would wonder the same thing except that Plex doesn't have good controls for limiting bandwidth to remote users. The user chooses the bandwidth on their end, rather than the server saying I have 100Mbps, and I'll balance the load across X number of remote users.

    I have a large Plex library with remote users and I frequently run out of upload bandwidth way before I run out of processing power. The issue is that a remote user can dial in a high desired bitrate or try to direct stream a high bitrate encode and kill it for other remote users.

    This method would allow @IamSpartacus to share only the re-encoded libraries with remote users to "know" that they are all using the correct settings all the while, putting almost zero CPU load on his server (other than the MASSIVE amount of time he will spend transcoding).

    @IamSpartacus, I can't remember what OS you are using on this server, but if it's Windows, these scripts work well. Otherwise, I would whip up a BASH script to find all .mkv, .mp4's, .avi's, etc. in the current path (let's say /storage), create a new directory for your transcoded files (i.e. /transcoded/movies/Movie Name (Year)), and use handbrake-cli with the parameters you want to output to this directory.

    That being said, I'm just waiting for server controls, as it's accidentally shown up in PMS Plexpass versions as an option, so I hate to waste the time and power transcoding lots of files if it's in the near future.
     
    #4
  5. IamSpartacus

    IamSpartacus Well-Known Member

    Joined:
    Mar 14, 2016
    Messages:
    1,421
    Likes Received:
    279
    @cptbjorn & @rubylaser My motivation actually has nothing to do with trying to limit my remote users. I have 300Mbps upload speed and I'm not even using 1/3 of that during my high volume streaming times. The problem is that my users are primarily family members with a few close friends and none of them are technical AT ALL. Thus they stream at the default setting their clients are set which I'm seeing is usually 4Mbps 720p. Most of my media is 1080p and even the 720p stuff is higher bitrate (8Mbps+) so everything is being transcoded. While my server has the CPU power to handle all the live transcoding, I'd like to free up some of that CPU power for other services.

    I'm running Plex in dockers on Ubuntu Server 16.04 but I can spin up any OS I need to on my vSAN cluster that will make this job easiest.
     
    #5
    gigatexal and Chuntzu like this.
  6. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    840
    Likes Received:
    225
    I wish I had a connection like that available at my home... oh well :) I completely forgot that this was running in Ubuntu, that makes BASH scripting super easy for stuff like this. You could use something like this script. Something like this script.

    Code:
    # !/bin/ksh
    mkdir -p /transcoded/movies
    cd /transcoded/movies
    find /storage/movies -name '*.avi' -o -name "*.mkv" -o -name "*.mp4" | \
    while read filename
    do
          directory=${filename%.*}
          filename=${filename%}
          mkdir /transcoded/movies/"$directory"
          echo "Starting transcode of $filename"
          echo "==============================="
          # Encode stuff here...
          transcode-video --target 720p=4000 --mp4 "$directory"/"$filename"
          echo "$filename completed transcoding successfully moving to destination"
          mv 
    done
    
    ** Caution: untested code written on a phone, please test with a few test files. May eat all your data and burn down your home :)
     
    #6
    Last edited: Jul 14, 2016
    gigatexal likes this.
  7. IamSpartacus

    IamSpartacus Well-Known Member

    Joined:
    Mar 14, 2016
    Messages:
    1,421
    Likes Received:
    279
    Thanks for the link and example @rubylaser. I'll give this a look over tonight and see what kind of luck I have in testing.
     
    #7
  8. Chuntzu

    Chuntzu Active Member

    Joined:
    Jun 30, 2013
    Messages:
    383
    Likes Received:
    98
    I have done exactly what you are looking to do. I will do a write up here shortly. Long and short is windows hpc server 2012 and ffmpeg with or without quick sync and nvenc.

    Sent from my SM-N920T using Tapatalk
     
    #8
  9. wiretap

    wiretap Active Member

    Joined:
    Jul 14, 2015
    Messages:
    125
    Likes Received:
    66
    Another option to avoid re-encoding all your media would be to give Emby Server (and the associated apps) a try. I switched over to it a while back, and the transcoding features they offer are the best I've used for a media server. It supports Intel QuickSync and Nvidia NVENC, so if you have an Intel processor or add a dedicated video card for transcoding purposes, it will cut your CPU use down to nil when users remote in and start playing content. You can also set the maximum bitrate allowed for transcoding so you don't kill your upload as well. Another nice feature is that it will transcode live TV too, so I can access all my TV stations right within the interface. I put all my transcoding cache on a cheap SSD, so it takes a few seconds less to start the transcoded video stream than with a normal HDD. Anyhow, with the hardware accelerated transcoding turned on, I have tested up to 4 concurrent streams and my CPU use wasn't even over 40%. (Xeon E3-1240) I also have my Emby Server setup in an ESXi VM so I can control overall resources given to the guest OS so I don't bog down the whole server.
     
    #9
  10. IamSpartacus

    IamSpartacus Well-Known Member

    Joined:
    Mar 14, 2016
    Messages:
    1,421
    Likes Received:
    279
    I'd be very interested to hear more about what you've done.


    I gave Emby a try a few months back and while there are many features I do like about it, the clients are just not there yet. Furthermore all family members and friends have been using Plex for years and as I mentioned in my OP they are not technical at all. Getting them to install and learn completely different app is all but an impossible task.
     
    #10
  11. Chuntzu

    Chuntzu Active Member

    Joined:
    Jun 30, 2013
    Messages:
    383
    Likes Received:
    98
    Hpc server 2012 update 3 is out and is super easy to install. It is just and executable. I made a power shell script the scans a directory for video files based on size and extensions and then did a forward command that passed each file into a job create command and submitted it to the job manager on the head node. Hardest part was the power shell command it that only took about an hour to whip up. It was a bit difficult since I haven't done a lot of power shell and need to figure out where to put single and double quotes. Still don't have a way to identify quick sync with the job submit command so I just have two commands set up one that goes to the nuc clusters and another that goes to dual e5-2670 nodes. But I figured this out for the same reason as you 720p with primary stereo audio and secondary 5.1 audio Plex copies. It works great. With the nuc clusters I set them with medium ffmpeg settings with constrainted bit rate min and max. All videos look more than fine, but x264 look better in some fast scenes on some of the larger films. But I am being very nit picky. I ran 1 encode per thread and across 16 nucs and 4 dual e5-2670 nodes. So able to do 192 encodes simultaneously it worked really fast. Next thing I may try is nvenc x265 and see if I can compress all these files down using cheap Geforce cards. Just picked up a 960 to play around with. Sucks that the geforce cards can only do 2 transcodes per node even if you have multiple cards. Would be nice to have a few m40 cards and those would crank a ton of fps each.

    Sent from my SM-N920T using Tapatalk
     
    #11
  12. MountainDew

    MountainDew Member

    Joined:
    Oct 19, 2015
    Messages:
    188
    Likes Received:
    7
    There's a good thread on reddit about this as well here. There's mention of a SickBeard app that seems to have good results.
     
    #12
  13. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    672
    Likes Received:
    233
    I'm sure there's plenty of tools to do this already but I think it's more fun to script it B^)

    I've used a similar setup to rubylaser's script method that just loops through files and runs them through ffmpeg/avconv. ffmpeg's a ludicrously powerful tool and as such has a panic-inducing manpage but setting it up for batch encoding a bunch of files with the same settings is pretty simple. I'll typically do the audio and video separately and then mux them into an MKV afterwards but a transcode command like this should get you started:
    Code:
    avconv -i input.mkv -sws_flags lanczos -vf "scale=1280:trunc(ow/a/2)*2" -an -c:v libx264 -crf 23 -preset medium -tune grain -threads 2 -c:s copy output.mkv
    -i specifies the input file
    -vf is a list of video filters and the order to apply them in (and the really powerful part of the program)
    Only one video filter being used here to downscale to 1280x??? (and make sure the vertical res is a round number). We set a flag before that to opt for lanczos scaling
    -an means don't process audio. To just copy the same audio stream from your input file I imagine you'd just go -c:a copy
    -c:v means the video encoding bit using x264, CRF of 23, preserve grain, limit number of CPU threads (I'm not in a hurry and don't want the system getting hot and ramping up the fan if I can avoid it). There's a bajillion ways of configuring x264 but this is one of the simplest for a quick'n'dirty transcode.
    -c:s copy means copy over any embedded subtitle streams directly
    ...and then chuck everything into output.mkv

    Doing things like crop detection and deinterlacing make things more complicated (I originally did this as a way to batch-encode a load of DVD VOBs to get some WAF points) but if you're already happy with your source files you shouldn't need to worry about stuff like that.
     
    #13
  14. Keljian

    Keljian Active Member

    Joined:
    Sep 9, 2015
    Messages:
    429
    Likes Received:
    71
    Mediacoder will do it for you, and use quicksync or NVENC(if you have an Nvidia graphics card).. will even do h265

    It is Windows only though
     
    #14
  15. maze

    maze Active Member

    Joined:
    Apr 27, 2013
    Messages:
    451
    Likes Received:
    61
    Im doing exactly that.

    All media is run through my reencode script every night, and then (for now) i manually move it to the desired folders aswell. I have 5 family members running off my plex, with appletv3's and 4's running plexconnect / native plex.

    Anyway, i reencode all the media and they directstream so i dont have to use cpu time transcoding.. as you, i have a decent upload. All media is 720p.
     
    #15
  16. DonJon

    DonJon Member

    Joined:
    Apr 9, 2016
    Messages:
    52
    Likes Received:
    5
    There is a simple setting in Plex that sets the bit rate for internet streamed content and streamed content to the LAN users. Have you checked it? You can set like 720p 1Mbps for internet stream and 1080p 20Mbps for LAN.
     
    #16
  17. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    840
    Likes Received:
    225
    Unfortunately, this setting is from the client side not the server. Even it ut was server side, it wouldn't help with his original goal to reduce the CPU load by eliminating transcodes so he could use the machine for some other things.
     
    #17
  18. maze

    maze Active Member

    Joined:
    Apr 27, 2013
    Messages:
    451
    Likes Received:
    61
    The only real way to reduce cpuload is by doing direct streaming .. which means a container/subs that runs natively on the end device.

    For example.. ATV3 with plexconnect runs mp4 containers with embedded subs (from srt subs) as directplay. And the same goes for ATV4 with the Plex client.
     
    #18
  19. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    840
    Likes Received:
    225
    That's exactly what @IamSpartacus said he wants to accomplish above. It's not that he doesn't have the CPU resources or the bandwidth to handle the remote users, he just wants to free up his processor on his Plex host to do other things. So, he's asking for the best way to re-encode all his files so that he has 4Mbps, 720p versions of each that are compliant with all clients so he can just direct stream them.

     
    #19
    Last edited: Jul 20, 2016
    Chuntzu likes this.
  20. IamSpartacus

    IamSpartacus Well-Known Member

    Joined:
    Mar 14, 2016
    Messages:
    1,421
    Likes Received:
    279
    Thanks everyone for the suggestions. Unfortunately I've been super busy all week so I haven't had a chance to really read over everyone's suggestions and/or test anything yet but I will certainly do so in the coming days.

    @rubylaser has my intentions correctly illustrated. My goal is to reduce CPU load on a consistent basis and the only way to do that outside of educating all my family members/friends (not gonna happen) is to provide only content that will support direct playback on most/all Plex clients.
     
    #20

Share This Page