Clear Linux used exclusively for FFMPEG on Kaby/Coffee etc.

Discussion in 'Linux Admins, Storage and Virtualization' started by Allan74, Aug 14, 2019.

  1. Allan74

    Allan74 New Member

    Joined:
    May 15, 2019
    Messages:
    14
    Likes Received:
    0
    I figure that I could get the best response here to a simple question considering the level of Lin-Fu present on this forum from what I have seen.

    With many of us Windows Users leaving Kaby Lake/Coffee Lake platforms behind as our daily drivers for new AMD Zen2 platforms, could Clear Linux provide the Intel Optimized environment that on paper anyways, looks like an upgrade from Windows platforms ? (along with beating up on most other distros in the process).

    While I understand that Clear Linux is a swear-word in some circles, thanks to Telemetry, could this front end be used to breathe a bit of life back into aging hardware rather than throwing it to either the used market or the black holes that we call our 'IT Closets' ?

    Here is what I am hoping sounds reasonable....
    Clear Linux on a headless box that sits on my network and does nothing but rip and process/transcode my BluRay rips to feed my Plex Media Server, using FFMPEG as the main tool.

    Can anyone comment on Clear Linux ?

    Again, I don't plan to use it as a desktop, I simply want to take advantage of all of the Intel Optimizations found within.

    Thanks in advance,
    Allan
     
    #1
  2. BoredSysadmin

    BoredSysadmin Active Member

    Joined:
    Mar 2, 2019
    Messages:
    217
    Likes Received:
    51
    The biggest and heaviest task in your desired workflow is the video encoding part. Intel happens to have a dedicated module in SOME of its CPUs called QuickSync. While sacrificing "a bit" of video quality (that is open for discussion how much and if it's acceptable to you), it provides HUGE speed up in terms of speed of video encoding with EVC/h264 and HEVC/h265. Similar or better performance could be gained using some of Nvidia GPUs using NVENC subsystem.
    Now, In my honest opinion, for the purpose of video encoding, OS-level optimizations are minuscule compared to using a dedicated hardware encoding.

    My suggestion is to stick with a much wider user and supported Linux distribution like Ubuntu or CentOS. Both should support both hardware encoding subsystems.

    Hope that gives you some guidance.
     
    #2
    Allan74 likes this.
  3. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    1,048
    Likes Received:
    351
    I'm building my latest system to be a useful ffmpeg-mini-monster so I can move the bulk of my long encodes to my server instead of my workstation. It's not an apples-to-apples by a long stretch, but my debian machine is a good 10% faster than my windows machine (3.5GHz haswell vs. 3.8GHz haswell-E respectively) in the same ffmpeg task.

    If you don't care that much about quality, then QuickSync and other fixed-function encoding solutions are perfectly fine. If you want to keep stuff around for archival purposes, I think it's better off you spend the (CPU) time encoding them with a software encoder to get the optimum quality for any given bitrate. x264 --preset slower was, last time I looked at least, the best option in perceptible quality vs. bitrate vs. encoding time (although I'm investigating AV1 encoding currently which makes glaciers look fast). As I'm generally transcoding for archival purposes, quality is paramount for me.

    I've had a good play with the H264 and H265 encoders in my Raven Ridge chip (again via debian and ffmpeg) and whilst the speed is fantastic, the quality is mediocre compared to what can be done with the software encoder (and five times as much time). But having a nice headless system to queue up encodes on for a few days so you don't have to think about them removes any degree of impatience for me. Ultimately it's a question of what's most important to you, but I stopped staying up late for my encodes to finish 15 years ago :)

    What's the predilection with Clear Linux though? If you stick to pure software encoding it's probably a few percent faster than other linux distros simply by way of more attention being paid to the toolchain but it's quite immature for most other server purposes and (tinfoil hatter alert) I'd prefer something that isn't under the control of a corporate. At the moment I see it as more of an experiment in pushing the performance option than as a distro for everyday use and supportability - it's not really been around long enough to prove itself and this isn't yet its stated goal, so long term maintenance might turn out to be a pain.

    I'd second BoredSysadmin's recommendation of sticking with a tried'n'trusted distro with ffmpeg support since it'll make the overall experience of everything else that isn't dependent on pure CPU performance that much better. If hardware encoding works for you then the distro is doubly irrelevant.
     
    #3
    Allan74 likes this.
  4. Goose

    Goose New Member

    Joined:
    Jan 16, 2019
    Messages:
    5
    Likes Received:
    0
    Just on the quicksync thing, later version so of inte'ls hardware blocks are truly amazing. The kabylake one handles basically everything and is fast too.

    I use QSVENC via staxrip on windows and i can deliver 4k transcodes at realtime to 1.5x speed that are "visually" lossless and smaller than comparable x265 encodes. Animation particularly benefits, due to heavy use of repeated/static content/frames. The magic is the hardware engine doesn't seem to get bogged down with lots of reference frames.

    --codec hevc --quality best --profile main10 --la-quality slow --la-window-size 15 --bframes 6 --ref 9 --b-pyramid --fallback-rc --icq 17

    Whilst this doesn't help you too much as the ffmpeg hooks aren't that great unfortunately, maybe it will make you reconsider.
     
    #4
  5. Deslok

    Deslok Active Member

    Joined:
    Jul 15, 2015
    Messages:
    986
    Likes Received:
    101
    Taking a look around my inital hunch(clear linux offering superior I/O performance giving it an advantage) isn't accurate base on some testing phoronix did. They actually had a h264 benchmark that showed a very small variance between distros(within margin of error for normal tests) They used GCC to compile ffmpeg perhaps compiling it with ICC would change things instead? Debian vs. Ubuntu vs. CentOS vs. openSUSE vs. Clear Linux Post-Meltdown Performance - Phoronix
     
    #5
  6. Allan74

    Allan74 New Member

    Joined:
    May 15, 2019
    Messages:
    14
    Likes Received:
    0
    OFF TOPIC, but a reply to your post Goose, While I use QSV for temporary transcodes of TV episodes, I did find that that it had a real problem with 'fast action scenes' in animation. I found this weakness very evident while prepping some American Dad episodes (not particularly known for 'Super Hero' type battle scenes) @1080p for the server and regardless of bitrate/Multi-Pass/quality profile etc, ended up with obvious smudges...not artifacts, not banding....but a clear smudge covering a few colors that were close in fast action scenes. I now use CPU/Software exclusively for Animation.

    Back on topic....
    I guess at the end of the day, I am just trying to find a better home (and purpose) for my collection of aging hardware, as I can't justify shelling out for HEDT level performance just to rip and transcode content for my Plex Server (or so the wife tells me).

    In another post, I had posted the desire to consolidate things in my home network a bit, as all I seem to have are 2 users left and a ton of old hardware, other than the Kaby machine that I am on now, although starting to show it's age nonetheless and will soon see an upgrade.

    At this point, my 2 choices (without going out of pocket or sacrificing my upcoming desktop to transcode duty) is somewhere between an 'Optimized' Kaby.Lk Linux Distro or plain old brute force on a Windows based 12c/24t x5670 Westmere setup that I have been sitting on and not wanting to part with since demoting it from daily driver use before going to the Kaby.

    I am not a proficient Linux User, but the 'Optimized for Intel by Intel' sales pitch kinda had me hook, line and sinker and I figured that I could struggle through the setup, as long as I never had to actually use the box as a desktop machine after getting the initial software and scripts implemented and ready for headless duty.
     
    #6
  7. Deslok

    Deslok Active Member

    Joined:
    Jul 15, 2015
    Messages:
    986
    Likes Received:
    101
    I actually use an X5690 desktop for video editing and typical use today so i'm suprised you went to kaby lake for replacing it it isn't till just now I'd say cpu's are advanced enough to replace it.

    On an aside KBL isn't a slouch for video encoding, are you really pushing enough through it that every second counts? Most of us would put ubuntu or debian and go as far as setting up some scritps and watch folders to automate things but probably wouldn't even bother to recompile ffmpeg with ICC over GCC to squeeze out that performance change even.
     
    #7
  8. Deslok

    Deslok Active Member

    Joined:
    Jul 15, 2015
    Messages:
    986
    Likes Received:
    101
    upload_2019-8-19_10-57-28.png
    Apparently I can't upload a picture in an edit, I meant to include this as to why I haven't moved cpu's yet, you can see an i5-7600k(KBL) didn't beat my X5690's for video exports out of shotcut the new 3xxx series zen chips and 9xxx series CFL chips might finally.
     
    #8
  9. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    1,048
    Likes Received:
    351
    ffmpeg itself and most of the underlying libraries use a lot of inline assembly for the performance-critical parts, so there's little difference in performance across distros and across compilers.

    If the only performance-critical part of your workload is ffmpeg software transcoding, using Clear Linux won't make much of a difference. If you want to use hardware encoding (putting quality vs. speed arguments to the side for the time being), the performance of ffmpeg itself becomes less relevant and you care more about how well the distro supports the encoding hardware.

    What's the actual performance problem you're trying to solve? Do you need to use realtime transcoding or are you performing batch encoding for storing in your library? Do you already have an existing automated workflow on your (presumably windows?) existing setup?
     
    #9
Similar Threads: Clear Linux
Forum Title Date
Linux Admins, Storage and Virtualization ZFS on Linux 0.8.0 released! May 28, 2019
Linux Admins, Storage and Virtualization RoCE on Linux Mar 1, 2019
Linux Admins, Storage and Virtualization Power management for Ubuntu 18.04 Linux NAS ? Feb 6, 2019
Linux Admins, Storage and Virtualization Advanced Guide to ZFS on Linux? Dec 5, 2018
Linux Admins, Storage and Virtualization Slow SMB Read Speed, Fast Write | ZFS on Linux | Ubuntu 18.10 Nov 4, 2018

Share This Page