unRAID Behemoth Build (Stage 2)

[IamSpartacus]

like only 2/3 of those those pretty much, I'm trying to get rid of unifi-video, only have 1 camera left for that (just going ring all around so I dont have to do local storage).
I migrated all of mine from esxi, so my containers are in a Ubuntu VM still and I've been lazy so far on migrating them into direct containers.

I had to do that same migration (migrated from 4-node ESXi vSAN cluster to single unRaid box). It wasn't bad. So much easier to maintain on Unraid natively then from within a VM.

As for UniFi video, I just don't trust my surveillance data being outside my network.

 

[T_Minus]

I had to do that same migration (migrated from 4-node ESXi vSAN cluster to single unRaid box). It wasn't bad. So much easier to maintain on Unraid natively then from within a VM.

As for UniFi video, I just don't trust my surveillance data being outside my network.

Boggles my mind how so many people are "OK" with this.
Afraid to see where things will be in 10 years

 

[IamSpartacus]

Boggles my mind how so many people are "OK" with this.
Afraid to see where things will be in 10 years

Yea it's crazy. If your internet cuts out or power goes out you have no video footage. I don't need internet at all and if the power goes out I have at least until my UPS dies to get footage.

 

[PigLover]

Yea it's crazy. If your internet cuts out or power goes out you have no video footage. I don't need internet at all and if the power goes out I have at least until my UPS dies to get footage.

+1 to this. Plus the definitely non-zero chance that Ring’s servers get hacked and you lose control of your video (both live and stored).

Also, don’t forget that they share with the popo (yeah, I know, only if you agree to it - wanna buy a swamp?). Don’t get me wrong, I’ll share freely with the cops - if it suits me and I control what gets shared.

 

[Spartacus]

+1 to this. Plus the definitely non-zero chance that Ring’s servers get hacked and you lose control of your video (both live and stored).

Also, don’t forget that they share with the popo (yeah, I know, only if you agree to it - wanna buy a swamp?). Don’t get me wrong, I’ll share freely with the cops - if it suits me and I control what gets shared.

I know there are folks thats a concern I'm willing to sacrifice for the convenience (I do have all the ring shiz on a separate vlan though).
All my cameras are external and if that should happen I'm ok with it worst thing they get is my address and watch me walk to my car shirtless outta that deal, no one wants to see that anyway.
Any videos that I'm concerned about saving I download immediately since they only keep them 30 days, only needed to happen once so far.

I have reasonable power backup that lasts longer than the network hub power (about 15m) in my neighborhood.
My grid is pretty solid overall anyway, and suddenlink with my own modem and equipment is solid other than their planned maintenance thats about 30 min once or twice a quarter.

 

[TLN]

I'm quite curious about what seems to be custom box where you have extra drives. Can you provide more info on that?
I quite like the idea, since I cannot find small and adequately priced case for my needs

 

[Spartacus]

I'm quite curious about what seems to be custom box where you have extra drives. Can you provide more info on that?
I quite like the idea, since I cannot find small and adequately priced case for my needs

Its actually a vanilla Corsair 750D, they design the drive holders to link together the sizing is perfect for 3 tall to attach to the 5.25” enclosure.
Heres what it looks like with the 2 it normally comes with:

Corsair sells extra parts including the hdd enclosures so I bought 3 of them here: HDD upgrade kit with 3x hard drive trays and secondary hard drive cage parts (Graphite 600T, 730T, 760T, 780T) (Obsidian 450D, 650D, 750D)

There are folks that have done full stacks of 4 as well but I liked the way it attached to the 5.25” bay mount and wanted to use a 5x 3.5” hot swap for that area anyway.
Only get a few extra drives and you lose a bit of stability when its not attached, wasn't worth it IMO.

Edit: I’ve been really happy with the layout and especially the low noise which was my primary goal.
The disk amount and spacing is above par too. Its perfect for 5400rpm drives temp wise always <45c even during the long parity checks.

 

[Spartacus]

Oh I’m dumb you meant the stand alone box, it was done by @Spotswood

Spotswood website: Home

He makes cases/frames etc. you can message or email him if you see something you like on his site or design something custom.

 

[TLN]

Yeah, that's exactly what I was looking for. Interesting info there on a website. Now I have some ideas....

 

[Spartacus]

No problem, its a really neat concept for a completely customizable case essentially.
Its not cheap, but not overly pricey either, all depends on the size, materials, and addons you want.
The custom box I had made was just under $100, he does a great job of packaging and providing extra screws, parts etc, even a screwdriver for it.
Instructions are a little lacking since its custom build (so its expected), but its easy to figure out if you're good with legos.

 

[Markess]

@Spartacus , how are you liking this build 7 or so months down the road?

I'm wanting to update my Unraid set-up from storage only to include more Apps, VMs and etc, and consolidate some VMs currently on an older Xeon DP system. But I'm still on the fence over newer E3 or something like the X10SRH-CF in the Great Deals thread (and that you gave me some advice on...thanks again).

Then I remembered you had this build thread up and thought I'd ask if you're satisfied? Enough horsepower to run everything you wanted..but not overkill? Anything you'd change?

Thanks.

 

[Spartacus]

Thanks for the ping on this, I needed to update it a bit, also moving into a new house next month GIGAPOWER INTERNET WOOO!~

So I love the build still overall, I made 2 major changes ~ docker'ized all of my stuff into containers and made the primary box plex only, the secondary box is all my nbz stuff and backups stuff.

TLDR Things I plan to change: Less ram (64), lower power CPU (1/2 TDP & only 8c), get NVME for the low write load use cases, and get hotswap.

Due to the above things I would change/am changing:
1) Less ram, I ended up migrating everything to docker and it uses minuscule amounts of ram in comparison. My full stack is using about 8-12 gb ram up only up to about 30-40GB when transcoding multiple streams (I use the ram for transcode temp repository). I'm going to sell off some of the sticks down from 128gb to 64gb (and maybe sell all and replace with 2x 32gb to reduce the power on both).

2) The CPUs I got are way overkill for sure, once again with full containerization even with multi stream transcoding it rarely hits above about 50% on the 2660 v3, the 2690 v3 barely breaks a sweat. I'm planning on picking up a pair of E5-2618L v3 (or something similar) to replace them with to reduce the power and a Quadro M2000 to passthrough for the main plex box to supplement hardware transcoding.


3) I like the raid 10 SSD, but its just overkill and takes 4 drive slots and I dont need 1TB of space on it. Due to the main box only having a Plex docker container and a few misc its using less than 100gb even holding the backups of the VM/containers of other box. I'm keeping the SAS SSD on the backup box for now as the 5PB endurance on those drives is perfect for their download/unrar encode/decode use case. I picked up a Supermicro aoc-slg3-2m2 ($40) and two M510 corsair 250GB NVME drives (45$ ea/ on sale at amzn) to use with my X10SRH-CLN4F board to replace the 4x evo 860 500gb drives as cache/vm repo, costs less and 4x the speed while still redundant.

4) I decided I really liked having the room for all the drives, but I want them to be hotswap, so I found another antec twelve hundred v3 on clist and am working on procuring another 5x 3.5" to 3x 5.25" supermicro enclosure so I can migrate everything over. Theres alot less vibration with those units as well (but costs alot more overall case wise about $400-500 in the end to reach 20 drives), I considered the Fractal 7 XL but was concerned with all the amount of vibration from the free standing attached drives.

That all said still very satisfied overall with the build and unraid in general.
Phew ok let me know if you have any other questions.

 

[Markess]

Thanks for the thorough rundown!

You've probably heard the carpenter's adage, "Measure twice, cut once"? When it comes to hardware, I'm the absolute king of "Buy twice, keep once". With this build I'm determined to break myself of that, research it properly FIRST, and not buy every shiny thing I see. So, a couple questions since you're offering to take them. Plus some thinking out loud (to see if anyone thinks my thinking is more faulty than usual).

You mention in the top post that you've had 4 drives for Cache/VM, how do you have those configured in Unraid? When I look at the Unraid forums, & documentation, I understand it to read only one Cache; two drives will mirror, and more drives "gives additional redundancy" without increasing capacity. So if I put four 250GB drives in "Cache" (for Cache and VM/Containers), I'd only have 250GB of usable, but highly redundant, space. It sounds like you have more than 500GB on yours (you mention 1TB & RAID 10 immediately above), so I guess its configured differently? ..and/or I just read the Unraid documentation/forum posts, and/or your post wrong.

Containers. That's some good stuff to know when selecting hardware! I've been staying away from containers, but your performance & utilization observations are a reason to dig in and figure them out. I've got 32GB on hand to use for this build, be it E3 or E5. Running things in containers sounds like a way to make the most of it.

The CPU observations are great, thank you for those. I was considering pulling the E5-2667 v3 (8C, 3.2Ghz) from my desktop for the Unraid build and getting a faster Broadwell CPU for the desktop. But it sounds like more cores and less clock is a more effective way to go here?

NVMe: Are you changing to that for cache primarily because you want the bay space, or because you find the 860 Evos aren't performant enough for you in Unraid? Right now in its "Storage only" role, my set-up has a mirrored pair of 250GB 850 Evos for Cache that I planned to keep. But I've got some Samsung 250GB NVMe drives (from the $16/each deal a while back) I could use if that made more sense. And if that was the case, it would definitely favor the E5 solution, since there's limited lanes and no bifurcation on an E3 board.

Are you backing up the primary to the secondary box, or to something/somewhere else? I'm going to have to come up with a backup solution that makes more sense than an external USB drive with one really really big disk in it.

Oh dang. I wrote another novel. I have way too much time on my hands right now!

Thanks again.

 

[Spartacus]

So lets start with the cache raid its built in but a little hidden on the option: FAQ for unRAID v6
You can choose raid 1 (default), 0 or 10 and raid 5/6 are in beta (only for brtfs cache) they added a nice gui selection in 6.8.3 recently I did it via command line previously.

I currently have 4x 500gb sata drives I just feel like its a bit of a waste of space both physical and digital, I didn't want to sacrifice a 5.25" slot for a 4x ssd enclosure when I have an extra x8 pcie slot (and the board supports bifurcation) so the NVME adapter board was the way to go. These 860 evos were left over from when I migrated from my 1635AX nas so didnt pay any extra.

Yes, I highly recommend the containers, what I did was first build some VMs running containers to get familiar with how containers work and to migrate Plex, unifi controller, unifi video, etc and keep all of my current watch counts/lists etc the guides are pretty straight forward (but it takes a while to copy Plex due to tons of tiny metadata files). From there I copied the container files and configuration options into a new template on unraid and they started right up pretty much with only a few minor tweaks.

On the CPU thats really up to you, frankly its not bad each box is using about 150w 95% idle, not horrible, but definitely feels like wasted usage when I dont need that much power that often and a much lower usage CPU is cheaper and still decent compute. I originally was thinking I wanted the highest frequency possible with good cores for the VMs, then containers happened.

I actually converted my array back to raid 1 from raid 10, didnt notice a bit of performance difference on my primary box with those sata drives.
I just did the NVME because e-peen and they drives I got have better endurance than sata, if the sata is working and doesnt feel slow keep it IMO.

My primary and secondary are clones from a array perspective, the cache is similar (both raid 10), I'm running DirSyncPro in a container to clone my media changes nightly. It was one of those when I transitioned to unraid I needed the array built up first, so I had to buy all the drives anyway, when I retired my qnap nas I then had 10 drives with no use, so build another box! (sitting at about 55TB used so not many externals available for that, gonna need to add another pair of drives soon too).

 

[Markess]

So lets start with the cache raid its built in but a little hidden on the option: FAQ for unRAID v6
You can choose raid 1 (default), 0 or 10 and raid 5/6 are in beta (only for brtfs cache) they added a nice gui selection in 6.8.3 recently I did it via command line previously.

Now how did I miss that? I have a hard time finding stuff on that forum.

And thanks for the advice and shared experiences on the rest. I can see a half dozen things I'm going to have to rethink already.

 

[Spartacus]

For sure, I just put together a build for a guy that was thinking about my QNAP actually and ended up steering him towards unraid instead.
Sub $1000 (without drives) if you were looking for ideas on builds.

CASE:
$300 Supermicro 846E16-R1200B BAREBONE 4U Server BPN-SAS2-846EL1 24x TRAYS W. Rails 818240931370 | eBay
They're based out of Dallas near your parents so you could save the 70$ shipping cost, and discuss with them replacing the PSU with the SQ version they're about half the noise of the regular ones.
The backpane can handle regular HDD fine, but multiple SSD will take up all the bandwidth you're using NVME anyway, but just FYI.

Motherboard:
https://www.ebay.com/itm/Supermicro-X10SRH-CF-Single-Socket-LGA2011-Motherboard/123880847605?ssPageName=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

$125? ~ I made an offer of 100$, they countered at 135, I countered at 125 they accepted (135$ total with tax)
You’ll have to either deal without the I/O plate, print one on a 3d printer, or order a cheap one from china.
You will have to flash the controller to IT mode its pretty straight forward and there are plenty of guides (the onboard lsi controllers are difficult to brick)

CPU:
Intel Xeon E5-2660v3 10-Core 2.6GHz / 3.3GHz LGA2011-3 CPU SR1XR 72931647104 | eBay

70$ (maybe less? Make them an offer) ~ One of the better bang for buck v3 cpu 2.6ghz with 3.3ghz turbo 10c/20t @ 105w TDP

Ram:
SK Hynix 16GB 2RX4 PC4-17000 DDR4-2133P-R HMA42GR7AFR4N-TF | eBay

$160-180 ~ If you wanna buy 4 to total 64gb you could likely offer about $40 each and see what they counter at. (this should be on the compatibility list for the above board)

NVME Card:
https://www.ebay.com/itm/Supermicro-aoc-slg3-2m2-pcie-adapter-for-nvme-ssds/283847616513?ssPageName=STRK:MEBIDX:IT&_trksid=p2057872.m2749.l2649

$35 2x NVME adapter (required bifurcation motherboard support, which the above board does)
This will work with your existing NVME PM983 drives.
You will potentially have to configure the slot for 4x4 configuration in the bios to get the nvme adapter card to work.

8643 to 8087 cable:
$16 - https://www.amazon.com/CableCreation-Internal-Cable-SFF-8643-SFF-8087/dp/B013G4FL0A/ref=sr_1_3?crid=3LPI4J8EFT3JM&dchild=1&keywords=8643+to+8087&qid=1587865783&sprefix=8643+to,aps,279&sr=8-3

CPU Cooler:
$65 - https://www.amazon.com/Noctua-NH-U12DX-Premium-Cooler-LGA20xx/dp/B00DWFQ42I/ref=sr_1_1?dchild=1&keywords=noctua+narrow+ILM&qid=1587855809&s=industrial&sr=1-1-catcorr

You need a specialized narrow ILM cooler for the cpu cooler and I <3 quiet so noctua is the way to go, or for about 10db higher and $20 less the dynatron isn’t horrible either:

https://www.amazon.com/Dynatron-Coo...narrow+ILM&qid=1587855851&s=industrial&sr=1-5

About $850-900 ish out the door with taxes total

 

[IamSpartacus]

@Spartacus so funny to see that you put Plex off on it's own. I just did the same thing with a new Xeon E build. Using that strictly for letsencrypt, plex and my home domain controller with my EPYC box running all the rest of my containers and all my workstation VMs .

And @Markess, can't recommend running containers on Unraid enough. You'll never go back.

 

[Markess]

Thanks for all links! I had no idea that the aoc-slg3-2m2 was so cheap! I had visions of it costing an arm and a leg and having to resort to a two slot riser I had in the parts box with a couple cheap NVME to PCIE drive adapters sticking out of it sideways!

This thread and your list got me thinking about where to put the E5 board if I go that route. My Unraid is in an SC933 right now, which I'd retrofitted with a PWS-605P-1H which isn't SQ, but is still pretty quiet. But, the SC933 its 3U, and Narrow ILM cooling for 3U is pretty dismal. Other than 2U stuff, I think there's a 92mm Noctua, and that's about it. But I've also gotten a 4U Chenbro case recently. I'd had a different project in mind for it, and its not hotswap, more like a rack mountable tower case on its side. But, I guess that also means it will have quieter air movement. At $40 shipped off Ebay, it was supposed to be bare, but it came with a bunch of stuff inside, including a Corsair AX760i Platinum PSU That would work nicely for a power supply, and I could use one of the larger Noctuas.

I'd have to put the other project somewhere else though. Ugh. More decisions.

 

[Spartacus]

Thanks for all links! I had no idea that the aoc-slg3-2m2 was so cheap! I had visions of it costing an arm and a leg and having to resort to a two slot riser I had in the parts box with a couple cheap NVME to PCIE drive adapters sticking out of it sideways!

This thread and your list got me thinking about where to put the E5 board if I go that route. My Unraid is in an SC933 right now, which I'd retrofitted with a PWS-605P-1H which isn't SQ, but is still pretty quiet. But, the SC933 its 3U, and Narrow ILM cooling for 3U is pretty dismal. Other than 2U stuff, I think there's a 92mm Noctua, and that's about it. But I've also gotten a 4U Chenbro case recently. I'd had a different project in mind for it, and its not hotswap, more like a rack mountable tower case on its side. But, I guess that also means it will have quieter air movement. At $40 shipped off Ebay, it was supposed to be bare, but it came with a bunch of stuff inside, including a Corsair AX760i Platinum PSU That would work nicely for a power supply, and I could use one of the larger Noctuas.

I'd have to put the other project somewhere else though. Ugh. More decisions.

No problem, so on the aoc-slg3-2m2 make sure your board is listed in the compatibility list otherwise only 1/2 ports will work as it wouldnt be configurable for the 4x 4x bifurcation. AOC-SLG3-2M2 | Add-on Cards | Accessories | Products - Super Micro Computer, Inc.

Its a pretty large list so high likelyhood most boards chosen will work.
Oddly enough the board I got to fit in the corsair 750D isnt part of that list the X10SRA-F (which is fine, I bought one of those X10SRH-CF.

 

[nthu9280]

Great thread.
@Spartacus - just a heads up. Not sure if you saw my post on the other thread. Don’t go with ‘L’ processors. If you can get cheaper v4 that could save some power during idle. But the idle power draw of L and regular cpus in the same gen is essentially the same.

@IamSpartacus you are missing important docker container - bonarr