Is this overkill?

[afcurry]

Hi all.

Been down the NAS rabbit hole for a month now and still struggling to make the right decision, here's where I'm at:

Currently taking a workstation build to turn into a NAS server...

1. I'm realizing that I can no longer use the computer as a workstation if I go the TrueNAS route - correct me if I'm wrong here.
2. Main purpose of this build is to amalgamate all my data into one centralized storage center where I can have multiple editors working from at once.

Specs:
Processors: (2) Intel® Xeon®E5-2670 2.60 GHz
Cooling: (2) Gamerstorm Captain 120ex Liquid CPU Cooler
Motherboard: ASRock EP2C602-4L/D16
Graphics Card: Nvidia GTX 980ti Hybrid
Memory: Hynix 8x8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1333 (PC3 10600) Server Memory Model HMT31GR7BFR4A-H9 (128gb)
SSD: Samsung 850 Evo 250gb
HDD: (5) Ironwolf Pro 16TB
HBA: Testing multiple cards including LSI 9211-8i
Power Supply: Cooler Master Silent Pro M1000 (1000w)
Case: Phanteks Enthoo Pro Series
ETH: MikroTik 5-Port Desktop Switch, 1 Gigabit Ethernet Port, 4 SFP+ 10Gbps Ports
NIC: 10Gb SFP+ PCI-E Network Card

So, 3 questions
1.) How are these specs for a NAS build? Intended for speed, redundancy, editing, rendering - is it overkill and better suited to keep as a workstation?

2.) Am I limited to windows server if I want the system to double as a workstation? Obviously a ZFS system is preferred to protect my data

3.) Can I achieve similar NAS goals using an older pc I have laying around that is not built with server components?

Basic Specs:
Processor: Intel i7-930 Costa Rica 2.80 GHz
Motherboard: Gigabyte GA-X58A-UD3R
Memory: GSkill DDR3-1600 4Gx6 PC3-12800 (24gb)

Thanks in advance for your help, it's very much appreciated.

Reply
Report

 

[itronin]

did you cut n paste this from the TN/FN forums? Just curious, cause the "reply" and "report" linkages go back there. Did you read the FreeNAS (TrueNAS) primer that's pinned in the their hardware forum? - also take a look at the "will it freenas" thread

you speak of rendering so I am going to assume that the editing is of video? Or maybe images? If text files this is all overkill.

I recommend against trying to use a NAS as a workstation - can it be done running a hypervisor (XCP/prox etc) as the baremetal OS with devices passed through - sure. However you may be taking away performance from the NAS at crucial times and suffer performance hits and quite possibly have an unstable hypervisor from time to time. Could it be done running TNC and byhve with a windows guest - probably - again - not IMO a wise thing to do. Could it be done with TNS - but same caveats - more actually from a file sharing performance stand point. FWIW this is also a fairly advanced path to take - Is that part of what you want to learn here or are you looking for a "toaster" so you can focus on what is important to you?

I think a few more details about your use case, workflow, and what you expect from a performance standpoint are...
For example do you expect each of your editors to get near 10Gbe performance simultaneously working on files stored on the NAS? (hint: you won't) Or for example are you copying files locally working on them and pushing them back up to the NAS?

re. your old hardware
Running any kind of NAS without the benefit of ECC memory subjects you to a risk of bitrot during DIM (data in motion). Not saying the risk will be high - but its there - is that acceptable?

Lots of folks here can give you advice on building a system and I think we all will gladly do that.

I believe that if your use case, workflow, and performance expectations are a bit more defined you will get much more pointed advice (which I think you are looking for). The advice itself may not be (what you are hoping for), but it will definitely be more focused to your end goals.

 

[BoredSysadmin]

I'd also chime in that your old workstation is an electric power guzzler, the GPU would be useless for Truenas, and your Samsung 850 Evo 250GB would not be a good choice for zil/slog device. You COULD use it for arc L2, but with 128GB of ram and typical home usage, it would be a waste.

Multiple editors? Video editors? - you'd need much more SSDs or RACKs of spinning rust to get to the levels of performance needed for multiple video editors (assuming modern 4k resolution). For Composing/VFX work at 4k, you might need more than 1.2GB/s (that's large B) throughput.
I like this table showing how many streams a robust, specially-designed storage video for video editing can sustain, depending on the video format.
see table under "Intense performance header: - EVO Shared Storage Workflow Solutions & Performance — SNS (Studio Network Solutions)
and more thoughts on designing a storage system for video editing here:

What's behind video stream counts - Part 2 (SNS)

Shared storage systems are often compared on video stream counts. But what's behind these numbers? We show you in this two-part article.

www.studionetworksolutions.com

 

[BoredSysadmin]

re. your old hardware
Running any kind of NAS without the benefit of ECC memory subjects you to a risk of bitrot during DIM (data in motion). Not saying the risk will be high - but its there - is that acceptable?

I'd like to share a personal and anecdotal story. Over many years I ran and later expanded my whitebox DIY truenas. never had any major issues. The worst ones like boot drive failure (a few times over cheap USB stick drives) - were easily recovered by a new stick, os rebuild and config restore from a backup. It did NOT have ECC, and one of the pools I ran as async (at least it was UPS protected), as sync performance was abysmal due to some hardware issue. Never had bit-rot or any data loss even after many drives were near or failed.
First time I had a major (weeks of downtime is after I moved to QNAP nas and had a major hardware failure of the motherboard component - more on this later after I get the unit from repair)

 

[amalurk]

Lose the GPU and one processor for just a NAS.

 

[afcurry]

did you cut n paste this from the TN/FN forums? Just curious, cause the "reply" and "report" linkages go back there. Did you read the FreeNAS (TrueNAS) primer that's pinned in the their hardware forum? - also take a look at the "will it freenas" thread

I did indeed. Can't seem to find this primer you speak of. Will take a peak at "will it freenas", thanks for the suggestions!

Sounds like I'm better off building a DAS within my workstation, is that correct?

To give a little more perspective. I am spinning up a video production company, building a small team and looking to organize and protect footage and other data that's spanned across 10 ext drives. For the scale of the business currently, we can get by with editing locally, yet having access to a server remotely would be a nice. I am most concerned with creating a centralized storage server for our media. We can continue to edit off of separate SSD's, transferring footage and projects on/off as needed, no problem.

Does this lead me to create a hardware or software RAID and run it through Windows Server? Something else? I've created a RAID array in the past... other than that I'm pretty clueless. I appreciate you all taking the time to help me out here.

The factors important to me:

1. Utilizing this workstation for what I built it for (Video editing/motion graphics/rendering)
2. Centralized storage with ample editing speeds internally (4K)
3. Ability to access/transfer media remotely at reasonable speeds

Feeling something like a RAID 6 array may be best for my needs. For context, I have 80TB of new drive space, 15-20TB to be transferred on, which would about cap me out if I did a simple mirror. May need to buy another drive so I have 3 pairs...

 

[afcurry]

I'd also chime in that your old workstation is an electric power guzzler, the GPU would be useless for Truenas, and your Samsung 850 Evo 250GB would not be a good choice for zil/slog device. You COULD use it for arc L2, but with 128GB of ram and typical home usage, it would be a waste.

Multiple editors? Video editors? - you'd need much more SSDs or RACKs of spinning rust to get to the levels of performance needed for multiple video editors (assuming modern 4k resolution). For Composing/VFX work at 4k, you might need more than 1.2GB/s (that's large B) throughput.
I like this table showing how many streams a robust, specially-designed storage video for video editing can sustain, depending on the video format.
see table under "Intense performance header: - EVO Shared Storage Workflow Solutions & Performance — SNS (Studio Network Solutions)
and more thoughts on designing a storage system for video editing here:

What's behind video stream counts - Part 2 (SNS)

Shared storage systems are often compared on video stream counts. But what's behind these numbers? We show you in this two-part article.

www.studionetworksolutions.com

Power is indeed a concern with this system...better to be used intermittently eh? Are the Evo's not fast enough for zil/slog? What's the reasoning?

Thanks for the rest of the info, it is very helpful. Will check that workflow out. Will just be a couple editors to start, and not all necessarily working at the same time. Think it's best to focus on building my library locally first and then think about a NAS upgrade in the next 6-12 months.

 

[afcurry]

I'd like to share a personal and anecdotal story. Over many years I ran and later expanded my whitebox DIY truenas. never had any major issues. The worst ones like boot drive failure (a few times over cheap USB stick drives) - were easily recovered by a new stick, os rebuild and config restore from a backup. It did NOT have ECC, and one of the pools I ran as async (at least it was UPS protected), as sync performance was abysmal due to some hardware issue. Never had bit-rot or any data loss even after many drives were near or failed.
First time I had a major (weeks of downtime is after I moved to QNAP nas and had a major hardware failure of the motherboard component - more on this later after I get the unit from repair)

Thank you for your input. Forgive me, i'm a little unclear on the specific point you are making with this experience of yours.

 

[itronin]

Thank you for your input. Forgive me, i'm a little unclear on the specific point you are making with this experience of yours.

I think @BoredSysadmin was gently razzing me and not really you. Bitrot happens but neither often nor frequently in the grand scheme of things - tends to be more isolated to a system and when you get it - you usually really get it (IME). Its really a matter of probability. BSA was (I think) reminding me we aren't fanatics over here...

Linkages for you.

Will it FreeNAS
First Pinned Post - FreeNAS Hardware Guide
Core Hardware Guide

I'll say that in this part of server world we are not uhmm as fervently fanatical as the TNC true-believers about what you can and can't use for building NAS's nor should and shouldn't. We're more of the ohhhh look at this part - wouldn't it be dandy to use that! SQUIRREL!!!

Rule of thumb on ARC is max out your memory first as in memory ARC will be fastest. The data structures for ARC consume quite a bit of in=memory and when you throw a large SSD out there well you've on the other side of 80/20 and dimishing returns. Till you get over oh say 512GB memory is more where its at with ARC rather than an offboard device. TNC has plenty of metrics to look at how your ARC is doing and can help guide when/if you should go offboard.

Also with some of this it may be best to start simple rather than complex out of the gate. incremental changes that you can measure their impact is again IMO good business practice.

Oh there's that squirrel again - boot device - pick up a couple of $20.00 SSD's and run them as a boot mirror.

Regarding that EVO and ZIL - IMO its the wrong type of device for a ZIL/SLOG esp with the size of the files and frequency I'd expect them to get pushed back on the NAS. Honestly I would not use it for ARC either. I'd want some SLC or high endurance nvme for both ZIL or ARC.

S6eq Read/Write perf on your array as you spec'ed willy likely be in the 4-6 Gbps range. Random IO will be worse. Hence my original comment (you won't) and BSA's comment about large large NxN array of spinning rust to get 10Gbps perf. with SSD's SAS or NVME you are more likely to get to 10Gbps. FWIW that perforance estimation is somewhat NAS/file server platform independent. More a function of physics first then software/services.

I think you asked the best question though about whether you should even run a NAS OS or just use windows. If you will be managing the storage (and associated hardware) and you aren't experienced with anything but windows - then I guess why run anything other than windows?

With that said I beleive that TNC can be a toaster if you put "the correct" hardware into it, don't tinker with it, and don't push it to do too much beyond what it is designed to do (like running windows VM's with gpu pass through under bhyve! hahaha!).

'cause it needs to be asked:

NAS (or even shared storage is only part of this) What's your backup plan? Gotta have a plan - you'r'e talking about this as a business - 3 - 2 -1 gotta have a plan. Likely costs there too. Now your old workstation - as a backup (2 - 2nd copy) nas that is NOT always on... could work for a while. Cloud or colocation for (3 - 3rd copy). Gotta have a plan.

I've prattled on long enough and hopefully inspire some thought.

 

[Sean Ho]

To add to the excellent advice given and questions posed: it sounds like the current primary use case is archival storage, rather than multi-seat video editing.

If that's the case, the hardware requirements for your archival NAS become significantly easier. If you want an easy-to-maintain appliance, Unraid would suffice (basically JBOD plus parity); dual-parity would have the same space efficiency as raid6 / raidz2, but be more flexible with mixed-size drives and expanding the pool incrementally. TNC is another great option and quite battle-tested.

Your 2670 system would be a great fit for such a NAS: plenty of RAM, room for drives (10x3.5" if you add a $50 4-in-3 cage to the 5.25" bays), plenty of PCIe for the HBA, NIC, and potentially NVMe if you like. E5 v2 chips are super cheap nowadays. Demoting this box to NAS duty would be an excuse to get a fancy new editing workstation for yourself! But if budget is tight, the 930 could do the NAS job in a pinch.

With a basic NAS thus built, if later on your video editors want to move away from copying projects on/off the NAS, one next step is simply to leverage proxy media: the main clip library can be on the NAS (mounted over SMB/NFS), but editing uses reduced-resolution proxy clips stored on NVMe local to each editing workstation. Then the final render pulls from the NAS over the network (or renders remotely on the render farm, if your software supports it).

10GbE is still useful; even a single spinner can easily saturate a gigabit link.

 

[afcurry]

Thank you @itronin and @Sean Ho for your detailed responses.

I've spent enough time trying to figure this out so I feel the simple solution is the best one for my current purposes. It's great to know my system specs are solid for a standalone NAS, but I can't afford a fancy new workstation hence trying to figure out how to set these drives up in a way that provides an accessible archive while still being able to edit/render footage on this thing.

I suppose I'll stick with Windows for now to achieve this. I've been steered away from hardware RAID, so I need to figure out how to implement a software RAID I suppose. My RAID controller (Areca 1680ix-12) is old, but still running fine. Thought about using that, but if it fails I'd need to buy another one to access my data, and that controller is now discontinued. So I've bought an LSI 9211 to use as and HBA. https://www.amazon.com/dp/B07JZ6FYVC?psc=1&ref=ppx_yo2ov_dt_b_product_details

Demoting this box to NAS duty would be an excuse to get a fancy new editing workstation for yourself! But if budget is tight, the 930 could do the NAS job in a pinch.

With a basic NAS thus built, if later on your video editors want to move away from copying projects on/off the NAS, one next step is simply to leverage proxy media: the main clip library can be on the NAS (mounted over SMB/NFS), but editing uses reduced-resolution proxy clips stored on NVMe local to each editing workstation. Then the final render pulls from the NAS over the network (or renders remotely on the render farm, if your software supports it).

10GbE is still useful; even a single spinner can easily saturate a gigabit link.

When you say "930" what are you referring to?

Using proxy is a great idea which I will utilize if I find it necessary, thanks for the advice on that. Money is tight, maybe I'll return the 10GbE equipment for now if I'm mostly using this locally. Sure learning a lot, that's for sure!

 

[itronin]

@afcurry

windoze software raid - take a look at windows storage spaces and see if that will suffice.

I think @Sean Ho was referring to the i7-930 on your old "can I use this instead" to build a NAS system...

 

[afcurry]

@afcurry

windoze software raid - take a look at windows storage spaces and see if that will suffice.

I think @Sean Ho was referring to the i7-930 on your old "can I use this instead" to build a NAS system...

Ah, yes...the 930 processor, got it.

Just set up a RAID 10 and am getting around 350-400mbps read/write speed. Will try the storage spaces route and see what my speeds are like.

 

[Sean Ho]

yep on i7-930; it's old and draws a bit more power but should suffice. For HBA, ASR-7805 is $12 and doesn't need reflashing; there's a config option to flip it to HBA mode. Just need cables from SFF-8643 to whatever your backplane needs (or 8482 if no backplane).

ADAPTEC ASR-7805 8INT PORT 6GBPS SATA CARD for sale online | eBay

Find many great new & used options and get the best deals for ADAPTEC ASR-7805 8INT PORT 6GBPS SATA CARD at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[afcurry]

Storage spaces speeds are about 100mbps less. What if I were to skip the HBA/Raid Controller and attach these 5 drives to my open SATA 2 ports? My MB is a server board that supports RAID, so am I wasting money by using a drive controller anyways? Should be able to connect the drives straight to the board to achieve similar results yeah? https://www.asrockrack.com/general/productdetail.asp?Model=EP2C602-4L/D16#Specifications

yep on i7-930; it's old and draws a bit more power but should suffice. For HBA, ASR-7805 is $12 and doesn't need reflashing; there's a config option to flip it to HBA mode. Just need cables from SFF-8643 to whatever your backplane needs (or 8482 if no backplane).

ADAPTEC ASR-7805 8INT PORT 6GBPS SATA CARD for sale online | eBay

Find many great new & used options and get the best deals for ADAPTEC ASR-7805 8INT PORT 6GBPS SATA CARD at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Wow, 12 bucks! How are they making any money with free shipping too? Thanks for the link

 

[Sean Ho]

The primary value of a SAS HBA is to provide the option of buying cheaper (in $/TB) used SAS HDDs; a close second is the significantly higher number of drives you can add with an expander (or expander backplane).

If all your drives are SATA2, yes just use the onboard ports on your motherboard. Use software drive pooling like WSS, zfs, mdadm, etc. Hardware RAID (e.g., LSI card in IR mode) is of decreasing utility given how easily modern CPUs can do the parity calculations. PCH fake raid (as with your C602 board) is worthless.

Is the sequential r/w throughput of WSS presenting a bottleneck for your current workload?

 

[bigfellasdad]

1.) How are these specs for a NAS build? Intended for speed, redundancy, editing, rendering - is it overkill and better suited to keep as a workstation?

Overkill if you have any concerns of power usage. Any e3 xeon (v6 or better really for better C states), or indeed i3/5/7 that has an integrated graphics card, has 5/6 sata connections (or the onboard/pcie expansion card and alos allows ECC will suffice. Install linux/proxmox what ever and it will run silently and probably sub 35/40w so nice and cool, find a case that fits your 5 drives and you are golden. Shove the server out of the way in a dust free environment. The 10g network is going to pump the heat/power up though. *if* you need it then go for it, else, just get a dual 2.5gb nic and a 2.5gb managed switch. It will cut down on recabling everything and give plent of performance, i doubt the spinners would flood your network.

2.) Am I limited to windows server if I want the system to double as a workstation? Obviously a ZFS system is preferred to protect my data

Not if you do above, keep you windows PC on only when you need it, and care less about what GPU power usage etc as its not on all of the time

3.) Can I achieve similar NAS goals using an older pc I have laying around that is not built with server components?

Yep! read 1.

HTH

 

[afcurry]

yep on i7-930; it's old and draws a bit more power but should suffice. For HBA, ASR-7805 is $12 and doesn't need reflashing; there's a config option to flip it to HBA mode. Just need cables from SFF-8643 to whatever your backplane needs (or 8482 if no backplane).

ADAPTEC ASR-7805 8INT PORT 6GBPS SATA CARD for sale online | eBay

Find many great new & used options and get the best deals for ADAPTEC ASR-7805 8INT PORT 6GBPS SATA CARD at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Got this controller but it is only giving me 260mb/s read speeds with 5 drives in a RAID 10, vs 400 mb/s via an lsi 9211-8i. It is also screaming at me with a very loud alarm upon boot that will not stop unless I disconnect the controller. Have you used this controller before? Am i missing something in the settings? Got two of them and the issue is replicated between both.

 

[afcurry]

1.) How are these specs for a NAS build? Intended for speed, redundancy, editing, rendering - is it overkill and better suited to keep as a workstation?

Overkill if you have any concerns of power usage. Any e3 xeon (v6 or better really for better C states), or indeed i3/5/7 that has an integrated graphics card, has 5/6 sata connections (or the onboard/pcie expansion card and alos allows ECC will suffice. Install linux/proxmox what ever and it will run silently and probably sub 35/40w so nice and cool, find a case that fits your 5 drives and you are golden. Shove the server out of the way in a dust free environment. The 10g network is going to pump the heat/power up though. *if* you need it then go for it, else, just get a dual 2.5gb nic and a 2.5gb managed switch. It will cut down on recabling everything and give plent of performance, i doubt the spinners would flood your network.

2.) Am I limited to windows server if I want the system to double as a workstation? Obviously a ZFS system is preferred to protect my data

Not if you do above, keep you windows PC on only when you need it, and care less about what GPU power usage etc as its not on all of the time

3.) Can I achieve similar NAS goals using an older pc I have laying around that is not built with server components?

Yep! read 1.

HTH

Thank you for your response. I know this thing draws a lot of power, just going to deal with that for now. Decided against the NAS route for now and simply building a RAID array as i'm done fighting this and want to move forward with business.

 

[Sean Ho]

All PCIe cards designed for servers, but especially ASRs, need a good bit of airflow over the heatsink. I tried an ASR-7805 once with the chassis lid off (hence no airflow), idle with no drive activity, and it overheated (beeping alarm) within a couple minutes.

If increasing airflow through the PCIe area isn't possible, consider ziptying an 80mm fan to the heatsink.