New video storage build

Patrick

Administrator
Staff member
Dec 21, 2010
11,891
4,849
113
I would also love to hear about networking and offsite backup.
 

Patrick

Administrator
Staff member
Dec 21, 2010
11,891
4,849
113
What specifics would you like to know?
Did you build a box and colo it? Using B2/ S3 or similar?

On the networking side, what NICs/ switch are you using. I know many of the video editing guys opt for huge bandwidth.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
6,974
1,561
113
CA
Wow, awesome :D
More pics, and build come on it's been a couple hours :p :p
 

GCM

Active Member
Aug 24, 2015
137
43
28
Did you build a box and colo it? Using B2/ S3 or similar?

On the networking side, what NICs/ switch are you using. I know many of the video editing guys opt for huge bandwidth.
"All of the above"

ZFS replication offsite.

Also S3 with versioning turned on.

I'll post more pictures / updates soon. I'm trying to play around with napp-it, but OmniOS just reboots after getting to the install menu.
 

wiretap

Active Member
Jul 14, 2015
128
85
28
Michigan
Maybe he's actually using it for something useful like 4K R3D capture, since it would be quite a waste of money just to store stationary files on it as a file server. OP -- elaborate?
 

DavidRa

Infrastructure Architect
Aug 3, 2015
268
118
43
Central Coast of NSW
www.pdconsec.net
If he's storing and editing video, he may well be using RAID 10 (or ZFS mirrors) or similar - great random IO, but you lose 50% of the space.

Or, 42 x 8TB drives could, for example, be:
  • 150TiB of RAID 10 / RAIDz1
  • 200TiB of RAID 6 [7 x (4D+2P)] or RAIDz2 [7 x (4D+2P)]
  • 232TiB of RAID 6 [4 x (8D+2P) or RAIDz2 [4 x (4D+2P)] with 2 spares
  • 174TiB of RAID z3 [6 x (4D+3P)]
  • 174TiB of RAID z3 [3 x (8D+3P)] with 9 spares (this seems very unlikely)
There are of course many other possible scenarios - temporary storage, separate editing and final storage, etc. I've only shown a couple of the many possible "OBFR"* configurations.

--
* OBFR: One Big ... er ... "Fast" ... RAID. Yeah, Fast, that's it.
 

maze

Active Member
Apr 27, 2013
556
84
28
What drive models and raid setup did you go with? 42x8tb SMR drives could be an interesting (yet a bit dangerous) experiment :)
 

wiretap

Active Member
Jul 14, 2015
128
85
28
Michigan
With 256GB of RAM, it would be pretty awesome to setup a good chunk of it to RAM cache using PrimoCache. o_O When I had my dual E5-2670 w/ 128GB RAM setup with Windows Server, I dedicated 32GB of the RAM to cache for testing purposes, and it worked extremely well.
 
  • Like
Reactions: Chuntzu

Chuntzu

Active Member
Jun 30, 2013
383
98
28
I did a similar experiment with 32 x 6tb drives, 16 x 400gb s3700, and 2x 400gb nvme drives. in 4 x 2u supermicro 12 x 3.5" servers using storage spaces direct using mirroring and parity spaces. It was very performant but would have just built with all nvme drives and no ssds if I would do it over again. It turned out it was alot faster than my older storage spaces direct platform using a few dual sas jbods and 32 x 3tb sas drives and similar amounts of ssds. Definatly not an apples to apples comparison but per u of space the storage spaces direct platform performed better and with much cheaper hardware. But can be a pain in the butt to set up, though it has gotten alot easier since tp5 and forward.

One thing that I would keep in mind is depending on what os your workstations are using, a windows based storage platform can benefit you from a networking stand point. IE Windows Server networking to windows workstations is crazy fast (make note there has been a post on the forum somewhere explaining ins and outs of how to get smb multichannel working with non domain joined computers. I think it has to show network as private. long and short is just setup a domain and its all good. Bit of a tangent but may save you a ton of time in trouble shooting). I have had remote access speeds in the millions of iops and 20+ gigabytes per second from workstation computers using server 2012 r2 over smb 3.02 literally limited by the cpus ability to read and write. It is amazing and was very easy to set up.

I look forward to seeing your progress and can't wait to read more. The bigger the better!
 

wildpig1234

Well-Known Member
Aug 22, 2016
1,775
264
83
45
comparatively I only got a 32gb msata module on the d30 2x e5-2670 64gb ram WS I am typing out this email right now .. :)

I cant imagine how long it would take to do a back of data on this.... if you got raid 6, rebuilding would take days too? what kind of network are you running to it?
 

ecosse

Active Member
Jul 2, 2013
373
64
28
Jealous of the disk - I wish I could afford to spend that amount on disks in one massive lump!!