New video storage build

Discussion in 'Chassis and Enclosures' started by GCM, Sep 23, 2016.

  1. GCM

    GCM Active Member

    Joined:
    Aug 24, 2015
    Messages:
    137
    Likes Received:
    43
    [​IMG]

    It's finally here!

    Controller + JBOD (no onsite HA, replicating offsite)

    Some random stats: 256GB RAM, Dual E5, 42 * 8TB drives.
     
    #1
  2. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,775
    Likes Received:
    1,116
    Might be overkill...but WTH, overkill is fun!
     
    #2
  3. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,559
    Likes Received:
    4,489
    I would also love to hear about networking and offsite backup.
     
    #3
  4. GCM

    GCM Active Member

    Joined:
    Aug 24, 2015
    Messages:
    137
    Likes Received:
    43
    What specifics would you like to know?
     
    #4
  5. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,559
    Likes Received:
    4,489
    Did you build a box and colo it? Using B2/ S3 or similar?

    On the networking side, what NICs/ switch are you using. I know many of the video editing guys opt for huge bandwidth.
     
    #5
  6. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,831
    Likes Received:
    1,484
    Wow, awesome :D
    More pics, and build come on it's been a couple hours :p :p
     
    #6
  7. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,559
    Likes Received:
    4,489
    Or video :)
     
    #7
  8. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,775
    Likes Received:
    1,116
    Really, Patrick, what makes you think he has enough storage to host video? :)
     
    #8
    CreoleLakerFan likes this.
  9. GCM

    GCM Active Member

    Joined:
    Aug 24, 2015
    Messages:
    137
    Likes Received:
    43
    "All of the above"

    ZFS replication offsite.

    Also S3 with versioning turned on.

    I'll post more pictures / updates soon. I'm trying to play around with napp-it, but OmniOS just reboots after getting to the install menu.
     
    #9
    Chuntzu, Patrick and T_Minus like this.
  10. wildpig1234

    wildpig1234 Well-Known Member

    Joined:
    Aug 22, 2016
    Messages:
    1,620
    Likes Received:
    191
    whats the build for? why so much HD space?
     
    #10
  11. Patrick

    Patrick Administrator
    Staff Member

    Joined:
    Dec 21, 2010
    Messages:
    11,559
    Likes Received:
    4,489
    Thread title says "video storage" so I am guessing movies.
     
    #11
  12. wildpig1234

    wildpig1234 Well-Known Member

    Joined:
    Aug 22, 2016
    Messages:
    1,620
    Likes Received:
    191
    so thats over 13000 BD movies at 25GB each!? that's about $10K worth of hdd? wow....
     
    #12
  13. wiretap

    wiretap Active Member

    Joined:
    Jul 14, 2015
    Messages:
    128
    Likes Received:
    82
    Maybe he's actually using it for something useful like 4K R3D capture, since it would be quite a waste of money just to store stationary files on it as a file server. OP -- elaborate?
     
    #13
  14. DavidRa

    DavidRa Infrastructure Architect

    Joined:
    Aug 3, 2015
    Messages:
    253
    Likes Received:
    111
    If he's storing and editing video, he may well be using RAID 10 (or ZFS mirrors) or similar - great random IO, but you lose 50% of the space.

    Or, 42 x 8TB drives could, for example, be:
    • 150TiB of RAID 10 / RAIDz1
    • 200TiB of RAID 6 [7 x (4D+2P)] or RAIDz2 [7 x (4D+2P)]
    • 232TiB of RAID 6 [4 x (8D+2P) or RAIDz2 [4 x (4D+2P)] with 2 spares
    • 174TiB of RAID z3 [6 x (4D+3P)]
    • 174TiB of RAID z3 [3 x (8D+3P)] with 9 spares (this seems very unlikely)
    There are of course many other possible scenarios - temporary storage, separate editing and final storage, etc. I've only shown a couple of the many possible "OBFR"* configurations.

    --
    * OBFR: One Big ... er ... "Fast" ... RAID. Yeah, Fast, that's it.
     
    #14
  15. maze

    maze Active Member

    Joined:
    Apr 27, 2013
    Messages:
    540
    Likes Received:
    77
    What drive models and raid setup did you go with? 42x8tb SMR drives could be an interesting (yet a bit dangerous) experiment :)
     
    #15
  16. wiretap

    wiretap Active Member

    Joined:
    Jul 14, 2015
    Messages:
    128
    Likes Received:
    82
    With 256GB of RAM, it would be pretty awesome to setup a good chunk of it to RAM cache using PrimoCache. o_O When I had my dual E5-2670 w/ 128GB RAM setup with Windows Server, I dedicated 32GB of the RAM to cache for testing purposes, and it worked extremely well.
     
    #16
    Chuntzu likes this.
  17. Chuntzu

    Chuntzu Active Member

    Joined:
    Jun 30, 2013
    Messages:
    383
    Likes Received:
    98
    I did a similar experiment with 32 x 6tb drives, 16 x 400gb s3700, and 2x 400gb nvme drives. in 4 x 2u supermicro 12 x 3.5" servers using storage spaces direct using mirroring and parity spaces. It was very performant but would have just built with all nvme drives and no ssds if I would do it over again. It turned out it was alot faster than my older storage spaces direct platform using a few dual sas jbods and 32 x 3tb sas drives and similar amounts of ssds. Definatly not an apples to apples comparison but per u of space the storage spaces direct platform performed better and with much cheaper hardware. But can be a pain in the butt to set up, though it has gotten alot easier since tp5 and forward.

    One thing that I would keep in mind is depending on what os your workstations are using, a windows based storage platform can benefit you from a networking stand point. IE Windows Server networking to windows workstations is crazy fast (make note there has been a post on the forum somewhere explaining ins and outs of how to get smb multichannel working with non domain joined computers. I think it has to show network as private. long and short is just setup a domain and its all good. Bit of a tangent but may save you a ton of time in trouble shooting). I have had remote access speeds in the millions of iops and 20+ gigabytes per second from workstation computers using server 2012 r2 over smb 3.02 literally limited by the cpus ability to read and write. It is amazing and was very easy to set up.

    I look forward to seeing your progress and can't wait to read more. The bigger the better!
     
    #17
    aydinpr, Cipher, lmk and 1 other person like this.
  18. wildpig1234

    wildpig1234 Well-Known Member

    Joined:
    Aug 22, 2016
    Messages:
    1,620
    Likes Received:
    191
    comparatively I only got a 32gb msata module on the d30 2x e5-2670 64gb ram WS I am typing out this email right now .. :)

    I cant imagine how long it would take to do a back of data on this.... if you got raid 6, rebuilding would take days too? what kind of network are you running to it?
     
    #18
  19. ecosse

    ecosse Active Member

    Joined:
    Jul 2, 2013
    Messages:
    354
    Likes Received:
    59
    Jealous of the disk - I wish I could afford to spend that amount on disks in one massive lump!!
     
    #19
  20. wildpig1234

    wildpig1234 Well-Known Member

    Joined:
    Aug 22, 2016
    Messages:
    1,620
    Likes Received:
    191
    haha, its probably all company's money anyway....
     
    #20
Similar Threads: video storage
Forum Title Date
Chassis and Enclosures VGA and HDMI video Oct 27, 2016
Chassis and Enclosures EPYC2 Software Defined Storage: Gigabyte R272-Z32 vs. Lenovo SR655 Nov 3, 2019
Chassis and Enclosures McGuyver a CSE-216 from a Quanta 2U server or Netapp 2246 storage? Jul 23, 2019
Chassis and Enclosures Dell PS6100 Storage Array hookup/testing Help! Apr 8, 2019
Chassis and Enclosures How do you use 1U rackmount case CSE-502L-200B to create Unraid storage array? Mar 12, 2019

Share This Page