New storage box for mixed workload

[vrod]

Hi all,

I was lucky and got my hands on a 4U supermicro case with 36 bays. Gonna pop a s2600cp in that case with a couple of e5 v2 cpu’s to make a new storage box.

I have been wondering how to setup it up, been using both freenas and ubuntu to serve nfs and iscsi for some time to my current vsphere env.

I plan to use 36x3TB HGST Ultrastars and a P3700 ssd (cache) but I’m not sure what would be best. A lot of people recommend going for mirroring, so I think I would do that too.

Workload will be UHD movies and VM’s (vSphere).

What would be the best config for the hdds? 3x12,1x24 + 1x12 or 1x36 pool design?

What OS is the better one to serve either iscsi or nfs? I’m a little tipped on ubuntu but also had some good experiences with freenas.

In that case, iSCSI or NFS? What do people recommend?

If I had several pools, it probably wouldn’t be a problem to split the P3700 into several partitions and use it for slog for all the pools, or?

 

[MiniKnight]

Raid 10 and p3700 for VMs.

He's for movies in z2

If you do use the p3700 for l2arc on movie storage you'll need to set fill rate so you can have a chance to cache movies.

 

[vrod]

Thanks for your input! How many vdevs would I make in the raidz2? just 1 or several? The box will have 256gb of memory so hopefully I won't need l2arc. I stream 4K UHD movies which can be up to about 70Gb, so I guess that would also fit entirely in memory. I never heard about "fill rate" for l2arc, would you be able to elaborate?

 

[Connorise]

Hey, there I agree with this recommendation fully.

Raid 10 and p3700 for VMs.

As the option, I see the configuration with ZFS on FreeBSD quite good for this case.

This is definitely the right way to go here.

 

[gea]

For ESXi and a mixed workload I would use multi-mirrors. For pure video, a raid-z2 would be ok unless you have many users.
I would use an Intel Optane 900P with a 20GB vdisk for Slog (for VMs) and the rest for the storage VM and L2Arc with read ahead enabled.

The fastest ZFS is a genuine ZFS and Oracle Solaris. For Open-ZFS FreeBSD and Illumos (free Solaris fork with distributions like OmniOS or OpenIndiana) are the best options. I prefer Solarish based systems with best ZFS integration and NFS and SMB as kernelbased ZFS properties and Comstar, an enterprise FC/iSCSI stack

more
Configuring Storage Devices With COMSTAR - Oracle Solaris Administration: Devices and File Systems
http://napp-it.org/doc/downloads/optane_slog_pool_performane.pdf

 

[vrod]

I bought a 280gb optane 900p after hearing how well it performs as a slog and because it also has the 10DWPD.

That means I will probably be getting rid of my P3700, in case the HDD array performs to my liking. thanks for the suggestion!

I have about 15TB of media content so I gotta figure out what to do. I’d honestly really like to have everything in one raid10 pool for the spindle performance. With 18x3 tb I would still get about 50TB of usable space or so.

 

[gea]

I would probably prefer a large pool from disks and a second fast ssd only pool for performance sensitive data. But with an Slog like the Optane 900p and a multiple raid-10 setup performance should not be a concern even when secure sync write is enabled (see my Optane benchmarks with a 4 x HGST HE 8 Pool and the Optane 900P as slog)