FreeNAS Pool & vdev config

[tbatliner]

Hi,

hopefully next week my new Server will arrive. It's a (used) supermicro Server with 36bays, 4x1GB LAN, dual Xeons and 96GB ECC ram.
Currently I'm using 7 4TB HDD's in a raid5 configuration, which is crammed full. So my idea was to buy 13 more 4TB disks, and start with a 10 disk raidz2 vdev in the new server, copy everything over and finally adding the existing 7 drives plus the 3 spare drives to a second 10 disk raidz2 vdev in the same pool.

This is for home use (1GB clients) , mostly media and backups. Am I assuming correctly that that will give me more than enough sequential io (media streaming) and also more than enough performance for random io because of the ram? And 10 disk raidz2 should be reasonable as well?

Any input is highly appreciated, tia!

 

[Patrick]

I am guessing you will be OK. Maybe worth testing before you are finished moving everything off.

 

[tbatliner]

Thanks! What exactly should I test?

 

[gea]

For a home media and backup server, you are far above any performance questions.Your RAM is overkill. There are not that many commercial setups with such an amount of RAM.

I know that FreeNas suggests 1 GB RAM per TB data, what would mean 64 GB with 64 TB max data. Oracle on the other side claims a minimum of 2 GB RAM for Solaris, the origin of ZFS regardless the poolsize (unless you do do not use dedup). 96 GB RAM gives you a huge readcache that you will not need or use especially as the cache ist intended for small random reads not large sequential data. The really needed RAM size can be between. I would say 8-32 GB is enough.

Beside that, sequential performance scale with number of datadisks (16 x a single disk, count 80-200 MB/s per disk) while iops scale with number of vdevs (2 x a single disk, count about 100 iops per disk)

 

[tbatliner]

Thank you!
One use case I forgot to mention will be jails/vm... I'm still undecided if I dare to virtualize FreeNAS, though. Safest bet for me would probably be to install FreeNAS as host /os and then use "what's possible" with virtualbox...

 

[gea]

Virtualisation is a key question where you have the options
- use a full size OS and virtualize based on a virtualisation layer in the OS (jails,zones,..)
- use a layer above the OS (Virtualbox etc)

In both cases, you must think about backup/ restore procedures.
As you use a full size/ complex installation, this can be quite time consuming..

This is why I prefer a type -1 vm solution with ESXi as base. This is installed
and reinstalled after a crash within minutes as you always only need a base setup
where you simply can create/import VMs.

The same with storage. I prefer a minimalistic storage VM without any services
beside storage. This is why I prefer Solaris based solutions as Sun included everything
in a base setup with FC/iSCSI, NFS, SMB and network virtualisation. In case of
problems, such a server is reinstalled in 5 minutes ready to use. A BSD solution
is possible as well but I would avoid services beside storage.

Services that require configuration (Windows AD, Webserver, Mediaserver, Databases)
are virtualized on ZFS storage with security, performance, versioning, replication
with the best suited OS - from BSD over Linux, OSX, Solaris to Windows.
On any crash, you only need to reimport - either last state or a previous version
(ex using Windows "previous version" over SMB)

 

[tbatliner]

Wow thank you very much for your detailed response. Now I'm even more undecided what I should go for... Really like the ESXi way but a little bit afraid that I could loose my data...

 

[gea]

Which data?
ESXi and the storage VM is uncritical. If you loose them, this is only a matter of half an hour
to reinstall them and to import pools and VMs. All important data and VMs are and should be on secure ZFS storage

 

[tbatliner]

Which data?
ESXi and the storage VM is uncritical. If you loose them, this is only a matter of half an hour
to reinstall them and to import pools and VMs. All important data and VMs are and should be on secure ZFS storage

Yes I referred to the data on the zfs storage, which would be on the disks of a freenas vm.
I just keep 'hearing' cyberjock's "all will work until someday something bad will happen"

 

[gea]

Its a bad idea to have any important data on your system disk / your storage VM on unsecure vmfs storage.
All important data is on your ZFS datapool where it is safe unless no failure happens that exceeds the pool redundancy -

beside that, you need an external disaster backup with ZFS as well (fire, flash, theft)

 

[tbatliner]

Yes probably I wasn't clear how the setup should look like...
I'd like to have esxi and "storage vm" on a separate disk, then set up a freenas-vm with pci passthrough of the disks for the zfs pool for the "real" data.