second what
@zack$ says about virtualiz-ing. (for now) I have 1 bare-metal freenas and 2 others: virtual primary & backup. I feel I get a lot of bang for my buck on the virtualized servers and am not really giving anything up. Same primary server holds my emby instance, freenas (media), 2 DNS servers, a windows 10 VM and a handful of other items. FreeNAS has an HBA (spinner pool) and an optane 900 AIC (slog) passed through. Emby and Windows 10 each have an nvidia GPU passed through. this server is *always* on as are its VM's.
If you are doing 12x 12TB and hoping for close to 10Gbe with spinners as large an ARC as you can afford will likely be critical to success. For writes definitely need a SLOG. Remember SLOG is not cache but it helps getting those writes on media that is not your main pool.
Probably 2x6Rz2. maximizing performance will be more like 6x2 mirror. Just remember on a mirror if both drives in a mirror fail then the pool is tot.
IMO 1x12Rz2 is about as big as I would go but not sure you'll get the performance you are looking for.
per drive performance (on board cache) etc. will probably be a little bit better with SAS vs. SATA - but that's all about your comfort level. If it were me I'd be looking for
a good SAS3 controller ala 9400-16i to pass through to a virtualized freenas. (just an example no relationship with seller).
I like planning out a virtual server by drawing out what will run on it, creating budgets and allocating which resources are going where . That helps me steer to the correct hardware choice though it is a circular process of refinements and ultimately trade-offs.
budgety-things:
CPU cores/threads, memory, onboard storage ports, pcie lanes/slots with lanes, hot swap bays, not hot swap or "velcro bays"...
Now, what VMs: FreeNAS, linux1, linux2, Windows 10 etc. etc. and line up your resources.
look at hardware options then do it again and refine it until happiness is reached.
If you use VMware and intend this to be an always on server then I heartily recommend a hardware raid controller and R1 SSD boot array for ESXI, maybe a pair of 1.6TB intel DC S3500 series. If you go with proxmox - different story (and I'm in the early stages of learning about that). Size the boot array to hold everything on day 1 and if you can afford the cost double that that storage size.
You could also burn two SATA ports and install a pair of SATADOM's to use for your FreeNAS boot pool - gives you some flexibility to boot FreeNAS bare metal if VMware took a dump.
did I mention memory? I think 128-256GB is probably your overall system goal with 64-128 given over to the FreeNAS VM. FreeNAS likes its memory.
I'm not a super huge fan of all integrated on motherboard components. For a single CPU board I really am fond of the X9/X10SRL-F but as
@zack$ says ESXI is likely to throw out support for those older boards... The
Fuji board in the deals thread might be of interest to you too.
72-120TB is a lot of storage even 20% utilized in the beginning is a big chunk, potentially larger than most USB backup drives. What's your backup plan?
If you go with ESXI and it will be your only server I'd research the limitations of the free VMware and make sure its a fit. If you think you'll add another vmware server then look at
VMUG advantage.
My requirements are not yours neither are my needs YMMV - just some ideas for you to kick around.
So here's an example build with resource budget from my rack, everything was purchased used (exceptions noted) via the bay or private sellers.
Compellent SC030 chassis upgraded to 920SQ power supplies (CSE-836)
X10SRL-F
256GB - 8x32 GB PC4-2400T memory
E5-2680v4 (14c/28T)
SM 3U cooler
Mellanox ConnectX3 dual 10Gbe (x8) (ESXI)
Nvidia P2000 (x8 in x16 phys) (CentOS Emby)
Nvidia P620 (x8 in x16 phys) (Windows 10)
LSI 9400-16i (x8) IT mode (NIB from above linky) (FreeNAS)
LSI 3008-8i (x4 in x8) (R mode) (ESXI)
intel Optane 900P AIC (x4 in x8) (FreeNAS)
2 x intel DC3520 1.6TB 2.5" SSD's (connected to LSI 3008 in R1 array)
16 x HGST 8TB SAS3 HDD (mix of 4201, 4200, 5200), configured as 2x8RZ2
ESXI 6.7
FreeNAS 11.3U5 6 vcpu, 128GB, 940016-i, Optane 900P AIC
CentOS Emby 8 vcpu, 8GB
CentOS 4 v cpu, 4GB
CentOS 4 vcpu, 4GB
CentOS 4 vcpu, 4GB
Centos 8 vcpu, 8GB
Debian 4 vcpu, 4GB
Windows 10 8 vcpu, 16GB
Vcenter and a few others...