New Homeserver built: TrueNAS Scale with PLEX, HA, NC: Hardware recommendations for SFF?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AlexGee

Member
Aug 3, 2022
49
18
8
Dear all,

I'm currently running Proxmox with Plex and HA (home Assistant) - never got NC (Nextcloud) running properly - on some old hardware (Z77, 2600K). It's time for a new built from scratch :)
Low Power consumption and Virtualization capability of CPU and more Cores for VMs are the drivers here. Therefore no older server platforms.

I'm planning to switch to TrueNAS Scale OS and run NC, HA, PLEX from there! Objective is to build a robust home server for the next 5 years.

First: easy topics
  • Case
    • I want to build a SFF Server and I think the Intertech IM-1 Pocket with 20L would do the job
    • minimum mATX mainboard
    • ATX power supply fits
  • Hard drives, quantity, model numbers, and RAID configuration, including boot drives:
    • OS (TrueNAS): SATA SSD - single or mirror - is 128GB sufficient for TrueNAS?
    • VMs: NVME boot drive (2x 1TB Samsung 970 EVO - PCIe x3.0) - mirrored
    • Storage Pool: 4x 16-18TB in ZFS pool - striped mirrored vDev
      • CMR, Helium, 512e, e.g. Toshiba Enterprise
  • Hard disk controllers:
    • Mainboard SATA ports should be sufficient in terms of number
    • Any benefit of HBA Controller card in terms of speed vs. Mainboard SATA?
  • Network cards:
    • PCIe 10GbE - probably X520-DA2
  • RAM quantity:
    • 2x16GB, or 4x 16GB
    • DDR4 ECC, or DDR5 ECC (on-die ECC is not meeting my requirements for data integrity)
  • Motherboard make and model
    • mATX motherboard
      • AM4, e.g. Asrock Rack X470D4U
      • wait for AM5 if DDR5 ECC is available soon
      • Intel socket 1700 (W680 chipset to gain ECC support)
    • 4x RAM would be nice
    • 2x NVME
    • 6-8x SATA
  • CPU make and model
    • 6c/12T with iGPU (if no Mainboard with GPU) and ECC support
      • AMD Pro 5650G
      • Intel 12500/12600 (with W680 mainboard) - Quicksync might be useful if I would like to use Plex Hardware transcoding
      • Intel Xeon E-2336 with C252/256 Mainboard - no quicksync

Questions from my end:
  • Which combination of Mainboard, RAM and CPU would you choose?
  • Any information about broader availability of DDR5 ECC?
  • Are there any hardware restrictions regarding TrueNAS scale OS (e.g. PCIe NIC chip)?

Looking forward to hear your opinions and thank you
Alexander
 

AlexGee

Member
Aug 3, 2022
49
18
8
Some questions have answered themself, as I won an online auction today for a server rack:

Platform 1
Asrock Rack X470D4U, 2700X, Noctua Cooler, 2x16GB ECC DDR4

Platform 2
2011-3 Mainboard, E5-2630L v3, 4x 8GB DDR4

2x HBA Controller
3x SSD 4TB Samsung 860
5x SSD 500GB Samsung 860
2x SSD 500GB Samsung 850
2x HDD 4TB WD Red Pro
2x SSD 120GB SSD
1x NVME 250GB Crucial
1x NVME 500GB Samsung 970 EVO Plus

2 power Supplies (one 80+ Titanium)

therefore I will shift my attention to the vDev Design with all these drives:

1 have additional storage which I could utilize:
1x SSD 128GB Crucial
1x SSD 512GB Crucial
2x NVME 1TB Samsung 970 Evo
1x NVME 2TB Intel 660p (QLC)
2x HDD 4TB WD Red
2x HDD 2TB Samsung (very old)

Plus new large HDDs if needed - to buy

How could a good vDev Design look like in terms of storage space, power consumption, data integrity and speed?
 

nabsltd

Active Member
Jan 26, 2022
340
208
43
  • CMR, Helium, 512e, e.g. Toshiba Enterprise
Don't restrict yourself unnecessarily...ZFS has no worse performance on 4Kn drives as on 512e or 512n, and might even be better. Unless you really tinker with your vDevs and pools, ZFS always writes far more than 4K bytes at a time.

You could also grab a pair of 2-4TB NVMe drives and partition off 512GB-1TB on each to use as a small block special vDev tied to your spinners. Then, use the rest of the drive for VMs.
 
  • Like
Reactions: Markess and T_Minus

AlexGee

Member
Aug 3, 2022
49
18
8
Don't restrict yourself unnecessarily...ZFS has no worse performance on 4Kn drives as on 512e or 512n, and might even be better. Unless you really tinker with your vDevs and pools, ZFS always writes far more than 4K bytes at a time.
Price is the same for 4kn and 512e - I read that there is no performance difference anyway, but 512e might be a littler bit more compatible
I will do some more research, even though it probably doesn't matter

You could also grab a pair of 2-4TB NVMe drives and partition off 512GB-1TB on each to use as a small block special vDev tied to your spinners. Then, use the rest of the drive for VMs.
Can you explain more? Why should I partition the NVME-Mirror? As cache (SLOG, L2ARC,...)?
 

nabsltd

Active Member
Jan 26, 2022
340
208
43
Why should I partition the NVME-Mirror? As cache (SLOG, L2ARC,...)?
The whole 512e thing got me thinking that you had workloads with small blocks, and that's why you wanted them.

For that, you can use a special vDev in your pool that only holds small blocks. You'd partition off the NVMe before mirroring so you can use it for two (or more) different things. So, one would be small block, and the other would be whatever else you wanted (a different storage pool, L2ARC, etc.). I wouldn't use something as high latency and low endurance as a Samsung 970 as SLOG...you want Optane or really low latency NVMe (like Intel P3700 series).
 

AlexGee

Member
Aug 3, 2022
49
18
8
The whole 512e thing got me thinking that you had workloads with small blocks, and that's why you wanted them.

For that, you can use a special vDev in your pool that only holds small blocks. You'd partition off the NVMe before mirroring so you can use it for two (or more) different things. So, one would be small block, and the other would be whatever else you wanted (a different storage pool, L2ARC, etc.). I wouldn't use something as high latency and low endurance as a Samsung 970 as SLOG...you want Optane or really low latency NVMe (like Intel P3700 series).
Makes sense...

I like it simple - therefore dedicated (low endurance) NVME for VMs - no partitioning

SLOG: Optane drives seem to be the most viable option BUT, the small/cheap ones with 16GB ( MEMPEK1W016GAXT ) seem to be really slow and the proper one like P1600X 118GB are around 250$ new

SLOG, yes, or no and any "cheap option" which has some durability?
 

nabsltd

Active Member
Jan 26, 2022
340
208
43
SLOG, yes, or no and any "cheap option" which has some durability?
The P3700 used are your best bet for cheap, as they have very low latency, 10 DWPD endurance, and more than fast enough speed to handle a 10Gbps network link. The used ones I bought had less than 100TB written, out of a endurance spec of over 28000 TB.

You don't need SLOG unless you have a lot of sync writes, which you likely would not if VMs local to the TrueNAS machine are your primary source of writes.