Performance / optimal setup for TrueNAS - long

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Networks101

New Member
Jul 10, 2020
7
2
3
Australia NSW
Hi All,
I’ll preface this with I’ve worked with large scale isilon, netapp, VMAX, etc solutions, mostly 5years + ago, and mostly focused about low latency high speed access + reliable access (supercompute storage + processing + 300
X 10G desktops accessing 40-80GB datasets + long term slower storage for archived projects)

But I’m a complete noob when it comes to freenas/truenas, likewise some of the recent tech I.e. nvme, pci based flash drives

I’m testing different configs at the moment, and wouldn’t mind some critiquing / suggestions as to what I can do better, and to assist with speed and expansion

That being said
* Dell R720 dual E5-2620 v0
* 128GB DDR3, ECC
* Onboard 2x10Gbps NDC
* Dell 12Gbps SAS HBA adapter - external, PCI3.0, installed in one of the x16 slots.
* Integrated H310 flashed to IT mode
* no name brand NVME to PCI convertor in x4 slot
* 500 GB Crucial P2 3D NAND NVME
* Dell MD1220 with 4 x Kingston 1TB SSD’s, 16 x 1.2TB 10k SAS former netapp drives (reformatted to 512 bytes)
4 x 3.5” 8TB WD 7200rpm sata drives - unconfigured at the moment
* TrueNas 12.x beta installed as a VM under ESXI 7.0
* both the sas HBA and the h310 are passed through to the VM (32GB, 12 vCPU), likewise the NVME card, however the system detects it just as a drive rather than the card
* have 1 x 10Gb port directly passed through to the VM, and 1 x 10G vNIC


IPerf shows 7.9Gbps with MTU of 1500, low 9’s for 9000.

Test PC is using a RAM drive rather than disk for the copying

TrueNAS is setup as follows

* 2 x vdev pools of 8 drives with z1
* L2ARC is using the Crucial NVME
* 2 x 1TB SSD in raid 0 for log drive
* 1 x 1TB SSD as the dedup drive (not sure this actually does anything )

My max read / write tends to top out at 450MB/s from a client on a diff VM, and from a VM on the same box.

I can’t help but think the performance should be a lot better? Especially for writes. It’s my understanding that it should be writing to the ARC, then L2ARC, and then finally to the spinning rust? The files in question are not exceeding the size of either cache.

The 8TB SATA drives haven’t been added yet, as I’m not sure how to integrate them?

* do I add just another vdev, and assign to the pool, or create a new pool for “slow storage”
* the 8TB drives are still somewhat in use in my current NAS, and not ready to move them over until I’m happy with the TrueNAS setup
* or can I add the SATA vdev to the main pool?

I suspect it will be the former


I realise that 400-450MB/s via CIFS on a single transfer isn’t bad, but I was expecting 600-700

The 400/450, would seem to match the 6Gbps max speed of the MD1220 enclosure

Specs for the Crucial P2 aren’t fantastic, but should be more than enough ?

Crucial P2 500GB 3D NAND NVMe PCIe M.2 SSD,CT500P2SSD8 Crucial P2 500GB 3D NAND NVMe PCIe M.2 SSD,CT500P2SSD8: Amazon.com.au: Computers & Accessories




PCI NVME card (I’m using the sata port as well for vmlogs)


In summary, what can I do to make this better ?

* should I ditch running it under VM? And go bare metal, hard choice, due to the system resources that would goto waste.

* should i replace the l2arc with something a bit more enterprise grade - I.e intel P/D series

* how can I make use of the slower, but larger capacity spinning disks

* what tweaks can I look at to get the transfer speeds up? Either TrueNAS or client OS

* or does the numbers my setup return seem expected?

* would a 12Gbps enclose make any difference with the cache already not being utilised? (Drives should be fine due to spindle count)

* am I missing something really basic? I.e CPU too underpowered to handle the parity calcs while handling the IO from .10Gbps NIC + HBA.

* does TrueNAS have the ability to report on IOPs? / what stats should I be looking at

* If I was to go the path of a Intel DC PCIE storage card, is it possible to partition it so it could be used for multiple pools?

I have tried Unraid, which was absolutely dismal - 120-150Mbps for read, 40-50Mbps for write.

Sorry I know this is a lot to read through on a Sunday, but any assistance from those who’ve more experience would be highly appreciated, and more than happy to buy a carton of beer for the help!

Btw this is not commercial, this is not for a company, this is purely a home lab setup.
 

svtkobra7

Active Member
Jan 2, 2017
362
87
28
In summary, what can I do to make this better ?
Exec sum:
Proper pool layout
Proper use of SLOG, Ditch the L2ARC
Reseach

I can’t help but think the performance should be a lot better? Especially for writes. It’s my understanding that it should be writing to the ARC, then L2ARC, and then finally to the spinning rust? The files in question are not exceeding the size of either cache.
  • Some research is in order ...

  • [*]Using a SLOG for asynchronous write scenarios
    The ZFS filesystem can tier cached data to help achieve sizable performance increases over spinning disks. Users can set up flash-based L2ARC read cache and SLOG (Separate ZFS Intent Log, sometimes called a ZIL) ‘write cache’ devices. While an L2ARC read cache will speed up reads in most use cases, the SLOG only speeds up synchronous writes.
    The ZIL caches writes to guarantee their completion in the case of a power failure or system crash. The ZIL normally exists as part of the ZFS pool, but with a SLOG, it resides on a separate, dedicated device. This speeds up performance by batching data together for synchronous writes for more efficiency. These performance gains help with database operations, NFS operations such as virtualization where the operating system explicitly requests synchronous writes. If you aren’t using something that is known to use synchronous writes like NFS or databases, chances are your SLOG will not help performance. A potential solution here is to set your pool to “sync=always”. This ensures that every write goes to the write cache, improving write performance.
    [*]
  • FreeNAS: A Worst Practices Guide - iXsystems, Inc. - Enterprise Storage & Servers


    Use case generally?
* 2 x vdev pools of 8 drives with z1
* L2ARC is using the Crucial NVME
* 2 x 1TB SSD in raid 0 for log drive
* 1 x 1TB SSD as the dedup drive (not sure this actually does anything )


Assumes 170 MB/s for your 16 x 1.2TB 10k SAS HDDs (whether accurate or not, the following points hold).

Mirrored Pairs = 8 vdevs x 2 drives per vdev = MORE PERFORMANT, LESS CAPACITY = LET 'ER RIP
  • 1360 MB/s writes
  • 2720 MB/s reads
  • 9.6 TB
RaidZ1 = 2 vdevs x 8 drives per vdev = LESS PERFORMANT, MORE CAPACITY = DISCARD
  • 680 MB/s writes
  • 2720 MB/s reads
  • 16.8 TB
Misc
* should I ditch running it under VM? And go bare metal, hard choice, due to the system resources that would goto waste.
  • No
* should i replace the l2arc with something a bit more enterprise grade - I.e intel P/D series
  • Max RAM before thinking of L2ARC.
  • With 128 GB RAM, your L2ARC shouldn't exceed 5 x 128GB.
  • Ditch the L2ARC altogether, and free up $$$ for a proper SLOG?
* how can I make use of the slower, but larger capacity spinning disks
  • Mirrored pairs for 16 TB?
* what tweaks can I look at to get the transfer speeds up? Either TrueNAS or client OS
  • See above.
* or does the numbers my setup return seem expected?
  • They aren't unexpected given suboptimal configuration.
* would a 12Gbps enclose make any difference with the cache already not being utilised? (Drives should be fine due to spindle count)
  • No.
* am I missing something really basic? I.e CPU too underpowered to handle the parity calcs while handling the IO from .10Gbps NIC + HBA.
  • At 2.00 GHz (2.5 GHz turbo) I think it is woefully under-powered.
  • If that NVMe drive was properly allocated as a SLOG ... can a E5-2620 run it at full throttle?
  • I'd have to see a block diagram, but C602 chipset = PCIe 2.0. PCIe off the E5 should be 3.0. Your slot is?
* does TrueNAS have the ability to report on IOPs? / what stats should I be looking at
  • Yes.
  • There are a number of ways to evaluate raw pool performance, but since you discuss VM performance, why not step closer to a "real world" result and measure in a VM?
* If I was to go the path of a Intel DC PCIE storage card, is it possible to partition it so it could be used for multiple pools?
  • Absolutely.
NB1: Capacities offered don't represent usable storage amount after considering parity, padding, slop, need to allow free space, etc.
NB2: I'm far from a pro, but saw your thread unanswered and figured I'd at least give you a shove in the right direction ... a true pro will come along and clean up any inaccuracies in my commentary.
 

jcl333

Active Member
May 28, 2011
253
74
28
Following along as I feel a kindred spirit in you, you remind me of myself earlier in my research.
I am not the best person to give FreeNAS advise, at least not yet.

Some things I do want to add that I have learned:
- I know this is lab, but I would stay away from consumer SSDs - you need endurance and power loss protection (PLP)
- Those 1.2TB 10K disks probably are not worth the heat and power, we have tons of those at work, and I won't take them even free

I read your post, what are a couple of use cases you want to test in your lab?
 

zack$

Well-Known Member
Aug 16, 2018
701
315
63
The ballgame with L2ARC is changing (changed):

The issue of indexing L2ARC consuming too much system RAM was largely mitigated several years ago, when the L2ARC header (the part for each cached record that must be stored in RAM) was reduced from 180 bytes to 70 bytes. For a 1TiB L2ARC, servicing only datasets with the default 128KiB recordsize, this works out to 640MiB of RAM consumed to index the L2ARC.
Also, with the implementation of fusion pools and persistent L2ARC, TrueNAS has turbocharged read intensive application: Persistent L2ARC might be coming to ZFS on Linux

Fusion pools are a new feature of TrueNAS. Haven't tried it but it does look like an alternative to the whole SLOG + L2ARC conversation in the consumer/homelab context where users are trying to implement some of the benefits of the former but in a wrong use case.

Maybe a fusion pool may be worth a try?
 

MrGuvernment

Member
Nov 16, 2020
39
7
8
@jcl333 I believe some consumer drives like the Samsung Pro's offer PLP now. I would need to confirm but i do recall last few years more of them are coming out with PLP.
 

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
I would need to confirm but i do recall last few years more of them are coming out with PLP.
A ssd with a controller which uses DRAM will require batteries/capacitors for powerloss protection. This adds up costs which consumers don't want to pay.
The newer DRAM less controllers use a smaller SRAM (it's a magnitude faster than DRAM, requires less power and costs a lot more) cache. The idea here is that the data from the SRAM is flushed fast enough to avoid data loss :D
 

tomspappola

New Member
Sep 15, 2020
2
0
1
I have appreciated really much this post.
I'm new to truenas.

I 'm trying to setup a server with 128gb of ram and

4x 960GB enterprise NVE ssd
and
10X 10TB 12Gb sas 4k enterprise HDD

What could be the best config for backups from veeam and s3 compatible application ?

I read that ZFS need drive configuration with this formula:

  • (N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 8

My idea was to run 1 pool with
2 raidz vdev of 5 HDD each
and use 2 or all 4 ssd as ZFS LOG device for write cache (in mirror configuration)

Or In term of performance it will be better to run

3 raidz vdev with 3hdd
and maybe in future buy 2 more HDD to order to add a new raidz to the pool ?

What happen if I use a non standard configuration like
3 raidz vdev of 4 disks ?

My server could receive up tp 30Gbps from the network during backups and has 4x25Gbps ethernet cards.

Thanks
Tom.
 

Stephan

Well-Known Member
Apr 21, 2017
920
697
93
Germany
@tomspappola Since this is for Veeam storage then ZFS-wise I would create a single RAIDZ2 with the 10 HDDs and add two mirrored Optane using Intel Optane SSD DC P4801X 100GB for SLOG. Optanes either as PCIe card or U2 2.5" drive, I somewhat mistrust m2 not throttling under load due to temperatures. You will not ever need more than 30 Gbps * 3 TXG in flight * 5 seconds = 3800 MByte * 15 ~= 56 GiB. In all likelyhood, SLOG will never go beyond 16 GiB usage in your case. So choosing smallest 100 GB Optane with power loss protection is plenty.

Reasons for Optane: All you care here is write speed and that will probably be some 800-1000 MByte/s for sync writes, maybe 1000-1500 MByte/s for async writes. Optane is also durable in the sense of petabytes written, and that is what you want for backup storage. Your NVMe SSDs might slow everything down depending on what drive you use, should their write latency shoot up because it is doing some garbage collecting or stuff.

For system OS a mirrored 960 GB NVME is probably total overkill.

If you need more storage, either add a second vdev to the pool that already has the RAIDZ2, or, and that is probably wiser and more performant, add a second server and make that known to Veeam. Split backup jobs to go to either storage with parallel jobs pushing data to both server also in parallel. If you run a SAN make sure to use a Veeam Backup Proxy VM with sufficient CPU cores (8+) and sufficient RAM to precompress all data before it is sent from VM host to Veeam storage.
 
  • Like
Reactions: nikalai