TrueNAS Performance Questions

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Hi All,

I recently started foolin around with an old server in my lab. I'm looking to see if my performance numbers are out-of-whack because they seem low to me. I've built a poor-man's all flash array using the following
  • A Supermicro 216 Chassis
  • A Supermicro x8dth-6 motherboard (SAS2008-IT onboard)
  • 2 additional LSI 9211-8is
  • 2x Xeon X5680s
  • 96GB of RAM
  • 22 Crucial BX500 120GB SSDs (11 mirrored VDEVs) ashift=12
  • Intel X520-DA2
  • Running TrueNAS Scale latest beta
1634188677285.png

fio results:
1634189702797.png

Sooo 16,000 IOPS and 2000 MiB/s read and write...

A ZVOL Mounted via ISCSI on Windows over a 10gigabit network:
1634189720939.png

As a point of reference I compared these results to my production server. The production server is faster than the all flash server? This is NOT apples-to-apples but I'm rather confused??
  • Dell R720
  • H310 IT Mode
  • LSI 9205-8i IT Mode
  • 2x Xeon E5-2920s
  • 192GB RAM
  • Fusion Pool with 12x4TB WD RED (CMR) drives and 2x Samsung SM953 480GB NVME SSDs
  • TrueNAS CORE 12.0U6
1634189072350.png

Production server FIO results
1634189399422.png

Sooo 17,000 IOPS and 2100 MiB/s read and write..

A SMB share over the same 10gigabit network to the same desktop
1634188771167.png
 

Attachments

Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113

They're $20 SSD known for bad performance... I mean it scores above a HDD, but barely in many of these tests...

For all those HBA and drives, and electric usage (if that matters to you) you're way way better off with a single NVME from a performance stand point.
 
  • Like
Reactions: nickf1227

Rand__

Well-Known Member
Mar 6, 2014
6,626
1,767
113
With 8 jobs you're not utilizing your 11 vdevs fully, mirror performance depends on #jobs, its not X to 1 pool but X to Y mirror vdevs.
Also it might be test size dependend, a lot of your data might might the SSD cache on your prod box
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
I'm going to try and re-do the testing with some 9205-IT mode cards, but since my system is PCI-E 2.0 I'm not sure if it will have an appreciable benefit. SAS2308 is generally better than SAS2008 chips in any case. But still, I was expecting closer to 3500 MB/s...at least inline with modern M.2 performance...with 3 separate 2008 controllers and 24 drives with no SAS expander I'm not sure why that's not possible??

I can also try and swap the drives for some old 180GB Intel SSD 520s I have to see if that changes anything. The CT120BX500SSD1 SSDs I have are DRAM-less and any efficiency increase with more modern NAND vs the old 520s is probably lost to that. Worth experimenting, I wish I had a 3rd set of drives to compare against but that's all I have to play with.

1634262019138.png
1634262026409.png

I've also bid on an X9DRi-LN4F so I can get some faster Ivy Bridge Xeons (E5-2667 v2?) and see if that helps after I do the above testing...

Any other thoughts are appreciated, I know this is all older hardware but I'm not sure what the biggest bottleneck is....This should be significantly faster than 12 spinning disks in a Raid-Z2 by the nature of the fact that they in a mirrors pool and they are SSDs....?


They're $20 SSD known for bad performance... I mean it scores above a HDD, but barely in many of these tests...

For all those HBA and drives, and electric usage (if that matters to you) you're way way better off with a single NVME from a performance stand point.
Yea, it's more of an experiment :)
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
So I did the SSD swapout to the SSD Intel 520s

1634274415467.png
Now we are at 12K IOPS and 1500 MB/s read and write.

Crystal Disk Mark over ISCSI performance
1634274736120.png

I know these are really old Sandforce drives, so I wasn't sure what to expect here. Still, performance seems rather low....

I will try and swap in the LSI 9205s tomorrow. Any other ideas are appreciated.
 
Last edited:

Terry Wallace

PsyOps SysOp
Aug 13, 2018
197
118
43
Central Time Zone
Your production server is raidz2 not a collection of mirrored vdevs. Assuming your cpu is fast enough the raidz2 is going to be hitting more disks for data access than your mirrored vdevs are. Also a consideration.
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Your production server is raidz2 not a collection of mirrored vdevs. Assuming your cpu is fast enough the raidz2 is going to be hitting more disks for data access than your mirrored vdevs are. Also a consideration.
I'm not sure how that's possible? A RAID Z2 of 12 Hard Drives isn't going to "hit more disks" than a mirrored array of 24 SSDs? If anything, on I'm hitting twice as many disks on read and an equal on writes...but there's no parity calculation so I should be inherently faster.

Additional testing has ensued...

Replaced 9211s with 9205s
1634441948743.png

1634441974170.png

13k IOPS and 1700MB/s

Less than with the Crucial drives, faster than 9211s.

Won the action for the LGA-2011 board. Will test with the E5-2620s that it came with (the same CPUs that are in my production server).
1634441997771.png

RAM is the same exact DIMMS as are in my production server. Experiment is on hold until that comes. If it's not significantly faster with the new board I'm sorta at a loss??

Is my testing methodology not correct?
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
So my new motherboard arrived today...and as I had hoped... It is significantly faster...

Code:
fio --bs=128k --direct=1 --directory=/mnt/lol/fio --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based
1634876081434.png
Sitting at 25.k IOPS and 3215MB/s

I wish I had some SAS3 cards/SAS3 backplane to see if performance scales JUST by moving to SAS3 hardware...but they are still pretty pricey

I think I will pickup some faster single threaded CPUs next and see what benefit that gives me. 2620s don't exactly scream, and the X5680s I had in the other board are probably faster. It's kinda silly, because over ISCSI it's still only a little faster than a single SATA SSD...
1634876089931.png
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
I'd pick up one of these Intel Xeon CPU E5-2667 V2 3.30GHz 25MB Cache 8 Core LGA2011 Processor SR19W 757897191358 | eBay for $65. assuming the board take a v2 cpu. That's about the fastest 2600v2 series (short of some expensive workstation models) 2620's are the "we'll ship you a server that runs... just not very fast" option.
Bought V1s because mobo won't work with V2s...but they were DIRT cheap
Intel SR0KP Xeon E5-2667 2.90GHz 4M/15M 6-Core Socket 2011 CPU Processor LGA2011 | eBay

Should make a pretty substantial increase?
1635000684662.png
 

acquacow

Well-Known Member
Feb 15, 2017
784
439
63
42
Are you using SMB Multichannel or anything? My homelab TrueNAS box posts better numbers than that. I max out 1GB/sec writing to a raidz1 of 4 old intel S3500 SSDs, and about 750MB/sec to 8 HGST 4TB drives in a raidz2 on an older LSI 8xxx in IT mode.

Just a single CPU supermicro 1541 xeon-d board with 128B of DRAM.

Single CPU will usually yield better I/O than dual or quad CPU due to the lack of NUMA overhead.
Pull a CPU out of that thing and test again ;)

Did you disable autotune and put all the 10gige tunables in sysctl?
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Actually, I figured out what's wrong with my testing methodology. I forgot my desktop's NIC is PCI-E 2.0x8 card in a PCI-E 4.0x4 slot, not a x8 slot... So using that as an endpoint isn't gonna work.
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Received new CPUs already
1635221316428.png
26K IOPS 3272MB/s
Performance hasn't changed?
 
Last edited:
  • Like
Reactions: T_Minus

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
To be clear, it does seem to be scaling linearly, so maybe this is as fast as these drives go?

This is a single drive:
1635301062013.png

With 11 VDEVs of 2-drive mirrors I am getting 10x the performance

I've also tested with all 22 drives in a single RAIDZ1
1635301070257.png
23k IOPS and 2800MB/s
Performance is pretty close to mirrors, which goes against conventional wisdom I've seen here. It was always my understanding that mirrors had a high cost in storage efficiency but were always faster than RAIDZ. Although, we are talking about 22 drives and only a single drive worth of parity...not exactly production ready.

Two Raid-Z1 VDEVs of 11 drives each yields the best performance/storage efficiency so far
1635301077247.png
28k IOPS and 3500MB/s.

Still, not sure if I would put into production 1 drive of parity with 11 disks.

So next I made a 20-drive ZPool with 4 5-drive vdevs in a raid-z1
1635301094377.png
Losing 2 drives as "hot spares" plus 3 for parity for a total of 5 drives is still better storage efficiency than the mirrors(which lost 11), and still faster.
26K IOPS and 3200MB/s

Finally, I made an 11-drive-per-vdev RAIDZ-2 of 2 VDEVs. This offers better storage efficiency than the 3 RAIDZ-1 VDEV arrangement.
1635301102213.png
26k IOPS and 3200MB/s still....which is still on par with the mirror array *and from my test appears to be more consistent*

So some takeaways...
  • I'm not going to break 30k IOPS anytime soon
  • SAS3 cards may be an answer, but I don't honestly think they will be
  • I have 5 PCIE 16x slots I can put NVME drives and I'm fairly certain that I can do PCI-E bifurcation in this board so I may be able to do 10xNVME drives on this platform to see what that does.
  • If anyone has sone 256gb nvme drives from old laptops or something and you want to donate to the cause for science I'm here xD

I hope this helps someone later...
 

Attachments

  • Like
Reactions: T_Minus

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
For giggles, I wanted to get every drop of performance out of this. I put all 22 drives in a single vdev striped
1635306921629.png
38K IOPS 4700MB/s

The bottleneck for sure does not appear to be the HBAs...
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Thanks for sharing :) we haven't had any of these classic try-it-all for different performance threads in awhile, and I missss them :D even if they're not 1M+IOP testing :D Still fun and great to learn about bottlenecks etc.

Curious... how are your drive pairs on the mirror pool configured? All drives mirrored with other drives on the same HBA?
What about HBA PCIE layout configuration? All on CPU1?
CPU tuned\bios for performance mode?
 
  • Like
Reactions: ecosse

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Thanks for sharing :) we haven't had any of these classic try-it-all for different performance threads in awhile, and I missss them :D even if they're not 1M+IOP testing :D Still fun and great to learn about bottlenecks etc.

Curious... how are your drive pairs on the mirror pool configured? All drives mirrored with other drives on the same HBA?
What about HBA PCIE layout configuration? All on CPU1?
CPU tuned\bios for performance mode?
Good questions. I actually don't know the answer to whether or not the mirrors are on the same HBAs or not, they should be but I would have to check the alignment of the SAS lanes to confirm.

For the second part,
All HBA traffic is going through these 3 slots:
1635374988642.png

Which, according to the manual are all CPU 1:1635375040987.png

But, now that I am seeing that I have a problem I didn't realize I had! one of the HBAs is in a x4 slot
DOH! Thank you for asking, or I wouldn't have noticed that wasn't a real x8 slot....More testing will ensue...
 
  • Like
Reactions: T_Minus

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
So I moved the network card to CPU1 Slot 1 and the HBAs to the CPU2 slots:
1635376778254.png

So I redid my testing as an 11-drive-per-vdev RAIDZ-2 of 2 VDEVs.
1635377108868.png
Performance is about the same.

I re-did the mirrors configuration as well:
1635377528162.png
Performance is about the same.
 
  • Like
Reactions: T_Minus