TrueNAS Performance Questions

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
Hi All,

I recently started foolin around with an old server in my lab. I'm looking to see if my performance numbers are out-of-whack because they seem low to me. I've built a poor-man's all flash array using the following
  • A Supermicro 216 Chassis
  • A Supermicro x8dth-6 motherboard (SAS2008-IT onboard)
  • 2 additional LSI 9211-8is
  • 2x Xeon X5680s
  • 96GB of RAM
  • 22 Crucial BX500 120GB SSDs (11 mirrored VDEVs) ashift=12
  • Intel X520-DA2
  • Running TrueNAS Scale latest beta
1634188677285.png

fio results:
1634189702797.png

Sooo 16,000 IOPS and 2000 MiB/s read and write...

A ZVOL Mounted via ISCSI on Windows over a 10gigabit network:
1634189720939.png

As a point of reference I compared these results to my production server. The production server is faster than the all flash server? This is NOT apples-to-apples but I'm rather confused??
  • Dell R720
  • H310 IT Mode
  • LSI 9205-8i IT Mode
  • 2x Xeon E5-2920s
  • 192GB RAM
  • Fusion Pool with 12x4TB WD RED (CMR) drives and 2x Samsung SM953 480GB NVME SSDs
  • TrueNAS CORE 12.0U6
1634189072350.png

Production server FIO results
1634189399422.png

Sooo 17,000 IOPS and 2100 MiB/s read and write..

A SMB share over the same 10gigabit network to the same desktop
1634188771167.png
 

Attachments

Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,248
1,701
113
CA

They're $20 SSD known for bad performance... I mean it scores above a HDD, but barely in many of these tests...

For all those HBA and drives, and electric usage (if that matters to you) you're way way better off with a single NVME from a performance stand point.
 
  • Like
Reactions: nickf1227

Rand__

Well-Known Member
Mar 6, 2014
5,600
1,228
113
With 8 jobs you're not utilizing your 11 vdevs fully, mirror performance depends on #jobs, its not X to 1 pool but X to Y mirror vdevs.
Also it might be test size dependend, a lot of your data might might the SSD cache on your prod box
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
I'm going to try and re-do the testing with some 9205-IT mode cards, but since my system is PCI-E 2.0 I'm not sure if it will have an appreciable benefit. SAS2308 is generally better than SAS2008 chips in any case. But still, I was expecting closer to 3500 MB/s...at least inline with modern M.2 performance...with 3 separate 2008 controllers and 24 drives with no SAS expander I'm not sure why that's not possible??

I can also try and swap the drives for some old 180GB Intel SSD 520s I have to see if that changes anything. The CT120BX500SSD1 SSDs I have are DRAM-less and any efficiency increase with more modern NAND vs the old 520s is probably lost to that. Worth experimenting, I wish I had a 3rd set of drives to compare against but that's all I have to play with.

1634262019138.png
1634262026409.png

I've also bid on an X9DRi-LN4F so I can get some faster Ivy Bridge Xeons (E5-2667 v2?) and see if that helps after I do the above testing...

Any other thoughts are appreciated, I know this is all older hardware but I'm not sure what the biggest bottleneck is....This should be significantly faster than 12 spinning disks in a Raid-Z2 by the nature of the fact that they in a mirrors pool and they are SSDs....?


They're $20 SSD known for bad performance... I mean it scores above a HDD, but barely in many of these tests...

For all those HBA and drives, and electric usage (if that matters to you) you're way way better off with a single NVME from a performance stand point.
Yea, it's more of an experiment :)
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
So I did the SSD swapout to the SSD Intel 520s

1634274415467.png
Now we are at 12K IOPS and 1500 MB/s read and write.

Crystal Disk Mark over ISCSI performance
1634274736120.png

I know these are really old Sandforce drives, so I wasn't sure what to expect here. Still, performance seems rather low....

I will try and swap in the LSI 9205s tomorrow. Any other ideas are appreciated.
 
Last edited:

Terry Wallace

PsyOps SysOp
Aug 13, 2018
177
106
43
Central Time Zone
Your production server is raidz2 not a collection of mirrored vdevs. Assuming your cpu is fast enough the raidz2 is going to be hitting more disks for data access than your mirrored vdevs are. Also a consideration.
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
Your production server is raidz2 not a collection of mirrored vdevs. Assuming your cpu is fast enough the raidz2 is going to be hitting more disks for data access than your mirrored vdevs are. Also a consideration.
I'm not sure how that's possible? A RAID Z2 of 12 Hard Drives isn't going to "hit more disks" than a mirrored array of 24 SSDs? If anything, on I'm hitting twice as many disks on read and an equal on writes...but there's no parity calculation so I should be inherently faster.

Additional testing has ensued...

Replaced 9211s with 9205s
1634441948743.png

1634441974170.png

13k IOPS and 1700MB/s

Less than with the Crucial drives, faster than 9211s.

Won the action for the LGA-2011 board. Will test with the E5-2620s that it came with (the same CPUs that are in my production server).
1634441997771.png

RAM is the same exact DIMMS as are in my production server. Experiment is on hold until that comes. If it's not significantly faster with the new board I'm sorta at a loss??

Is my testing methodology not correct?
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
So my new motherboard arrived today...and as I had hoped... It is significantly faster...

Code:
fio --bs=128k --direct=1 --directory=/mnt/lol/fio --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based
1634876081434.png
Sitting at 25.k IOPS and 3215MB/s

I wish I had some SAS3 cards/SAS3 backplane to see if performance scales JUST by moving to SAS3 hardware...but they are still pretty pricey

I think I will pickup some faster single threaded CPUs next and see what benefit that gives me. 2620s don't exactly scream, and the X5680s I had in the other board are probably faster. It's kinda silly, because over ISCSI it's still only a little faster than a single SATA SSD...
1634876089931.png
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
I'd pick up one of these Intel Xeon CPU E5-2667 V2 3.30GHz 25MB Cache 8 Core LGA2011 Processor SR19W 757897191358 | eBay for $65. assuming the board take a v2 cpu. That's about the fastest 2600v2 series (short of some expensive workstation models) 2620's are the "we'll ship you a server that runs... just not very fast" option.
Bought V1s because mobo won't work with V2s...but they were DIRT cheap
Intel SR0KP Xeon E5-2667 2.90GHz 4M/15M 6-Core Socket 2011 CPU Processor LGA2011 | eBay

Should make a pretty substantial increase?
1635000684662.png
 

acquacow

Well-Known Member
Feb 15, 2017
619
326
63
40
Are you using SMB Multichannel or anything? My homelab TrueNAS box posts better numbers than that. I max out 1GB/sec writing to a raidz1 of 4 old intel S3500 SSDs, and about 750MB/sec to 8 HGST 4TB drives in a raidz2 on an older LSI 8xxx in IT mode.

Just a single CPU supermicro 1541 xeon-d board with 128B of DRAM.

Single CPU will usually yield better I/O than dual or quad CPU due to the lack of NUMA overhead.
Pull a CPU out of that thing and test again ;)

Did you disable autotune and put all the 10gige tunables in sysctl?
 

nickf1227

Active Member
Sep 23, 2015
154
79
28
30
Actually, I figured out what's wrong with my testing methodology. I forgot my desktop's NIC is PCI-E 2.0x8 card in a PCI-E 4.0x4 slot, not a x8 slot... So using that as an endpoint isn't gonna work.