TrueNas core vs TrueNas Scale iSCSI performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nev_neo

Active Member
Jul 31, 2013
184
58
28
I recently had the opportunity to test out a server with 24 10K drives for VM storage.
It was setup as 11vdevs Raid10 with 2 hot spares connected to esxi via dual 10gbit (4 paths vlan'd).
Single host was connected via 4x1gig connections, RoundRobin with iops=1.

I loaded it up with Core 13.0-u6.2 first then upgraded/switched to Scale (dragonfish).
Tests were run on a VM running 2022 with PV scsi drives.

It seems Core was more quite a bit more performant in the RND (Read/Write) metrics - which I think matters more for VM storage.

Core:
[Read]
SEQ 1MiB (Q= 8, T= 1): 468.461 MB/s [ 446.8 IOPS] < 17859.14 us>
SEQ 1MiB (Q= 1, T= 1): 206.553 MB/s [ 197.0 IOPS] < 5070.92 us>
RND 4KiB (Q= 32, T= 1): 408.790 MB/s [ 99802.2 IOPS] < 320.26 us>
RND 4KiB (Q= 1, T= 1): 25.302 MB/s [ 6177.2 IOPS] < 161.36 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 468.147 MB/s [ 446.5 IOPS] < 17855.56 us>
SEQ 1MiB (Q= 1, T= 1): 215.602 MB/s [ 205.6 IOPS] < 4858.70 us>
RND 4KiB (Q= 32, T= 1): 239.211 MB/s [ 58401.1 IOPS] < 547.17 us>
RND 4KiB (Q= 1, T= 1): 23.341 MB/s [ 5698.5 IOPS] < 174.77 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 523.640 MB/s [ 499.4 IOPS] < 15951.04 us>
SEQ 1MiB (Q= 1, T= 1): 210.139 MB/s [ 200.4 IOPS] < 4986.20 us>
RND 4KiB (Q= 32, T= 1): 289.934 MB/s [ 70784.7 IOPS] < 451.24 us>
RND 4KiB (Q= 1, T= 1): 24.156 MB/s [ 5897.5 IOPS] < 168.89 us>

SCALE:

[Read]
SEQ 1MiB (Q= 8, T= 1): 468.878 MB/s [ 447.2 IOPS] < 17851.04 us>
SEQ 1MiB (Q= 1, T= 1): 196.298 MB/s [ 187.2 IOPS] < 5336.07 us>
RND 4KiB (Q= 32, T= 1): 319.338 MB/s [ 77963.4 IOPS] < 408.65 us>
RND 4KiB (Q= 1, T= 1): 9.882 MB/s [ 2412.6 IOPS] < 412.82 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 465.517 MB/s [ 444.0 IOPS] < 17940.84 us>
SEQ 1MiB (Q= 1, T= 1): 210.540 MB/s [ 200.8 IOPS] < 4976.17 us>
RND 4KiB (Q= 32, T= 1): 193.882 MB/s [ 47334.5 IOPS] < 675.05 us>
RND 4KiB (Q= 1, T= 1): 9.571 MB/s [ 2336.7 IOPS] < 426.07 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 540.060 MB/s [ 515.0 IOPS] < 15473.02 us>
SEQ 1MiB (Q= 1, T= 1): 203.832 MB/s [ 194.4 IOPS] < 5137.22 us>
RND 4KiB (Q= 32, T= 1): 277.687 MB/s [ 67794.7 IOPS] < 471.18 us>
RND 4KiB (Q= 1, T= 1): 10.238 MB/s [ 2499.5 IOPS] < 398.73 us>

I'm not sure what this performance disparity is about. Technically everything should've been served up off Arc cache.

I didnt get a chance to test any further as I was running out of time.

Thoughts ?
 

nasbdh9

Active Member
Aug 4, 2019
186
120
43
can try core 13.3 to see if the performance is the same as scale, since the main components of 13.3 use the same code as scale.
I believe some changes in ixsystems caused the performance degradation on 4k.
 

TRACKER

Active Member
Jan 14, 2019
289
122
43
core may be still better performing than scale, but scale has one trick under its sleeves, it is called 'RDMA' ;)
 

nev_neo

Active Member
Jul 31, 2013
184
58
28
can try core 13.3 to see if the performance is the same as scale, since the main components of 13.3 use the same code as scale.
I believe some changes in ixsystems caused the performance degradation on 4k.
I'm going to have to try this again on another system. I think I did try 13.3 on another system - but didnt really see any change between 13.0 and 13.3 in terms of iscsi performance.

Honestly this test was just to see how well Scale had supposedly improved but I was surprised that there was an actual regression.
The test sample was just 1gb, so it should've been served up from ARC.
I wish I had more time to test this out.
 

nev_neo

Active Member
Jul 31, 2013
184
58
28
Yes, it is not iWARP though, it is iSER (iSCSI over RDMA using RoCE v1/v2).
Hey Tracker, I'm at the stage where i'm back at testing Core vs Scale. Do you have any links/howtos on implementing iSER in scale ?
 

TRACKER

Active Member
Jan 14, 2019
289
122
43
Hey Tracker, I'm at the stage where i'm back at testing Core vs Scale. Do you have any links/howtos on implementing iSER in scale ?
Hi nev_neo :)
i've used these two to configure my TNS and ESXi:
 

nev_neo

Active Member
Jul 31, 2013
184
58
28
Ok, that made a pretty BIG difference.
I'm still testing but preliminary results are Scale is on par or quite a bit faster than Core with iSER connections.

My test system is a Supermicro 24bay on a SAS 3 expander backplane (I know), going to an Epyc system:
Supermicro H11SSL running 7402p
256 GB ECC
1 x Samsung 250gb evo boot drive
2 x p1600 optane drives for MetaData vDev
24 x 3.84 TB SAS3 drives - various used Netapp storage drives from ebay.
2 x dualport 25gbe Connectx-4 cards

The disks are split up into 3xz1 vdevs of 7 drives each. 3 drives are hotspares.

Internal testing shows I'm maxing out the backplane ( I think)

Code:
sudo fio --bs=128k --direct=1 --directory=/mnt/Z3/test --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based
randrw: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=posixaio, iodepth=32
...
fio-3.33
Jobs: 12 (f=12): [m(12)][100.0%][r=7676MiB/s,w=7676MiB/s][r=61.4k,w=61.4k IOPS][eta 00m:00s] 
randrw: (groupid=0, jobs=12): err= 0: pid=1140608: Mon Mar 17 16:17:30 2025
  read: IOPS=62.1k, BW=7764MiB/s (8141MB/s)(455GiB/60006msec)
   bw (  MiB/s): min= 5163, max=11738, per=100.00%, avg=7764.62, stdev=104.41, samples=1428
   iops        : min=41308, max=93907, avg=62114.26, stdev=835.33, samples=1428
  write: IOPS=62.2k, BW=7770MiB/s (8148MB/s)(455GiB/60006msec); 0 zone resets
   bw (  MiB/s): min= 5178, max=11671, per=100.00%, avg=7770.72, stdev=104.38, samples=1428
   iops        : min=41422, max=93366, avg=62162.97, stdev=835.05, samples=1428
  cpu          : usr=4.31%, sys=1.03%, ctx=1229122, majf=0, minf=438
  IO depths    : 1=0.6%, 2=1.3%, 4=3.1%, 8=16.8%, 16=61.9%, 32=16.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=0.7%, 16=1.0%, 32=2.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=3727001,3730025,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=7764MiB/s (8141MB/s), 7764MiB/s-7764MiB/s (8141MB/s-8141MB/s), io=455GiB (489GB), run=60006-60006msec
  WRITE: bw=7770MiB/s (8148MB/s), 7770MiB/s-7770MiB/s (8148MB/s-8148MB/s), io=455GiB (489GB), run=60006-60006msec
 

nev_neo

Active Member
Jul 31, 2013
184
58
28
I need to do some further testing as my current test environment is running in my working homelab environment - will need to isolate from regular network traffic. I have a host already, just need to reconfigure it.

Anyway, the test data so far:

Truenas Core 13.3 -


[Read]
SEQ 1MiB (Q= 8, T= 1): 3797.210 MB/s [ 3621.3 IOPS] < 2207.35 us>
SEQ 1MiB (Q= 1, T= 1): 1932.632 MB/s [ 1843.1 IOPS] < 542.09 us>
RND 4KiB (Q= 32, T= 1): 393.436 MB/s [ 96053.7 IOPS] < 332.67 us>
RND 4KiB (Q= 1, T= 1): 33.865 MB/s [ 8267.8 IOPS] < 120.75 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 5188.120 MB/s [ 4947.8 IOPS] < 1610.10 us>
SEQ 1MiB (Q= 1, T= 1): 1752.733 MB/s [ 1671.5 IOPS] < 597.62 us>
RND 4KiB (Q= 32, T= 1): 294.705 MB/s [ 71949.5 IOPS] < 444.21 us>
RND 4KiB (Q= 1, T= 1): 31.200 MB/s [ 7617.2 IOPS] < 131.05 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 4180.296 MB/s [ 3986.6 IOPS] < 1999.19 us>
SEQ 1MiB (Q= 1, T= 1): 1955.199 MB/s [ 1864.6 IOPS] < 535.51 us>
RND 4KiB (Q= 32, T= 1): 336.154 MB/s [ 82068.8 IOPS] < 386.57 us>
RND 4KiB (Q= 1, T= 1): 32.341 MB/s [ 7895.8 IOPS] < 126.39 us>

Profile: Default

Test: 16 GiB (x1) [E: 0% (0/80GiB)]




Truenas Scale 24.10.2 (Regular)


[Read]
SEQ 1MiB (Q= 8, T= 1): 1685.724 MB/s [ 1607.6 IOPS] < 4969.76 us>
SEQ 1MiB (Q= 1, T= 1): 732.363 MB/s [ 698.4 IOPS] < 1430.88 us>
RND 4KiB (Q= 32, T= 1): 128.744 MB/s [ 31431.6 IOPS] < 1017.37 us>
RND 4KiB (Q= 1, T= 1): 25.188 MB/s [ 6149.4 IOPS] < 162.34 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 1232.908 MB/s [ 1175.8 IOPS] < 6777.40 us>
SEQ 1MiB (Q= 1, T= 1): 649.591 MB/s [ 619.5 IOPS] < 1612.87 us>
RND 4KiB (Q= 32, T= 1): 140.511 MB/s [ 34304.4 IOPS] < 932.18 us>
RND 4KiB (Q= 1, T= 1): 22.860 MB/s [ 5581.1 IOPS] < 178.91 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 1455.893 MB/s [ 1388.4 IOPS] < 5742.72 us>
SEQ 1MiB (Q= 1, T= 1): 585.283 MB/s [ 558.2 IOPS] < 1790.07 us>
RND 4KiB (Q= 32, T= 1): 134.036 MB/s [ 32723.6 IOPS] < 976.81 us>
RND 4KiB (Q= 1, T= 1): 24.325 MB/s [ 5938.7 IOPS] < 168.10 us>

Profile: Default

Test: 16 GiB (x1) [E: 0% (0/80GiB)]



Truenas Scale 24.10.2 (iSER)


[Read]
SEQ 1MiB (Q= 8, T= 1): 5449.134 MB/s [ 5196.7 IOPS] < 1537.69 us>
SEQ 1MiB (Q= 1, T= 1): 1368.070 MB/s [ 1304.7 IOPS] < 765.84 us>
RND 4KiB (Q= 32, T= 1): 479.527 MB/s [ 117072.0 IOPS] < 270.68 us>
RND 4KiB (Q= 1, T= 1): 44.106 MB/s [ 10768.1 IOPS] < 92.64 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 3641.920 MB/s [ 3473.2 IOPS] < 2295.59 us>
SEQ 1MiB (Q= 1, T= 1): 1448.776 MB/s [ 1381.7 IOPS] < 722.90 us>
RND 4KiB (Q= 32, T= 1): 246.009 MB/s [ 60060.8 IOPS] < 532.16 us>
RND 4KiB (Q= 1, T= 1): 29.892 MB/s [ 7297.9 IOPS] < 136.71 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 5162.027 MB/s [ 4922.9 IOPS] < 1618.09 us>
SEQ 1MiB (Q= 1, T= 1): 1307.544 MB/s [ 1247.0 IOPS] < 801.07 us>
RND 4KiB (Q= 32, T= 1): 443.406 MB/s [ 108253.4 IOPS] < 294.72 us>
RND 4KiB (Q= 1, T= 1): 36.896 MB/s [ 9007.8 IOPS] < 110.73 us>
 

TRACKER

Active Member
Jan 14, 2019
289
122
43
Guys, beware!
In Truenas Scale 25.04 iSER feature is disabled by purpose because it is "Enterprise" feature.
So in case you upgrade from 24.10 to 25.04 and you are not "Enterprise" you will be scr**ed.
 

TRACKER

Active Member
Jan 14, 2019
289
122
43
I don't think they are blocking you from manually setting up iSER like the person on level1tech did.
I've read the article but i did not get how to do it, can you share some details please?

p.s. i've found it :)
 
Last edited: