TrueNas core vs TrueNas Scale iSCSI performance

nev_neo · Sep 27, 2024

I recently had the opportunity to test out a server with 24 10K drives for VM storage.
It was setup as 11vdevs Raid10 with 2 hot spares connected to esxi via dual 10gbit (4 paths vlan'd).
Single host was connected via 4x1gig connections, RoundRobin with iops=1.

I loaded it up with Core 13.0-u6.2 first then upgraded/switched to Scale (dragonfish).
Tests were run on a VM running 2022 with PV scsi drives.

It seems Core was more quite a bit more performant in the RND (Read/Write) metrics - which I think matters more for VM storage.

Core:
[Read]
SEQ 1MiB (Q= 8, T= 1): 468.461 MB/s [ 446.8 IOPS] < 17859.14 us>
SEQ 1MiB (Q= 1, T= 1): 206.553 MB/s [ 197.0 IOPS] < 5070.92 us>
RND 4KiB (Q= 32, T= 1): 408.790 MB/s [ 99802.2 IOPS] < 320.26 us>
RND 4KiB (Q= 1, T= 1): 25.302 MB/s [ 6177.2 IOPS] < 161.36 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 468.147 MB/s [ 446.5 IOPS] < 17855.56 us>
SEQ 1MiB (Q= 1, T= 1): 215.602 MB/s [ 205.6 IOPS] < 4858.70 us>
RND 4KiB (Q= 32, T= 1): 239.211 MB/s [ 58401.1 IOPS] < 547.17 us>
RND 4KiB (Q= 1, T= 1): 23.341 MB/s [ 5698.5 IOPS] < 174.77 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 523.640 MB/s [ 499.4 IOPS] < 15951.04 us>
SEQ 1MiB (Q= 1, T= 1): 210.139 MB/s [ 200.4 IOPS] < 4986.20 us>
RND 4KiB (Q= 32, T= 1): 289.934 MB/s [ 70784.7 IOPS] < 451.24 us>
RND 4KiB (Q= 1, T= 1): 24.156 MB/s [ 5897.5 IOPS] < 168.89 us>

SCALE:
[Read]
SEQ 1MiB (Q= 8, T= 1): 468.878 MB/s [ 447.2 IOPS] < 17851.04 us>
SEQ 1MiB (Q= 1, T= 1): 196.298 MB/s [ 187.2 IOPS] < 5336.07 us>
RND 4KiB (Q= 32, T= 1): 319.338 MB/s [ 77963.4 IOPS] < 408.65 us>
RND 4KiB (Q= 1, T= 1): 9.882 MB/s [ 2412.6 IOPS] < 412.82 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 465.517 MB/s [ 444.0 IOPS] < 17940.84 us>
SEQ 1MiB (Q= 1, T= 1): 210.540 MB/s [ 200.8 IOPS] < 4976.17 us>
RND 4KiB (Q= 32, T= 1): 193.882 MB/s [ 47334.5 IOPS] < 675.05 us>
RND 4KiB (Q= 1, T= 1): 9.571 MB/s [ 2336.7 IOPS] < 426.07 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 540.060 MB/s [ 515.0 IOPS] < 15473.02 us>
SEQ 1MiB (Q= 1, T= 1): 203.832 MB/s [ 194.4 IOPS] < 5137.22 us>
RND 4KiB (Q= 32, T= 1): 277.687 MB/s [ 67794.7 IOPS] < 471.18 us>
RND 4KiB (Q= 1, T= 1): 10.238 MB/s [ 2499.5 IOPS] < 398.73 us>

I'm not sure what this performance disparity is about. Technically everything should've been served up off Arc cache.

I didnt get a chance to test any further as I was running out of time.

Thoughts ?

nasbdh9 · Sep 27, 2024

can try core 13.3 to see if the performance is the same as scale, since the main components of 13.3 use the same code as scale.
I believe some changes in ixsystems caused the performance degradation on 4k.

TRACKER · Sep 27, 2024

core may be still better performing than scale, but scale has one trick under its sleeves, it is called 'RDMA'

nev_neo · Sep 28, 2024

TRACKER said:
core may be still better performing than scale, but scale has one trick under its sleeves, it is called 'RDMA'

Does it really ? Last I checked iWarp was not supported on any of them.

nev_neo · Sep 28, 2024

nasbdh9 said:
can try core 13.3 to see if the performance is the same as scale, since the main components of 13.3 use the same code as scale.
I believe some changes in ixsystems caused the performance degradation on 4k.

I'm going to have to try this again on another system. I think I did try 13.3 on another system - but didnt really see any change between 13.0 and 13.3 in terms of iscsi performance.

Honestly this test was just to see how well Scale had supposedly improved but I was surprised that there was an actual regression.
The test sample was just 1gb, so it should've been served up from ARC.
I wish I had more time to test this out.

TRACKER · Sep 28, 2024

nev_neo said:
Does it really ? Last I checked iWarp was not supported on any of them.

Yes, it is not iWARP though, it is iSER (iSCSI over RDMA using RoCE v1/v2).

nev_neo · Mar 17, 2025

TRACKER said:
Yes, it is not iWARP though, it is iSER (iSCSI over RDMA using RoCE v1/v2).

Hey Tracker, I'm at the stage where i'm back at testing Core vs Scale. Do you have any links/howtos on implementing iSER in scale ?

gea · Mar 17, 2025

maybe NVMEoF/RoCE is an option
https://www.reddit.com/r/truenas/comments/1fh3rfl

TRACKER · Mar 17, 2025

nev_neo said:
Hey Tracker, I'm at the stage where i'm back at testing Core vs Scale. Do you have any links/howtos on implementing iSER in scale ?

Hi nev_neo

i've used these two to configure my TNS and ESXi:

100 Gbe & TrueNAS Scale 23.10 iSCSI - Performance Unleashed

Hi All, I thought I would post a quick “how-to” for those interested in getting better performance out of TrueNAS Scale for iSCSI workloads. TL;DR I was able to get almost 4GB/s throughput to a single VMware VM using Mellanox ConnectX-5 cards and TrueNAS Scale 23.10. That’s more than double...

forum.level1techs.com

Configuring iSCSI and iSER Adapters and Storage

Before ESXi can work with iSCSI SAN, you must set up your iSCSI environment.

techdocs.broadcom.com

nev_neo · Mar 17, 2025

Ok, that made a pretty BIG difference.
I'm still testing but preliminary results are Scale is on par or quite a bit faster than Core with iSER connections.

My test system is a Supermicro 24bay on a SAS 3 expander backplane (I know), going to an Epyc system:
Supermicro H11SSL running 7402p
256 GB ECC
1 x Samsung 250gb evo boot drive
2 x p1600 optane drives for MetaData vDev
24 x 3.84 TB SAS3 drives - various used Netapp storage drives from ebay.
2 x dualport 25gbe Connectx-4 cards

The disks are split up into 3xz1 vdevs of 7 drives each. 3 drives are hotspares.

Internal testing shows I'm maxing out the backplane ( I think)

Code:

sudo fio --bs=128k --direct=1 --directory=/mnt/Z3/test --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based
randrw: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=posixaio, iodepth=32
...
fio-3.33
Jobs: 12 (f=12): [m(12)][100.0%][r=7676MiB/s,w=7676MiB/s][r=61.4k,w=61.4k IOPS][eta 00m:00s] 
randrw: (groupid=0, jobs=12): err= 0: pid=1140608: Mon Mar 17 16:17:30 2025
  read: IOPS=62.1k, BW=7764MiB/s (8141MB/s)(455GiB/60006msec)
   bw (  MiB/s): min= 5163, max=11738, per=100.00%, avg=7764.62, stdev=104.41, samples=1428
   iops        : min=41308, max=93907, avg=62114.26, stdev=835.33, samples=1428
  write: IOPS=62.2k, BW=7770MiB/s (8148MB/s)(455GiB/60006msec); 0 zone resets
   bw (  MiB/s): min= 5178, max=11671, per=100.00%, avg=7770.72, stdev=104.38, samples=1428
   iops        : min=41422, max=93366, avg=62162.97, stdev=835.05, samples=1428
  cpu          : usr=4.31%, sys=1.03%, ctx=1229122, majf=0, minf=438
  IO depths    : 1=0.6%, 2=1.3%, 4=3.1%, 8=16.8%, 16=61.9%, 32=16.4%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=96.1%, 8=0.7%, 16=1.0%, 32=2.2%, 64=0.0%, >=64=0.0%
     issued rwts: total=3727001,3730025,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=7764MiB/s (8141MB/s), 7764MiB/s-7764MiB/s (8141MB/s-8141MB/s), io=455GiB (489GB), run=60006-60006msec
  WRITE: bw=7770MiB/s (8148MB/s), 7770MiB/s-7770MiB/s (8148MB/s-8148MB/s), io=455GiB (489GB), run=60006-60006msec

nev_neo · Mar 17, 2025

I need to do some further testing as my current test environment is running in my working homelab environment - will need to isolate from regular network traffic. I have a host already, just need to reconfigure it.

Anyway, the test data so far:

Truenas Core 13.3 -

[Read]
SEQ 1MiB (Q= 8, T= 1): 3797.210 MB/s [ 3621.3 IOPS] < 2207.35 us>
SEQ 1MiB (Q= 1, T= 1): 1932.632 MB/s [ 1843.1 IOPS] < 542.09 us>
RND 4KiB (Q= 32, T= 1): 393.436 MB/s [ 96053.7 IOPS] < 332.67 us>
RND 4KiB (Q= 1, T= 1): 33.865 MB/s [ 8267.8 IOPS] < 120.75 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 5188.120 MB/s [ 4947.8 IOPS] < 1610.10 us>
SEQ 1MiB (Q= 1, T= 1): 1752.733 MB/s [ 1671.5 IOPS] < 597.62 us>
RND 4KiB (Q= 32, T= 1): 294.705 MB/s [ 71949.5 IOPS] < 444.21 us>
RND 4KiB (Q= 1, T= 1): 31.200 MB/s [ 7617.2 IOPS] < 131.05 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 4180.296 MB/s [ 3986.6 IOPS] < 1999.19 us>
SEQ 1MiB (Q= 1, T= 1): 1955.199 MB/s [ 1864.6 IOPS] < 535.51 us>
RND 4KiB (Q= 32, T= 1): 336.154 MB/s [ 82068.8 IOPS] < 386.57 us>
RND 4KiB (Q= 1, T= 1): 32.341 MB/s [ 7895.8 IOPS] < 126.39 us>

Profile: Default

Test: 16 GiB (x1) [E: 0% (0/80GiB)]

Truenas Scale 24.10.2 (Regular)

[Read]
SEQ 1MiB (Q= 8, T= 1): 1685.724 MB/s [ 1607.6 IOPS] < 4969.76 us>
SEQ 1MiB (Q= 1, T= 1): 732.363 MB/s [ 698.4 IOPS] < 1430.88 us>
RND 4KiB (Q= 32, T= 1): 128.744 MB/s [ 31431.6 IOPS] < 1017.37 us>
RND 4KiB (Q= 1, T= 1): 25.188 MB/s [ 6149.4 IOPS] < 162.34 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 1232.908 MB/s [ 1175.8 IOPS] < 6777.40 us>
SEQ 1MiB (Q= 1, T= 1): 649.591 MB/s [ 619.5 IOPS] < 1612.87 us>
RND 4KiB (Q= 32, T= 1): 140.511 MB/s [ 34304.4 IOPS] < 932.18 us>
RND 4KiB (Q= 1, T= 1): 22.860 MB/s [ 5581.1 IOPS] < 178.91 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 1455.893 MB/s [ 1388.4 IOPS] < 5742.72 us>
SEQ 1MiB (Q= 1, T= 1): 585.283 MB/s [ 558.2 IOPS] < 1790.07 us>
RND 4KiB (Q= 32, T= 1): 134.036 MB/s [ 32723.6 IOPS] < 976.81 us>
RND 4KiB (Q= 1, T= 1): 24.325 MB/s [ 5938.7 IOPS] < 168.10 us>

Profile: Default

Test: 16 GiB (x1) [E: 0% (0/80GiB)]

Truenas Scale 24.10.2 (iSER)

[Read]
SEQ 1MiB (Q= 8, T= 1): 5449.134 MB/s [ 5196.7 IOPS] < 1537.69 us>
SEQ 1MiB (Q= 1, T= 1): 1368.070 MB/s [ 1304.7 IOPS] < 765.84 us>
RND 4KiB (Q= 32, T= 1): 479.527 MB/s [ 117072.0 IOPS] < 270.68 us>
RND 4KiB (Q= 1, T= 1): 44.106 MB/s [ 10768.1 IOPS] < 92.64 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 3641.920 MB/s [ 3473.2 IOPS] < 2295.59 us>
SEQ 1MiB (Q= 1, T= 1): 1448.776 MB/s [ 1381.7 IOPS] < 722.90 us>
RND 4KiB (Q= 32, T= 1): 246.009 MB/s [ 60060.8 IOPS] < 532.16 us>
RND 4KiB (Q= 1, T= 1): 29.892 MB/s [ 7297.9 IOPS] < 136.71 us>

[Mix] Read 70%/Write 30%
SEQ 1MiB (Q= 8, T= 1): 5162.027 MB/s [ 4922.9 IOPS] < 1618.09 us>
SEQ 1MiB (Q= 1, T= 1): 1307.544 MB/s [ 1247.0 IOPS] < 801.07 us>
RND 4KiB (Q= 32, T= 1): 443.406 MB/s [ 108253.4 IOPS] < 294.72 us>
RND 4KiB (Q= 1, T= 1): 36.896 MB/s [ 9007.8 IOPS] < 110.73 us>

TRACKER · Apr 16, 2025

Guys, beware!
In Truenas Scale 25.04 iSER feature is disabled by purpose because it is "Enterprise" feature.
So in case you upgrade from 24.10 to 25.04 and you are not "Enterprise" you will be scr**ed.

WhiteNoise · Apr 17, 2025

I don't think they are blocking you from manually setting up iSER like the person on level1tech did.

TRACKER · Apr 17, 2025

WhiteNoise said:
I don't think they are blocking you from manually setting up iSER like the person on level1tech did.

I've read the article but i did not get how to do it, can you share some details please?

p.s. i've found it

Search

TrueNas core vs TrueNas Scale iSCSI performance

nev_neo

Active Member

nasbdh9

Active Member

TRACKER

Active Member

nev_neo

Active Member

nev_neo

Active Member

TRACKER

Active Member

nev_neo

Active Member

gea

Well-Known Member

TRACKER

Active Member

100 Gbe & TrueNAS Scale 23.10 iSCSI - Performance Unleashed

Configuring iSCSI and iSER Adapters and Storage

nev_neo

Active Member

nev_neo

Active Member

TRACKER

Active Member

WhiteNoise

Member

TRACKER

Active Member