This has been driving me nuts this week. My Terramaster F4-424 Max is 2x faster when reading from SMB share (Raid1 with NVMe cache) than reading from TrueNAS Raid 1 NVMe dataset.
My setup is
VM TrueNAS Scale 24.10.2 in Proxmox (q35, VirtIO SCSI single)
8 cores of Epyc 7402
96GB RAM to TrueNAS
2 x NVMe PM983 PCIe passthrough
1 x Mellanox ConnectX 4-Lx 25gbpe PCIe passthrough
I have raid 1 mirror from 2 PM983 exported as SMB Multichannel share

Samba 4.20 - super modern, deffo supports SMB multichannel and io_ring
Its perfectly capable to determine interface speed and RSS support, but I also tried explicit via cli
via
with subsequent smbd restart
also tried (but as I read, they are old commands, totally useless with modern Samba)
NVMe is fast, fio for sequential reads reports this :
Network - iperf3 - 10gbps easily
Super basic setup as you can see, everything is fast and supposed to be fast for clients.
But it is not, the max I get copying files from SMB export share is 470mb/s

Client - Windows 11 Pro Mellanox ConnectX 4-Lx
SMB multichannel - 100% enabled and used (I even wasted day looking at Wireshark negotiating packets)
Thats it. I tried numerous sysctl changes - but no point as iperf3 shows perfect 10gbps speed.
I connect same Windows 11 client to Terramaster F4-424 Max (running 1x10gbps RJ45 Marvell AQtion) , I get immediate 1.15gb/s
RAID1 + NVMe cache
Terramaster runs Samba 4.15 with same settings

WHY TRUENAS IS SLOW ??
I did try 22 version and even nightly 25. Same speed.
Is this side-effect of running under Proxmox VM ? Why do I get perfect 10gbps saturation and 6GB/s NVMe speed inside the VM using specialized tools then ?
My setup is
VM TrueNAS Scale 24.10.2 in Proxmox (q35, VirtIO SCSI single)
8 cores of Epyc 7402
96GB RAM to TrueNAS
2 x NVMe PM983 PCIe passthrough
1 x Mellanox ConnectX 4-Lx 25gbpe PCIe passthrough
I have raid 1 mirror from 2 PM983 exported as SMB Multichannel share

Samba 4.20 - super modern, deffo supports SMB multichannel and io_ring
Its perfectly capable to determine interface speed and RSS support, but I also tried explicit via cli
Code:
interfaces = "x.x.x.x;capabilities=RSS,speed=25....lots of zeros"
Code:
service smb update smb_options=
also tried (but as I read, they are old commands, totally useless with modern Samba)
Code:
aio read size = 1 or 16 * 1024
use sendfile = yes
NVMe is fast, fio for sequential reads reports this :
Code:
$ fio --name=fio_test --ioengine=libaio --iodepth=16 --direct=1 --thread --rw=write --size=100M --bs=4M --numjobs=1 --time_based --runtime=60
Run status group 0 (all jobs):
WRITE: bw=6334MiB/s (6642MB/s), 6334MiB/s-6334MiB/s (6642MB/s-6642MB/s), io=371GiB (399GB), run=60001-60001msec
Code:
Accepted connection from 192.168.1.16, port 60898
[ 5] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60899
[ 8] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60900
[ 10] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60901
[ 12] local 192.168.1.40 port 5201 connected to 192.168.1.16 port 60902
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 299 MBytes 2.51 Gbits/sec
[ 8] 0.00-1.00 sec 299 MBytes 2.51 Gbits/sec
[ 10] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec
[ 12] 0.00-1.00 sec 292 MBytes 2.45 Gbits/sec
[SUM] 0.00-1.00 sec 1.15 GBytes 9.91 Gbits/sec
But it is not, the max I get copying files from SMB export share is 470mb/s

Client - Windows 11 Pro Mellanox ConnectX 4-Lx
SMB multichannel - 100% enabled and used (I even wasted day looking at Wireshark negotiating packets)
Code:
PS C:\Windows\System32> Get-SmbConnection
ServerName ShareName UserName Credential Dialect NumOpens
---------- --------- -------- ---------- ------- --------
192.168.1.40 video_fast KORESH\admin KORESH\otec 3.1.1 2
PS C:\Windows\System32> Get-SmbMultichannelConnection
Server Name Selected Client IP Server IP Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- --------- --------- ---------------------- ---------------------- ------------------ -------------------
192.168.1.40 True 192.168.1.16 192.168.1.40 6 2 True False
I connect same Windows 11 client to Terramaster F4-424 Max (running 1x10gbps RJ45 Marvell AQtion) , I get immediate 1.15gb/s
RAID1 + NVMe cache
Terramaster runs Samba 4.15 with same settings

WHY TRUENAS IS SLOW ??
I did try 22 version and even nightly 25. Same speed.
Is this side-effect of running under Proxmox VM ? Why do I get perfect 10gbps saturation and 6GB/s NVMe speed inside the VM using specialized tools then ?
Last edited: