Samba & FreeBSD 13.2 strange read behavior

[RonanR]

I'm actually trying to set up a high-speed samba share over a 40Gb direct connection between a Windows 10 system and a FreeBSD 13.2 server.

Regarding my network setup, I'm using two Chelsio 40Gb T580-LP-CR. Doing an iperf test between these two systems give me 35Gb/s with single thread and 40GB/s with 2 parallel threads or more.

About my server, it's a 36 SAS HDD system, with an Epyc 7282 and 64Gb of DDR4. Internal tests using iozone or fio for 1 sequential 150GB file give me 3200GB/s write and 4200GB/s read. With 5 sequential files of 150GB, I got 3000GB/s write and 3900GB/s read.

dd test results on a 100GB file gives 3,2GB/s write and 3,8GB/s read.

Compression is off on my pool, sync is disabled and I'm using 1M recordsize.

I'm using Samba version 4.16.11

When sharing my drive, I got a 2800MB/s write performance, even with a 500GB write.
The problem is with the read performances. If the data are in the ARC, I also got 2800MB/s. But as soon as my data are read from the disk, it drops to 1000MB/s.
I tried to disable ARC (primarycache set to none). Using fio or dd, I got the same performances (around 3000MB/s write and 4000MB/s read). On a samba share I still got 1000MB/s
I have no idea why when data is in ARC, read on smb share is fast, and when data isn't it's almost 3 times slower, while internally it doesn't change a thing whether data is in ARC or not.

Does anyone have an idea of what can cause this ? I'm quite stuck as I don't know where i should look.

here is the content of my smb4.conf file

[global]
        server string = FreeBSD Server - Samba %v
        workgroup = TEST
        log file = /var/log/samba4/smb.log
        max log size = 50
        load printers = No
        disable netbios = Yes
        map to guest = Bad User
        security = USER
        server role = standalone server
        create mask = 0775
        directory mask = 0775
        invalid users = nobody root
        read raw=yes
        write raw=yes
        strict sync = no
        sync always = no
        server multi channel support = no

[TestShare]
        comment = Files Samba Network Share
        path = /pool1/TestShare
        valid users = testuser
        writable  = yes
        browsable = yes
        inherit acls = Yes
        read only = No
        vfs objects = zfsacl, streams_xattr, acl_xattr
        nfs4: acedup = merge
        nfs4: mode = special
        nfs4:chown = Yes

Many thanks in advance for your insights.

Ronan

 

[oneplane]

I'd separate the problems; run a Samba share on a Ramdisk to verify SMB performance, and in parallel try a different protocol to see if that performs differently (i.e. NFS, DD over TCP via netcat or even FTP).

 

[RonanR]

I just tried with a 20GB ram disk, shared it, and got 2800MB/s in read and write. So, as far as I can tell, it doesn't seems to be a SMB performance problem.

 

[RonanR]

In order to advance on this strange behavior, I experimented a lot, here is what I can tell you:
On the exact same system (so no hardware changes at all, no new pool creation), using the same test method with files bigger than ARC :

With FreeBSD 14 RC1, I got better internal performances when tested with fio and iozone (3500MB/s write and 4500MB/s read), but same problem with samba share (read stuck around 1000MB/s).

With OmniOS 151046, I got less performances internally (around 3000MB/s write and 3200MB/s read), but a better smb read. Indeed, I got 1800MB/s in read and write on the smb share in all cases, with or without ARC.

With Linux Debian, tested with latest TrueNas Scale, internal performances are ok (3000MB/s write and 4000MB/s read) and samba share performance without ARC is better than all other systems: I got 2200MB/s in read and write. Read goes up to 2600MB/s when data are in ARC, if not, it's stable at 2200MB/s.

I tested this with very big files copied (between 50 and 300GB per file), and using AJA system test with file size bigger than my actual ARC size, which is around 50GB.
Of course, these tests have been run several times with reboot and power down, to be sure it wasn't a one time only (un)lucky test session.

In order to get a better grasp of what's happening, I put together 4 NVMe drives (Samsung 990Pro 1TB) configured in a Z1 pool on my system, using FreeBSD 14 RC1. I disabled ARC cache (primarycache=none) and just declared it on my smb4.conf file, with the exact same parameters as my HDD share. I got a stable and constant 3000MB/s in read and write on this share.

TLDR: there are some read problems with on FreeBSD, at least on a samba share (didn't have time to test other share protocols), as soon as your data isn't in ARC, and when using a HDD pool. Same HDD pool doesn't have this read problem with Linux samba share (same samba version) or in OmniOS smb share.
NVMe pool doesn't have this problem on FreeBSD.

If anyone has an idea of what can be the cause, thanks in advance for sharing your thoughts.