Strange problem: SMB read is capped to ~500MB/s

[gnattu]

My server setup:
omnios-r151042
X520-DA2 nic
Test pool is on Intel Optane 900P so Disk IO is not a problem

I've tried two differnt Windows clients, running both Windows 11 and Windows Server 2019, with 10G nic X557 and Mellanox CX3, and both of them can only read (copy from the server) at 500MB/s or around half of a 10Gbit link. If these two client read at the same time, they can archieve around 800MB/s aggregated speed.

The speed wrting to server is fine and I can hit solid 1GB/s.

iperf is showing 9.4Gbit/s both from and to the server (-R and no -R)

What could be the problem causing this "cap"?

 

[i386]

Test pool is on Intel Optane 900P so Disk IO is not a problem

You mean writes are fine
Or do you use l2arc on the optane?

 

[tinfoil3d]

I assume you verified that local read speeds for those files on that host is much higher than 500 or 800mb/s?

 

[gnattu]

You mean writes are fine
Or do you use l2arc on the optane?

No no L2arc at all, and I am on 128GB RAM

 

[gnattu]

I assume you verified that local read speeds for those files on that host is much higher than 500 or 800mb/s?

Yes, much higher At more than 2GB/s

 

[i386]

How are you testing?

(slow) Explorer copy?
Or something multithreaded like robocopy?

Edit:
More question(s)
I assume 500MByte/s for large io (movie, iso, etc. files)?

 

[gnattu]

How are you testing?

(slow) Explorer copy?
Or something multithreaded like robocopy?

Just explorer copy, but for a single video file I don't think it would cause any performance problem.

I copied a video file about 5GB from the server and to local nvme ssd, and the speed never go above 500MB(it does goes above from time to time, but not much and below 600), I even changed multiple SSDs to see if the ssd is the bottleneck, but no luck.

I've retired with iscsi and I'm getting better results and much closer to 1GB/s

 

[i386]

I don't think it would cause any performance problem

I rarely get over 700MByte/s with explorer copy, but can go over 1.3GByte/s with robocopy and 4+ threads...

Edit: try crystaldiskmark/diskspd with multiple threads to rule out any storage problems

 

[gnattu]

I rarely get over 700MByte/s with explorer copy, but can go over 1.3GByte/s with robocopy and 4+ threads...

Edit: try crystaldiskmark/diskspd with multiple threads to rule out any storage problems

Robocopy is not better in my case, but crystal diskmark is showing good results:



So what is the problem

 

[i386]

Robocopy is not better in my case, but crystal diskmark is showing good results:

View attachment 24044

Nice, you're maxing out the 10GBE link
[/QUOTE]

So what is the problem

you can rule out the underlying storage, it's fast enough

for explorer/copy: microsoft has documentation about tweaking it (don't have the link right now). usually I wouldn't change these settings...
for robocopy: try different parameters, for large files I use the "/j" paramter (unbuffered io)

 

[gnattu]

for explorer/copy: microsoft has documentation about tweaking it (don't have the link right now). usually I wouldn't change these settings...
for robocopy: try different parameters, for large files I use the "/j" paramter (unbuffered io)

robocopy /J is about the same speed as explorer.

No matter how fast I can get from the benchmarks, if the real work performance is not good than it means nothing.

I just tried a very indirect way: running another Linux server, mount the nfs from omnios, then share it via samba

And I'm getting faster speed than windows to omnios directly

This is really confusing, if omnios's smb server is slow, why it shows a great benchmark result?

 

[gea]

Most benchmarks use large sequential transfers while "realworld" is often many small files. Another point are benchmarks with filesizes < read/write cache. This is why you often use workload based benchmarks ex Aja for Videofiles or filebench ex fivestream read/write or wevserver load.

I have no explanation why SMB - OmniOS should be slower than SMB - SAMBA - NFS - OmniOS. Often the multithraded OmniOS SMB server is faster than singlethreated SAMBA - although not as fast as Solaris with native ZFS. There must be a setting in OmniOS involved. May be an item is SMB Multipath on Linux or some dependencies Windows - OmniOS.

What I have seen on OmniOS performance is a dependency to nic drivers on Windows when an update of my Intel 10G nic drivers had increased performance massively as well as some Windows settings like disabling int throttelling. Also OmniOS ip settings should be optimized (run napp-it default tuning). Jumbo frames can be helpful.

 

[gnattu]

Most benchmarks use large sequential transfers while "realworld" is often many small files. Another point are benchmarks with filesizes < read/write cache. This is why you often use workload based benchmarks ex Aja for Videofiles or filebench ex fivestream read/write or wevserver load.

I have no explanation why SMB - OmniOS should be slower than SMB - SAMBA - NFS - OmniOS. Often the multithraded OmniOS SMB server is faster than singlethreated SAMBA - although not as fast as Solaris with native ZFS. There must be a setting in OmniOS involved. May be an item is SMB Multipath on Linux or some dependencies Windows - OmniOS.

What I have seen on OmniOS performance is a dependency to nic drivers on Windows when an update of my Intel 10G nic drivers had increased performance massively as well as some Windows settings like disabling int throttelling. Also OmniOS ip settings should be optimized (run napp-it default tuning). Jumbo frames can be helpful.

I just tried macOS and got similar read speed, so it could not be Windows side problem but rather an OmniOS SMB problem

I have the default tuning from napp-it and using MTU 9000 on both side.

The SMB - SAMBA - NFS - OmniOS path is only faster in read, and much slower in write. The crystal diskmark also shows a much slower speed, but the copy speed is really close to it(~900 in benchmark, ~800 in a real-world copy).

The thing really confuses me is that, I'm copying a really large single file, so I should get what the benchmark shows me or at least something close, but I only get about half from it.