Need help with slow samba file copy speeds

[kiteboarder]

I have low windows-to-linux samba file copy speeds and I can't figure out why. I have 2 machines that I can dual boot windows and linux. Windows-to-Windows file copy has no problem maxing out gigabit ethernet at 110+ MB/s. But when I reboot one machine into linux, it can barely hit 60-65 MB/s file copy windows-to-linux. I'm in linux using dolphin just trying to copy/paste a multi GB file to give plenty of time to saturate the interface.

For the hardware, I have 2 Windows machines, server 2012 and win10, Intel NICs all around, SSD drives on both ends, i7 processors. Two other machines that are dual boot, similar specs. All four machines connected to the same switch. Using "top" while copying, I'm not seeing high CPU utilization. It's not the hardware... Same performance on both "client" machines when booted into linux.

I've tried kubuntu, manjaro, kde neon, centos, all updated to latest versions. I've tried the common suggestions of changing options in the global section of smb.conf. I'm still seeing read speed of about ~60 MB/s regardless of what I do. Watching network utilization on the source windows box, it stays about 600 Mbit/s. So I have no idea why the linux boxes can't get a windows source to serve it up any faster than that.

Thanks in advance for any suggestions. I've googled for the past two days and I'm completely stuck at this point. Would be more than happy to buy you a 12 pack if you can solve this one!

 

[TuxDude]

Now I'm just guessing, but it almost sounds like Samba isn't even involved here. If I understand this right, from a KDE desktop, you are opening Dolphin, entering a smb://.... address that points at a windows box, and then copying files around. If that is correct, then Samba is not involved, and no amount of changes to smb.conf will make any difference. When using Dolphin that way, you are using one of KDE's KIOs as the SMB client which connects to the windows server. KIOs are almost like FUSE modules in that they execute in user-space, except instead of the standard FUSE API's, they implement a bunch of KDE APIs and are only usable by KDE applications. But like FUSE, don't expect performance to be great.

If you want high performance SMB from linux, make sure you are connecting with the kernel-level CIFS module (which also doesn't require a samba server to be setup). You should have an entry in your /etc/fstab file to automatically mount the CIFS filesystem as needed, and in Dolphin you should be just using a local path which is where you mounted the server connection to.

 

[kiteboarder]

TuxDude, love your avatar. Looks like you're exactly who I needed.

Your guesses are correct, that's exactly how I am trying to copy/paste. Obviously I'm a beginner, so I thought by using smb:// in Dolphin, that meant I was using samba. I'll do some research tonight about how to mount using the fstab file. Will report back.

Will send you a PM if I get this working. Seriously, thanks.

 

[Rand__]

Additionally try using ftp or scp (if the machine is beefy enough) to establish its in fact smb and not network drivers...

 

[kiteboarder]

Now I'm just guessing, but it almost sounds like Samba isn't even involved here. If I understand this right, from a KDE desktop, you are opening Dolphin, entering a smb://.... address that points at a windows box, and then copying files around. If that is correct, then Samba is not involved, and no amount of changes to smb.conf will make any difference. When using Dolphin that way, you are using one of KDE's KIOs as the SMB client which connects to the windows server. KIOs are almost like FUSE modules in that they execute in user-space, except instead of the standard FUSE API's, they implement a bunch of KDE APIs and are only usable by KDE applications. But like FUSE, don't expect performance to be great.

If you want high performance SMB from linux, make sure you are connecting with the kernel-level CIFS module (which also doesn't require a samba server to be setup). You should have an entry in your /etc/fstab file to automatically mount the CIFS filesystem as needed, and in Dolphin you should be just using a local path which is where you mounted the server connection to.

Unbelievable... It works. I quoted your reply so that anyone in the future will know how important this is.

I used this website for the how-to:
TipsAndTricks/WindowsShares - CentOS Wiki

There are dozens of samba performance questions out there where your post is the solution. I don't understand how this isn't more widely publicized in FAQs / sticky / best practices. I literally googled for two days without ever finding any mention of this. Thanks again, great info.

Two follow up questions:
Copying from windows-to-linux does max out, but it hiccups in that it will drop down to 70 MB/s range and then ramps back up to full speed. Stays maxed for a few seconds then hiccups again. Strange.

Second issue - I only max out when copying windows-to-linux. When I try to copy linux-to-windows I'm still in the ~35 MB/s range (sustained, no variation). Any last magic suggestions? I have the same behavior on both of my linux systems when writing to two different windows boxes, so again, it's not the hardware.

I'm so close... Hoping you guys can give me the last secrets.

 

[TuxDude]

Well - as I said originally, you aren't really using Samba in such a config, so no amount of Googling Samba-related things will return anything useful.

As to your followup questions...

1. Sounds like the destination HDD is having problems keeping up with the transfer. What will happen is that the data will transfer over the network as fast as it can go, but the linux destination won't be writing it to disk yet - it will be just saved in RAM. Eventually, the linux kernel will decide there is too much dirty data in the cache, and will attempt to flush it to disk as fast as possible. The result is you have your file-transfer process and the kernel both trying to move a bunch of data around at the same time, and you see those little hiccups. To verify if that is the case, I usually like to use the 'iostat' tool to watch the load on HDDs while the transfer is going, run something along the lines of 'iostat -kx 1' - where the 'k' will format the output in KB (for columns measuring bytes at least), the 'x' gives you extended stats (more columns, with latency and such), and the '1' means to print updated stats every 1 second. Ignore the very first row of output (that is the average since the system booted), and every line after that is the average over the previous interval, which we made 1 second. If your HDD is going back and forth between writing as fast as it can go, then nothing at all, you are seeing the problem I described. You can fix it by tuning the linux virtual-memory subsystem, telling it to keep less dirty data in cache, to flush more often, etc. Here's a random google result discussing some of the things you can tweak, or feel free to do further searches on any of those parameters: Better Linux Disk Caching & Performance with vm.dirty_ratio

2. And you are copying to a mount point using the kernel CIFS module, not a SMB:// path in Dolphin still? Assuming so, nothing comes to mind - I've never had any issues with the CIFS module being unable to use 100% of the bandwidth available (whether the limit be gig-ethernet, or a slow drive on one end). You'll have to find out where the bottleneck is, there's always one somewhere otherwise things would complete instantly. The most likely thing is usually one of the drives (though now with SSDs gig-ethernet is more commonly becoming the slowest bit), so watch source and destination for high latency. It's doubtful that its CPU or memory limited, but then CPU usage is so easy to watch it doesn't hurt to check/eliminate it as a possibility. Maybe something not working right in the network layer somewhere... Could be lots of things.