Slow rsync performance moving files from one zpool to another with SSH (unraid)
- Thread starter custom90gt
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
You can also specify your ssh cipher, enable sparse, and disable compression. Different ciphers will have different CPU overhead and can get you better transfer rates.
rsync -vr --sparse -e "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" --progress * root@192.168.0.206:/data/
rsync -vr --sparse -e "ssh -T -c aes256-gcm@openssh.com -o Compression=no -x" --progress * root@192.168.0.206:/data/
This is how much ram is assigned to your zfs pool
It prints col1 that is "size" and col3 divided by 1048576 to show you in MB
For best performance I would recommend setting up to 70% of your ram as limit.
ZFS on Linux - Proxmox VE
How much ram does your system have? (also how much does it have while its transferring the files?)
Ahh I should have deduced that. I have 256GB in each system so I was going to let Plex and the rest of the dockers I have running have access to ~56GB of ram. I may reduce it a little bit though.This is how much ram is assigned to your zfs pool
It prints col1 that is "size" and col3 divided by 1048576 to show you in MB
For best performance I would recommend setting up to 70% of your ram as limit.
(tho i recommend half the ram on the system.)ZFS on Linux - Proxmox VE
pve.proxmox.com
How much ram does your system have? (also how much does it have while its transferring the files?)
I finally got a second to try zfs send and I get around 300MB/s which is certainly better but not amazing by any stretch of the imagination. I may try netcat instead, but I have to read up on that, lol.
I still think you are expecting to much I/O from your RAID-Z2 pool which receives the data.
Rsync is heavy on I/O and metadata writes. So you can’t expect much more I/O then 1 of your disk can offer.
See here: Six Metrics for Measuring ZFS Pool Performance Part 2
You can try to get better rsync performance by adding a special vDev for metadata (and very small files if you want to). But be cautious: the metadata vDev needs at least same redundancy than your pool. Loose the special vDev means loos the whole pool!!!
Rsync is heavy on I/O and metadata writes. So you can’t expect much more I/O then 1 of your disk can offer.
See here: Six Metrics for Measuring ZFS Pool Performance Part 2
You can try to get better rsync performance by adding a special vDev for metadata (and very small files if you want to). But be cautious: the metadata vDev needs at least same redundancy than your pool. Loose the special vDev means loos the whole pool!!!
I'd actually been looking into special metadata via Wendell's video on youtube, but that's because my steam folder has millions of small files (sadly more than my small 118GB optane drives can hold). It looks like zfs send is a better way to go, perhaps with less I/O overhead, but may be the limit for SSH. I'll take a look at that article when I have a free few minutes, I appreciate it.
I do appreciate everyone's help so far, it's a fun journey.
Yeah, this really sounds like your ssh throughput limit, and you measured that at 300 MB/sec on the previous page. A different cipher might bring you 10 or 20% more, so not great. You should also be able to observe 100% CPU for the ssh process on one of the machines.
If you want quick and dirty you could try rsh
But that authorizes a lot, so netcat would be better.Yeah makes sense to me, google searches show that people have been having issues getting more than 2-300MB/s with SSH. *on edit* it does seem to peg a single core of the Epyc 7F52 to 90-95% fairly consistently. I was hoping the "frequency" based epyc would be able to keep up.Yeah, this really sounds like your ssh throughput limit, and you measured that at 300 MB/sec on the previous page. A different cipher might bring you 10 or 20% more, so not great. You should also be able to observe 100% CPU for the ssh process on one of the machines.
If you want quick and dirty you could try rsh
Netcat and BBCP are interesting to me, but neither are included in Unraid so I'll have to get the packages. Some research shows that the latest versions of bbcp have issues though.
Last edited:
Try rsh just once to verify that you get better speed without ssh's encryption. There is no guarantee that the rest of the system is much faster, even if ssh is the bottleneck right now.
Do you have cstream?
It used to be that you could turn all ciphers off for ssh, but not anymore I guess. Found some tips here if you want to try maybe faster ciphers here. They aren't enabled by default, though:
How can I disable encryption on openssh?
I'm having performance problems using openssh (server) and putty (client) combination to use a remote webproxy. I'd like to disable encryption and test the results to see if it makes a difference. ...
Did you scan that paper I linked back in my first reply?
I do not have cstream either sadly.Do you have cstream?
It used to be that you could turn all ciphers off for ssh, but not anymore I guess. Found some tips here if you want to try maybe faster ciphers here. They aren't enabled by default, though:
How can I disable encryption on openssh?
I'm having performance problems using openssh (server) and putty (client) combination to use a remote webproxy. I'd like to disable encryption and test the results to see if it makes a difference. ...serverfault.com
Did you scan that paper I linked back in my first reply?
I will check out the ciphers for sure. That article is super interesting btw. Rather big differences based on cipher and cpu speed. I guess theoretically the Epyc should be able to saturate 10Gb/s...
so doing -c aes256-gcm@openssh.com as @acquacow suggested I now get a max of 550MB/s!
That's actually a much bigger jump than I anticipated
That's actually a much bigger jump than I anticipated
I don't even think that's the fastest you can go, it's just one of the better performing ones I use at work. If you have full control of client and server, you can allow some weaker ciphers on the server and client side if you are fine with it.
Does anybody remember whether ssh (still?) has a mode where only auth is encrypted but the stream afterwards is not? I seem to remember that being available. Not pretty but beats netcat.
Probably depends on what ciphers were enabled at compile time. You can check what is abailable by running
Code:
sshd -T | grep ciphersI am totally fine with weaker ciphers, but I'm also pretty happy with 500MB/s too lol
list of ciphers:
chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
Here is another set of ssh cipher benchmarks:
blog.famzah.net
OpenSSH ciphers performance benchmark
:idea: Please review the newer tests. Ever wondered how to save some CPU cycles on a very busy or slow x86 system when it comes to SSH/SCP transfers? Here is how we performed the benchmarks, in ord…
blog.famzah.net