Slow rsync performance moving files from one zpool to another with SSH (unraid)

[custom90gt]

I will preface this by saying I am really new to all of the unraid/zfs fun but am learning and hope this is a reasonable spot for this post. I've been testing unraid 6.12 RC on my backup server with a zpool consisting of 8x10TB SAS drives and I've been happy with the performance. I decided last night to take the leap on my main server and re-create the array with a similar 8x10TB drives in a raidz2 zpool. Now I'm transferring everything back over with rsync over ssh (maybe that's my main problem) and the speeds are not great. I get around 200MB/s with large files. Maybe I'm expecting too much but that is much slower than I expected. IPERF3 is 9.41Gb/s and I was really thinking I'd be near maxing that out. Any thoughts/tips? I haven't tried zfs send yet, but maybe that's a better option.

Specs:
main server: Epyc 7F52, 256gb of ecc 2133mhz ram, SM H12SSL-I motherboard, 8x10TB drives in a raidz2 zpool
backup server: Xeon 2640v4 cpus, 256gb of ecc 2133mhz ram, 8x10TB drives in a raidz2 zpool
Network: 10Gb/s setup with mellanox ConnectX-3's connected to a brocade 6450

 

[unwind-protect]

You might be limited by ssh. 200 MB/sec sounds about right as the limit.

Abusing SSH for ZFS and Profit :: FreeBSD Presentations and Papers

papers.freebsd.org

With ZFS send/receive you can use netcat or the like.

 

[piranha32]

I will preface this by saying I am really new to all of the unraid/zfs fun but am learning and hope this is a reasonable spot for this post. I've been testing unraid 6.12 RC on my backup server with a zpool consisting of 8x10TB SAS drives and I've been happy with the performance. I decided last night to take the leap on my main server and re-create the array with a similar 8x10TB drives in a raidz2 zpool. Now I'm transferring everything back over with rsync over ssh (maybe that's my main problem) and the speeds are not great. I get around 200MB/s with large files. Maybe I'm expecting too much but that is much slower than I expected. IPERF3 is 9.41Gb/s and I was really thinking I'd be near maxing that out. Any thoughts/tips? I haven't tried zfs send yet, but maybe that's a better option.

Check what performance you can achieve when copying data between memory on both hosts:

cat /dev/zero | ssh user@remote.host "dd of=/dev/null status=progress"

This will help you eliminate (or confirm) ssh connection as the bottleneck.

 

[CyklonDX]

May I inquire about how you copy it over? Is it through NFS mount?
For rsync i would recommend running without delta transfer algorithm (by running it with '--whole-file' option)

With ssh/scp transfers you can enable compression, that should give you bit more throughput - but bit more cpu usage as well.

 

[custom90gt]

May I inquire about how you copy it over? Is it through NFS mount?
For rsync i would recommend running without delta transfer algorithm (by running it with '--whole-file' option)

With ssh/scp transfers you can enable compression, that should give you bit more throughput - but bit more cpu usage as well.

I will be honest and say that I don't actually know the answer to your first question. The share was created with unraid which makes a ZFS data set. I am just attempting to utilize the terminal with rsync utilizing their SSH connection flag. I will try the flag you mentioned. Are there ways to enable compression with the ssh option in rsync? Sorry I am not knowledgeable in this area, it's all new to me.

 

[CyklonDX]

rsync will have --compress-level= or -z for compression (you can use different types of compression, and will look different in terms of cpu performance)

 

[custom90gt]

Check what performance you can achieve when copying data between memory on both hosts:

cat /dev/zero | ssh user@remote.host "dd of=/dev/null status=progress"

This will help you eliminate (or confirm) ssh connection as the bottleneck.

Running that command I get around 300MB/s. Faster than rsync but still slower than I was anticipating.

 

[piranha32]

Running that command I get around 300MB/s. Faster than rsync but still slower than I was anticipating.

Try to send with compression (ssh -C ...). It should help a bit if the bottleneck is in transfers, but if memory/cpu are the limiting factors, compression will not help much, and even can slow transfers down.
Watch ssh CPU usage on both ends. If it pegs at 100% on either machine, you won't speed up transfers without going parallel, or using a different channel than ssh.

 

[custom90gt]

Well sadly enabling compression or --whole-file doesn't seem to help with speed. I need to mess around with ZFS send and see. It would be nice to go parallel but I don't have a clue there. I am totally open to something other than SSH. I believe the syncthing docker will do parallel transfers.

 

[piranha32]

Well sadly enabling compression or --whole-file doesn't seem to help with speed. I need to mess around with ZFS send and see. It would be nice to go parallel but I don't have a clue there. I am totally open to something other than SSH. I believe the syncthing docker will do parallel transfers.

With the /dev/zero to /dev/null transfers you have the best case scenario for compression (stream of identical values).
It can be helpful to test memory-memory transfer speeds on each of the hosts:

dd if=/dev/zero of=/dev/null  status=progress

This is the absolute best-case scenario, copying bytes directly between two memory-based file descriptors. Here, the only limiting factors are the kernel and dd. Even if dd has a piss-poor data copying implementation, I wouldn't expect much better performance from file transfers over ssh.

 

[custom90gt]

With the /dev/zero to /dev/null transfers you have the best case scenario for compression (stream of identical values).
It can be helpful to test memory-memory transfer speeds on each of the hosts:

dd if=/dev/zero of=/dev/null  status=progress

This is the absolute best-case scenario, copying bytes directly between two memory-based file descriptors. Here, the only limiting factors are the kernel and dd. Even if dd has a piss-poor data copying implementation, I wouldn't expect much better performance from file transfers over ssh.

I appreciate it, that makes sense. I will start seeing about utilizing a different protocol than ssh.

 

[piranha32]

I appreciate it, that makes sense. I will start seeing about utilizing a different protocol than ssh.

dd with default settings can actually slow down transfers quite a lot. By increasing block size I managed to more than double the transfer rate over ssh from localhost to localhost.
Add "bs=block_size" to dd and rerun the experiments. The optimal block size will depend on the computer. In my case it was "bs=600000", but for your systems it can be different.
If speeds you can achieve during experiments with ssh and dd significantly exceed what you get from rsync, the next step will be measuring transfer rates for single big files. To test read speeds replace /dev/zero with a big enough file, and to test write speed, use a non-existing file name on the file system you want to test instead of /dev/null (dd bs=... of=/path/to/your/file).
Adding compression on a fast link will hurt the transfer performance.

 

[CyklonDX]

You can try BBCP as i recall people were praising it quite a bit few years back (by pushing ~90% of nic util through it.)

Since you didn't see much performance uplift so far, I would be more keen that something else is slowing you down; Not a transfer protocol.

 

[mrpasc]

Isn’t RAID Zx limited to the bandwidth of one drive for writing? If so the „receiving“ pool of the rsync operation might be the limiting factor?

 

[CyklonDX]

Isn’t RAID Zx limited to the bandwidth of one drive for writing? If so the „receiving“ pool of the rsync operation might be the limiting factor?

No.
First in proper setup it would allocate as much as it can into ARC cache (ram), and later onto L2ARC (additional cache / nvme/ssd). It would slowly try to commit to disks after that. After the buffers are filled (on both pools), it would slow down to around 70% of all disks write performance. (depending on your cpu frequency, and sas controller performance, and source/destination compression level/type) (raidz2 and raidz3 would be loosing a lot more performance - but again only after running out of cache.)

 

[mrpasc]

ARC is read cache, doesn’t help for write.
„Write Cache“ for ZFS is RAM too, but limited to some seconds / GB of RAM.
So even if the source pool data for the rsync is completely held in ARC / L2ARC because data was read before: write to the destination Pool is limited.
And OP wrote he is rsyncincin to a Raid Z2 pool.

 

[custom90gt]

ARC is read cache, doesn’t help for write.
„Write Cache“ for ZFS is RAM too, but limited to some seconds / GB of RAM.
So even if the source pool data for the rsync is completely held in ARC / L2ARC because data was read before: write to the destination Pool is limited.
And OP wrote he is rsyncincin to a Raid Z2 pool.

It does write to ram in I believe 5 second bursts. My drives go from 0MB/s total writes to 700+MB/s every few seconds. It's not limited by a single drive speed, but I am too new to all of this to really wrap my head around what is going on. I will also test some disk to array transfers using my NVME drives and see.

 

[CyklonDX]

Hat for correcting me there, but yes it writes to TXG - a ram cache, and then slowly commits things to disks every few seconds. Thus still it should fill memory first before landing onto hardware itself. How much ram do you have assigned to zfs?

 

[custom90gt]

Hat for correcting me there, but yes it writes to TXG - a ram cache, and then slowly commits things to disks every few seconds. Thus still it should fill memory first before landing onto hardware itself. How much ram do you have assigned to zfs?

200GB

 

[CyklonDX]

care to run (on destination box)
awk '/^size/ { print $1 " " $3 / 1048576 }' < /proc/spl/kstat/zfs/arcstats