This is going to be convoluted.
The short story is that I have a physical FreeNAS server with about 13 TB of data on it (5GB to 15GB chunks). I want to backup that data to a Ubuntu server on a ESXi 6.5 VM on a nightly basis. When I do this with rsync, I see about 200 Mb/s (yes, Mb, NOT MB). When I use CP over a mounted NFS or CIFS share on the same directory and data, I'll see 3-4 Gb/s. HUGE difference. I have a windows box setup that I can duplicate this issue one too. I AM using both the -z and the -W flags in rsync. -W sped it up a little, but still nowhere close to NFS/CIFS
Now, the details:
I'm treating it like a local sync.
Actual line in my scrip: /usr/bin/rsync -rltDzvW --delete --stats /mnt/freenas/backup/freenas/ /mnt/x299prime/Backups/FreeNAS/ --log-file=$strLogFile
/mnt/x299prime/Backups/FreeNAS/ is mounted with NFS in fstab.
Help?
The short story is that I have a physical FreeNAS server with about 13 TB of data on it (5GB to 15GB chunks). I want to backup that data to a Ubuntu server on a ESXi 6.5 VM on a nightly basis. When I do this with rsync, I see about 200 Mb/s (yes, Mb, NOT MB). When I use CP over a mounted NFS or CIFS share on the same directory and data, I'll see 3-4 Gb/s. HUGE difference. I have a windows box setup that I can duplicate this issue one too. I AM using both the -z and the -W flags in rsync. -W sped it up a little, but still nowhere close to NFS/CIFS
Now, the details:
- Network
- All the effected machines use ConnectX-3 cards tied to a TP-Link T1700G-28TQ with DAC's under 3 meters.
- iperf3 shows ~9.8 Gb/s between all machines, both ways, with a single thread.
- Jumbo Frames, 9000 MTU on all machines, the ESXi host, and the switch.
- Machines
- FreeNAS is an Intel i7-3770 @3.4GHz, 32GB DDR3, HP 9207 HBA. 8 Each Western Digital WD4000FDYZ 4TB 64MB 7200RPM SATA 6.0Gb/s drives. Single ZFS pool.
- ESXi is an Two Intel Xeon E5-2670, 256 GB DDR3, 3 8TB WD Reds on a Adaptec 71605Q SAS controller in a RAID 5 (I know, I know). VM has 8 vCPU, 64GB of RAM, a 64GB primary drive and a dedicated storage array for the backup data (EXT4).
- Windows 10 is a i7-7800x with 64GB DDR4, a pair of EVO 960 512 (RAID 0), and a Lenovo 46C9110 IBM ServeRAID M5210
- What I need: a method to copy updated and new data from the FreeNAS to the backup pool on the ubuntu VM that takes advantage of the network speed and hardware. I'd like to think I can maintain 2-3 Gb/s. I need it to remove files on the VM that have been removed on the FreeNAS - this is the hangup that keeps CP from being a valid solution. I just need to embed this in my existing bash script to run on a nightly basis. If the method returned a exit code that I could notate that would be a positive.
I'm treating it like a local sync.
Actual line in my scrip: /usr/bin/rsync -rltDzvW --delete --stats /mnt/freenas/backup/freenas/ /mnt/x299prime/Backups/FreeNAS/ --log-file=$strLogFile
/mnt/x299prime/Backups/FreeNAS/ is mounted with NFS in fstab.
Help?
Last edited: