How to acheive SMOKIN' ZFS send/recv transfers

Discussion in 'FreeBSD and FreeNAS' started by whitey, Mar 27, 2017.

  1. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,745
    Likes Received:
    851
    Evening all, thought I'd decompress here and show some work I have been doing this evening to migrate my Vm pool of data (600GB roughly) from FreeNAS 9.10 to FreeNAS Corral 'the old fashioned way'. I am not quite ready to upgrade my FreeNAS 9.10 to Corral but have a Corral AIO that I wanted to protect my VM pool data over to. Thought I'd setup a FreeNAS peer but that only seems to work Corral to Corral...even tried a ssh host peer, no such luck. Didn't want to investigate since it's a 'one time backup/protection scheme' so this was what I resorted to. To maintain peak performance ensure your pool of storage is up to the task (sas3 HGST ssd's used in this case), your network is 10G, and FreeNAS hosts are connected 10G (vmxnet3 in my case between my AIO's), jumbo end-to-end.

    Could have used this 'one liner' type of syntax but I know ssh has an overhead for sure, never gonna reach maximum throughput via that method unless you take hpn-ssh route maybe.

    zfs send husmm-r0/nfs@auto-20170327.1913-2w | ssh 192.168.x.x zfs recv sas3/nfs-dr

    Ended up settling in on good ole' netcat (DO NOT use this on an unsecure network, ONLY use on trusted network)

    SRC - (run first) [root@freenas-esxi6a] ~# zfs send husmm-r0/nfs@auto-20170327.1913-2w | nc -l 3333

    DEST - (run second) [root@freenas] ~# nc 192.168.x.x 3333 | zfs recv sas3/nfs-dr

    The results:

    SRC FreeNAS 9.10 system
    upload_2017-3-27_20-5-14.png

    DEST FreeNAS Corral system
    upload_2017-3-27_20-5-43.png

    Hope this helps someone out there that may want to move data from FreeNAS 9.x to Corral. Tried and true method between any ZFS distro really.
     
    #1
    Last edited: Mar 31, 2017
    gigatexal, Patrick, K D and 1 other person like this.
  2. groove

    groove Member

    Joined:
    Sep 21, 2011
    Messages:
    56
    Likes Received:
    6
    #2
  3. Terry Kennedy

    Terry Kennedy Well-Known Member

    Joined:
    Jun 25, 2015
    Messages:
    1,000
    Likes Received:
    462
    I use zrep + bbcp (watch out for the newer versions of the bbcp port - some patches to add IPv6 support broke things badly).

    17TB in around 7.5 hours:

    [​IMG]

    Lots more info here, or start at the top. Note that this is bare FreeBSD CLI, not FreeNAS.
     
    #3
    K D and T_Minus like this.
  4. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    3,428
    Likes Received:
    487
    Great info here :) Thanks
     
    #4
  5. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    I was getting only ~550GB per hour transfers using rsync and zfs send/recv.

    Used this method to copy a 4.2TB data set between 2 systems connected via 10GBe. Took around 6 hours. Both are stock freenas AIOs. I ended up with a data set in the destination system that was unusable. In the storage view of FreeNAS, it threw an error message when trying to edit the data set. When I opened up the parent dataset's share from windows, it had a 0 kb file with the dataset name.

    Not sure where I messed up.

    Running this right now and I'm seeing ~1TB per hour copied.
     
    #5
  6. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    download.png

    This is my transfer speed.

    Source : 2vcpu, 16 GB RAM, 2x(4TBx4 RaidZ1), Intel S3700 400GB SLOG
    Dest : 2vcpu, 8 GB RAM, 8TBx8 RaidZ2

    Both are FreeNAS AIOs running on esxi, hosts are connected via Mellanox CX3 using a Ubiquiti US-16-XG switch.

    I'll eventually double the allotted RAM on both the source and dest VMs. What other parameters should I be looking at to increase the transfer speed?
     
    #6
  7. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,745
    Likes Received:
    851
    What type of disks are you using for capacity? I see you have a s3710 for ZIL, if you are using spinners this 'may' be all she's got Scotty :-D

    Assuming using my method you did end up w/ a usable ZFS dataset once data was sent/received whereas you said using mbuffer gave you issues.
     
    #7
  8. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    Yup. You are right on both counts. 7200 RPM HGSTs in the Source and 8TB reds in the dest systems. Your method did work for me while mbuffer gave issues. The first transfer of 4.2 TB finished succesfully in about 3:50 hrs and a second 15.4 TB just finished taking about 17hrs using your method. Everything working as expected.

    Have another total ~200 TB of transfers as I shuffle data around and was wondering if there was some way to speed things, even if it means changing something temporarily.
     
    #8
  9. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,745
    Likes Received:
    851
    Not that I know of, that's a whole lotta data to be playing shuffle game w/ :-D.

    40GbE and all-flash pools maybe hah $$$

    EDIT: 200TB, good hell let's hear abt that/those system/s.
     
    #9
  10. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    LOL. The total data set is around 45 TB (Media, files, backups etc). In process of moving away from Hardware raid for storage to ZFS. So moving data to another box, upgrade hardware and drives and moving it back.

    It doesnt help that the moment I think that one box is done, I think of some thing else to change and I have to repeat the whole darn thing again.
     
    #10
  11. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,745
    Likes Received:
    851
    No incrementals or staging/changed-data directory/folder/dataset? That's how I've managed the insanity in the past....either that or incremental snapshot send/recv but coming from HW raid (EWWWW...no eff that double EWWWWWWWWWW) YMMV :-D

    GL w/ data move, now repeat after me..."All your data are belong to ZFS" HAH!
     
    #11
    Last edited: Jun 11, 2017
  12. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    Unfortunately in all my wisdom (or lack of it) I decided to upgrade all my servers and disks at the same time (Both home and lab) as well as upgrade to 10gbe. So right now everything is a disorganized mess.
     
    #12
  13. Terry Kennedy

    Terry Kennedy Well-Known Member

    Joined:
    Jun 25, 2015
    Messages:
    1,000
    Likes Received:
    462
    If the eventual target is traditional hard drives which have no data (other than the initialized filesystem) on them and you are being limited by disk throughput, you should see a gradual slowdown in transfer performance as the disks fill up, due to the inner sectors being slower than the outer ones. For example (copied from my earlier post up-thread):

    [​IMG]

    At the size of transfers we're talking about here, your ZIL / SLOG device is going to fill up and then start flushing to the hard drives - at the most, it will compensate for bursty traffic and feed the disks at their maximum sustained I/O rate. At worst, it will hurt performance as it runs out of erased sectors and has to start erasing in order to store data. You may benefit from temporarily removing the log device and possibly setting sync=disabled on the pool, assuming the pool is empty when you start. Don't forget to re-add the log device and reset sync to default (normally with "zfs inherit", since setting it to default will still be counted as a local parameter change).

    The first thing you should do is see how fast your systems can actually talk to each other. In my case, since I use bbcp:
    Code:
    (0:1) srchost:~terry# bbcp -P 2 -s 8 /dev/zero desthost:/dev/null
    bbcp: Creating /dev/null/zero
    bbcp: 160620 06:06:45  0% done; 1.2 GB/s
    bbcp: 160620 06:06:47  0% done; 1.2 GB/s
    bbcp: 160620 06:06:49  0% done; 1.2 GB/s
    bbcp: 160620 06:06:51  0% done; 1.2 GB/s
    bbcp: 160620 06:06:53  0% done; 1.2 GB/s
    ^C
    If you aren't getting wire-speed there, you certainly won't get it once you involve disk I/O on both ends. Look at my earlier post for links where I go into this in a lot more detail.
     
    #13
    Stux and K D like this.
  14. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    Thanks @whitey and @Terry Kennedy.

    As a quick and dirty solution, I just connected the new disks to the same server, created the pool and locally used ZFS send/recieve. Copied 15TB in about 8 hrs. I think I'll just do this and transplant the disks to the target machines and import the pools there.
     
    #14
    Stux likes this.
  15. CJRoss

    CJRoss Member

    Joined:
    May 31, 2017
    Messages:
    61
    Likes Received:
    4
    #15
  16. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,745
    Likes Received:
    851
    Umm yes...lol unless living under a rock :-D

    Edit: Point is this method works on all ZFS systems I have ever tested.
     
    #16
    Last edited: Jun 12, 2017
  17. CJRoss

    CJRoss Member

    Joined:
    May 31, 2017
    Messages:
    61
    Likes Received:
    4
    Surprisingly, a lot of people missed the announcement. I was just making sure since you mentioned it in your post.
     
    #17
  18. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,736
    Likes Received:
    1,440
    @CJRoss check date in original post ;) prior to announcement.
     
    #18
  19. K D

    K D Well-Known Member

    Joined:
    Dec 24, 2016
    Messages:
    1,404
    Likes Received:
    297
    resurrecting an old thread

    Trying to migrate some data and encountering very slow speeds in zfs send recv.
    download.png

    iperf shows almost wire speed. Using an intermediate windows box to just "copy and paste" gives around 260MBps speed. Any thoughts?


    iperf.png
     
    #19
  20. Terry Kennedy

    Terry Kennedy Well-Known Member

    Joined:
    Jun 25, 2015
    Messages:
    1,000
    Likes Received:
    462
    FreeBSD / FreeNAS or some other ZFS implementation?
    How are you getting the data from A to B? SSH, something else? What does CPU usage look like on both boxes?Any chance you have synchronous writes only set on the destination pool?

    Traffic from ZFS send/recv tends to be bursty, so a transfer method with lots of buffering is good. I use bbcp w/ zrep.

    Remember, performance is limited by the sustained random write IOPS of your physical drives - even if you have a fast ZIL, it is going to fill up and you'll be limited by the speed it can flush to disk.
     
    #20

Share This Page