The work load consists of generating many files as fast as possible on very fast storage, (RAID 0 of nvme drives) and transferring that data to slow storage for long term storage while minimizing the bottleneck. I'm trying to minimize the time limit for the transfers to long term storage. When an HDD is full it gets pulled and replaced. If possible I'd like to maintain 16 TB writes per day on the long term storage.
Would ZFS be the ideal filesystem for this use case? Should the short term fast storage be a part of the ZFS filesystem? If the fast storage should be it's own file system, would I need other fast storage dedicated to the ZFS array to maximize write speed to the HDDs? Can a ZPool be rebuilt on another machine dedicated to housing the HDDs long term?
I'll be pulling 16 TB HDDs fairly often depending on the transfer rate. Is there any means of data redundancy for this use case?
Would ZFS be the ideal filesystem for this use case? Should the short term fast storage be a part of the ZFS filesystem? If the fast storage should be it's own file system, would I need other fast storage dedicated to the ZFS array to maximize write speed to the HDDs? Can a ZPool be rebuilt on another machine dedicated to housing the HDDs long term?
I'll be pulling 16 TB HDDs fairly often depending on the transfer rate. Is there any means of data redundancy for this use case?