Suggestions for fastest means (given HW used) to get my data off DeDuplicated FreeNAS array..?

[TrumanHW]

It took me a LONG time to figure out why my computer was so god awful slow!

I do have another (near identical) computer which I can install FreeNAS on and copy my data set to ...

DEDUPED MACHINE:
T320
8x 10TB 7200rpm UltraStar SAS drives
RAIDz2
48GB DDR3 ECC RAM
10GbE (SFP+)
About 33TB of data (haven't turned it on in a while)

MACHINE AVAILABLE FOR TEMP COPY:
T320
8x 6TB 7200rpm UltraStar SAS drives
-- should I use RAIDz1 if it's literally just for duration of copy..? --
48GB DDR3 ECC RAM
10GbE (SFP+)

I also have at least a few 3.78TB NVMe drives


I'm assuming I should use something like Rsync ..?
(too bad I can't easily just MOVE folders to organize all that I want in to one area and that which I don't elsewhere... as moving isn't fast -- and neither is deleting).

Thanks

 

[gea]

Use ZFS replication to copy filesystems between them. It is the fastest option as it is based on data streaming and not slow file compare like rsync and it keeps ZFS checksum protection intact.

I would use Z2 and enable lz4 compress. This gives at least 36 TB usable capacity

 

[ericloewe]

A bit over 35.4 TiB nominal capacity, really (thanks HDD manufacturers...). If it compresses reasonably well, it might fit.

That said, you don't need to move your data to a different machine. Just make new, non-deduped datasets and zfs send | zfs recv over to said new datasets. Sounds like you have plenty of space for that, and it should be more or less the same in terms of speed - unless your dedup table is larger than will fit in RAM. If that's the case, eating some IOPS with writes would slow things down even further, although I'm not sure it would make a practical difference, given how slow such a system would be.

In any case, once you're done copying, delete the old datasets to get rid of the Dedup Table. I think ZFS will get rid of it once all its referenced blocks are gone, but you will need to wait for the background freeing task to slowly reclaim the now-free space.

and neither is deleting

How slow is the deletion really? The actual hard part, marking the freed blocks as free, is done asynchronously, so free space won't be immediately available, but the data won't be there bogging things down, either. Still probably worth it to delete as much as possible first.

 

[TrumanHW]

Also, should I add a faster CPU..?
CPU utilization has high when reading any data that was written with dedupe enabled (like, from 2MB/s briefly and down to a few KB/s).

E5-2403 v2 Quad - Core @ 1.8GHz

The only ECC machines I have to this with is either:
• The above referenced T320 with a 1.8GHz processor, or...
• A 2010 MacPro5,1 with 12c 3.33GHz (With a pair of older Xeons model X5680 maybe..?)
(assuming rEFInd won't be overly difficult)

...OR, I could just upgrade the Dell (with the data already) to the fastest clock-speed CPU compatible:

So, either:

A. Upgrade the T320 (Single-Socket) Compatible CPU Family-Options:

• Xeon® E5-2400
• Xeon® E5-2400 v2
• Xeon® E5-1410
• Pentium® 1400

B. MacPro5,1 with a pair of 12c 3.33GHz (Older Xeons X5680) ..?

Suggestions..?

 

[TrumanHW]

Copying ZFS between host computers is best done via ZFS replication, as it's both:
• The fastest option, as it's based on data streaming vs slow file comparison ...
• Unlike rsync, it keeps the ZFS checksums protecting your data's integrity intact.

Ha, that's the first time wording a question badly redounded to my advantage! As in, I literally lucked out!
(I wouldn't have even thought to ask about the method of replication, thank you!)

My (temporary) target array's (config) is a non-issue atm.


ACTUAL ISSUE:
My first zVol had DEDUPE enabled ... & took me over 2 years to realize / learn was the issue.
EVEN DISABLED -- DATA WRITTEN WHILE DEDUPE WAS ENABLED REMAINS ALMOST AS SLOW TO THIS DAY.
Data written after disabling dedupe transfers (and generally behaves) 'hardware proportionate'..

Data WRITTEN while DEDUPE WAS ENABLED -- still reads horribly slow.
Data READ while DEDUPE is DISABLED ... that was WRITTEN WITH DEDUPE ENABLED is STILL SUPER SLOW: ~1-2MB/s


To my understanding, the bottleneck is the CPU, which YES, was freaking TORTUROUS! As it took OVER 2 YEARS of asking (pleading really) for help to learn why.

The initial explainer emphasized that, calculating dedupe data puts a high demand on the CPU, & I'm thinking data written with dedupe enabled retains an indices of that fact to avoid errors predicated on those dependencies, & is thus, STILL very CPU demanding, & my data would like to be released on parole, thank you!!


Thus, the CPU-centric emphasis and HOPES that ... ONE of these CPU option below will be fast enough...
(especially given what a TOTAL piece of JUNK the 1.8GHz that the T320 came with ...


All that matters for now is: DEDUPE, SMB & FreeNAS / TrueNAS compatibility

WHICH CPU SHOULD DEDUPE THE FASTEST?

• Option 1: MacPro5,1 (rEFInd) - Dual Xeon 6c (12c total) @ 3.33GHz X5680 (plus turbo).
• Option 2: Upgrade Dell T320 (CPUs on HCL listed below from current E5-2403v2 | 1.8GHz QC ).
• Option 2: i7-9800X - NON ECC (purchased for the 42 PCIe lanes).

CPU series list -- that are compatible (on the HCL) for DELL T320:

Compatible Series​	Fastest Option​	Base / Turbo​	Cores | L3 Cache	PCIe Lanes​	All Support ECC​	Intel Info​
CURRENT POS CPU​	E5-2403 v2	1.8 GHz - N/A ​	4 ( 4 ) 10 MB​	24x 3.0​	3× DDR3-1600	80 W TDP​
E5-2400 v1​	E5-2470 v1	2.3 GHz - 3.1GHz​	8 (16) 20 MB​	24x 3.0​	3× DDR3-1600	95 W TDP​
E5-2400 v2​	E5-2470 v2	2.4 GHz - 3.2GHz​	10 (20) 25 MB​	24x 3.0​	3× DDR3-1600	95 W TDP​
E5-2400 v2​	E5-2450 v2	2.5 GHz - 3.3 GHz ​	8 (16) 20 MB​	24x 3.0​	3× DDR3-1600	95 W TDP​

INTEL CPU SPECS (if you want them):

E5-2470 v1 20MB, 2.30 GHz | 3.10 GHz
E5-2470 v2 25MB, 2.40 GHz | 3.20 GHz
E5-2450 v2 20MB, 2.50 GHz | 3.30 GHz
• i7-9800X 16MB, 3.80 GHz | 4.50 GHz


If clock speed matters most: i7 -9800X < X5680 < 2450v2 < 2470v2
If ECC is the most important: X5680 < 2450v2 < 2470v2

 

[gea]

Performance problems with dedup are mainly due low RAM not so due a slow CPU. The problems happen when you cannot hold the dedup table in RAM but disk based on your pool. In such a situation even a simple snap destroy can last ages.

The RAM need depend on amount of dedup data and settings like recsize. Calculate between 1 GB and 5 GB per TB dedup data. In your case I would try to increase RAM. Another option with newer ZFS is to add a special vdev mirror (NVMe, 100 GB+) to hold dedup data, check How large is my ZFS dedupe table at the moment?

(additionally to dedup, you need at least 2-4GB RAM for OS, a lot for read performance due Arc caching and 2-4GB for write caching.)

ECC is not a strict requirement for ZFS altthough I would not miss with critical data as even ZFS cannot avoid data corruptions after RAM errors without ECC but at least ZFS may detect problems early and set the pool offline then.

 

[TrumanHW]

Performance problems with dedup are mainly due to low RAM and not so much due to slow CPUs.

I've heard this and assumed it as well ... but FWIW, when my R / W performance was worst:
- my RAM usage didn't seem high (less than 50%) and
- my CPU utilization would exceed 80%

In such a situation even a simple snap destroy can last ages.

destroying a snapshot ..? I haven't even MADE any snapshots.

The RAM need depend on amount of dedup data and settings like recsize.

I'm going to research recsize ... but I'm only using about 15TB Data (TOTAL) & have 48GB ECC.

Calculated between: 1 GB -- 5 GB per-TB-of-dedup'd-data.

I highly doubt I have more than 3TB of Deduped data (but I'm def. going to run the cmd to find out)

Another option with newer ZFS [TrueNAS] is to add a special vdev mirror (NVMe, 100 GB+) for DDT

Yup, I started working on that -- making a new array to copy the data off and back ... but I wanted to use a
8GB Radian 200 card (which has a BBU and supposedly is impeccable) for a ZIL
(But TrueNAS says I need either 4 (RAIDz2) or 2 (STRIPED) ...?? So, either equal or double the risk of 1?? WEIRD.

How large is my ZFS dedupe table at the moment?

I'm on it. Thank you.

In addition to dedup requisites, you need:
• ≤ 2-4GB RAM for OS, and ...
• ≤ 2-4GB RAM for Write caching
• a lot for Read performance (ARC caching), and ...

I'm going to research recsize ... but I'm only using about 15TB Data (TOTAL) & have 48GB ECC.

ZFS finds problems ... and will set the zPool to offline.

Watched a video of someone recovering ZFS (I do data recovery) and mentioned the problem with ZFS NOT taking it offline fast enough when it has a real problem.

Thank you again, VERY VERY MUCH.

The CPUs I'm looking at aren't expensive enough to not try one ... and I'm just GOING to get my data paroled from this deduped array.
I'd really, really like to use an all flash array ... and truly ..? I want a unit which DOES go to sleep bc I'm just one person...
I also really want it to be a dRAID instead of RAIDzX ...
-- And given that both dRAID & SSD mean much faster Resilvers (rebuilds) ...
I makes me want to use a SINGLE PARITY SSD and (while retaining 1-2 drives - free space)

But if 2 Zils are required (IF NOT 4!!) (mirrored -- if it'll let me !?? -- as I seem to have no discretion), but I think 1x Zil really is safe, no..?
And if I use a Fusion Pool for 4k - 32k ... (which hopefully it allows me to run as a mirror) means 3x 2.5" SFF U.2 NVMe ...

 

[TrumanHW]

I scrubbed my data before beginning the ZFS Replication Task (now started) ... which took about 24 hours! and used over 50% of my CPU.


Right now, I'm doing the ZFS Replication, as mentioned -- which is a BLOCK level (not object) transfer and thus faster ...
There's no chance this won't actually remove the dedupe-status of the data, right..?
(Obviously the destination machine does NOT have DeDupe enabled!! lol.