Does anybody know of a deduplication software for one-time deduplication (and consolidation) ?

[spyrule]

I have a NAS that has... a whole bunch of copies of the exact same files all over the place. I would like to find all duplicate files, list them, and consolidate those files. None of these files are system level files, so deleting exact matches won't be a problem.

Does anybody know of software to do this?

Basically, I'm trying to organize almost 1TB of documents/images (almost all of them in small file size, so ~3.5M total file count).

This happens to be on a synology, but I'm ok with the software being either MAC or Windows based (prefer windows actually).

 

[Blinky 42]

On *nix try hardlink...

hardlink(1) hardlink(1)

NAME
hardlink - Consolidate duplicate files via hardlinks
.....

Can take ages the first time through if you have millions of files but does what it says on the tin and saves a lot of space if the files are identical.

Not sure of a windows variant, you would be better off running something directly on the box anyway to avoid reading the entire archive contents across the network.

 

[Evan]

Alldup or dupeguru - free
Beyond compare 4 - paid

Plenty of others that run on a workstation like windows, just Google, download a few and see which interface you like most.

 

[cesmith9999]

Treesize - www.jam-software.com

Chris

 

[i386]

I like antitwin* for "normal" files and vispics for images.

*the 1.9beta supports >2gb files

 

[K D]

Ccleaner has a built in dupe finder.

For music, iTunes or media monkey.

For photos, if you have adobe Lightroom it is very powerful.

I've used Dupe guru but I seem to remember it was different separate applications for different file types.

 

[spyrule]

Awesome info guys, I'll be testing these this week.

 

[Terry Kennedy]

I have a NAS that has... a whole bunch of copies of the exact same files all over the place. I would like to find all duplicate files, list them, and consolidate those files. None of these files are system level files, so deleting exact matches won't be a problem.

Does anybody know of software to do this?

If you want to run it locally on the NAS (for example, if your NAS disk-to-CPU speed is a lot faster than your LAN connection), look at fdupes.

 

[NashBrydges]

If this is on a Synology, why not use the Storage Analyzer which has this function built right into it? Create a report and one of the options is called "Duplicate File Candidates"

 

[sfbayzfs]

Years ago when I was using a Windows machine in with my *nix ones, the best Windows deduper I used was an old portable version of "easy duplicate file finder" before it went paid - Easy Duplicate File Finder - The Portable Freeware Collection but since then I have found that fslint-gui in *nix is my favorite - much easier to use, and since Win10 claims to run linux binaries, it should work in Windows. If that isn't easy to figure out, I would just run a VM of the latest ubuntu and mount the NAS on it and install and run fslint from there.

fslint-gui is really easy to use - add the path(s) you want to check, flick the "find" button, wait for the report, and then choose what to do with the dupes. Duplicates are always sorted only by amount of space wasted from most to least, and CTRL-click and shift-click multi-select how you would expect, but there are a lot of great options if you right click on a header (select and unselect using wildcard are extremely powerful, and can match paths as well as filenames with *nix style globbing) right clicking on a file lets you "open folder" so you can easily open the folders containing all instances of a duplicate, and you can also "select from same folder" so if after you inspect 3 folders containing duplicates you can determine which one is oldest and then select all from that one in fslint and delete them with confirmation dialog boxes. If there's anything left that wasn't a duplicate you haven't lost it, and you still have the 3 folders open so you can move things around between them.

My ideal deduper would also be able to operate at a directory level and be able to identify entire duplicate file trees, but I haven't found one with that feature yet.

 

[Peterson Janes]

I have a NAS that has... a whole bunch of copies of the exact same files all over the place. I would like to find all duplicate files, list them, and consolidate those files. None of these files are system level files, so deleting exact matches won't be a problem.

Does anybody know of software to do this?

Basically, I'm trying to organize almost 1TB of documents/images (almost all of them in small file size, so ~3.5M total file count).

This happens to be on a synology, but I'm ok with the software being either MAC or Windows based (prefer windows actually).

I use a software called Duplicate Files Deleter. It will give you a comprehensive list of all those files and you can decide for yourself what you want to do with them. I hope it will help you.

 

[msg7086]

rdfind

 

[Davewolfs]

Hashbackup

 

[Evan]

Hashbackup

Quick look that's looking interesting.