What are alternatives to ZFS for deduplication?

Discussion in 'FreeBSD and FreeNAS' started by NeverDie, Feb 28, 2015.

  1. ATS

    ATS Member

    Joined:
    Mar 9, 2015
    Messages:
    96
    Likes Received:
    32
    Because there is actually very little market for large SMP machines. And those few large SMP machines which are sold are almost never used as large SMP machines, instead being partitioned, either hard or soft, into generally much smaller machines. And that market is further shrinking as core counts per socket expand.

    Money is certainly an issue in this segment. For the most part the large SMP machines are considered by everyone to be fully legacy machines. No one designing a new application system is targeting large SMP machines because they are all cost/performance losers. And as I pointed out, you pricing on the P595 doesn't pass the laugh test, esp since we have historical documented list pricing for a fully configured P595 from the 2008 TPC-C submission that completely disagrees with your statement. And Linux scales fairly well, despite your opinion on the matter.

    No the largest SMP servers ever released were SGI Origin 2K/3K(MIPS/IRIX based) and SGI Altix 3K/4K(IA64/Linux based) machines supporting upwards of 1024 sockets in fully cache coherence SMP systems. The x86 based successor to these systems the UV 2K and UV 300 systems support up to 256/32 sockets respectively in a single cache coherent SMP system and run Linux. As far as 16 socket x86 servers, IBM had them a decade ago. And yes, they ran Linux.

    So, while you may be ignorant of the actual history, the reality is that it exists and you can still buy them today if you want.
     
    #21
    Last edited: Mar 11, 2015
    T_Minus and Patrick like this.
  2. mstone

    mstone Active Member

    Joined:
    Mar 11, 2015
    Messages:
    505
    Likes Received:
    117
    Yup, linux scaled on SGI hardware to 1024 processors almost a decade ago and commonly ran in production partitions of 512. Their current generation has 256 sockets/4096 cores. The poster above who pointed out that sun's offerings were dogs is correct. Most of their systems were chopped up in order to support a number of smaller images in an HA configuration rather than being used for HPC workloads. (The achilles heel was always memory bandwidth, which is why SGI moved to NUMA almost 20 years ago and as sun did eventually. It is true that Sun delivered the high point for UMA SMP machines, but there's a reason everyone has moved away from that architecture. It probably wouldn't have gotten as far as it did if it weren't for the dot com dollars flowing in. (From a cost perspective it almost never made sense to buy an e10k to run a bunch of systems instead of buying a number of smaller systems unless you truly had money to burn and really wanted to be able to point to a giant purple rack.) Mainstream linux/x86 systems don't scale like that not because it's impossible, but because very few people want to do it. Application development for scaling to large systems is hard, and cache coherency has a cost. The reality is that sparc is a strictly legacy technology that's being revved for customers already tied to it (including oracle's own in-house products).
     
    #22
    Last edited: Mar 11, 2015
    T_Minus and Patrick like this.
  3. capn_pineapple

    capn_pineapple Active Member

    Joined:
    Aug 28, 2013
    Messages:
    356
    Likes Received:
    80
    It's about time you two started your own thread for this sort of discussion.
     
    #23
    nickscott18 likes this.
  4. Baddreams

    Baddreams New Member

    Joined:
    Mar 15, 2011
    Messages:
    21
    Likes Received:
    7
    Yes, deduplication in Windows Server 2012 R2. It works amazingly well.
     
    #24
  5. AERuffy

    AERuffy Member

    Joined:
    Dec 12, 2013
    Messages:
    102
    Likes Received:
    17
    Export File extents from freenas via iSCSI, mount in Windows Server, make it a file server and dedupe there

    Surprisingly still fast and tinnnny
     
    #25
  6. uOpt

    uOpt Member

    Joined:
    Dec 12, 2014
    Messages:
    41
    Likes Received:
    7
    I found that dedup doesn't need a lot of RAM during normal operations. Massive amounts of fast deletions might. I found that re-mounting a filesystem after a crash will require large amounts of RAM for mounting. However, nothing bad happened to it on low RAM, it just wouldn't complete until I temporarily connected the entire array to a large RAM mainboard.
     
    #26
  7. methos

    methos New Member

    Joined:
    Dec 19, 2013
    Messages:
    20
    Likes Received:
    0
    You and me both minus the day trader but trying to be a setup trader.
     
    #27
  8. general.zod2

    general.zod2 New Member

    Joined:
    Oct 18, 2016
    Messages:
    5
    Likes Received:
    0
    Just came across this thread. Absolutely fascinating discussion.
     
    #28
  9. dswartz

    dswartz Active Member

    Joined:
    Jul 14, 2011
    Messages:
    377
    Likes Received:
    28
    Nice thread necro, dude... :)
     
    #29
  10. ttabbal

    ttabbal Active Member

    Joined:
    Mar 10, 2016
    Messages:
    723
    Likes Received:
    193
    [​IMG]

    :)
     
    #30
    tjk and dswartz like this.
  11. general.zod2

    general.zod2 New Member

    Joined:
    Oct 18, 2016
    Messages:
    5
    Likes Received:
    0
    This is old, but the HAMMER filesystem on DragonflyBSD does deduplication. It works quite well. But then you'd need to use DragonflyBSD because the filesystem isn't available elsewhere.
     
    #31
  12. BackupProphet

    BackupProphet Well-Known Member

    Joined:
    Jul 2, 2014
    Messages:
    785
    Likes Received:
    278
    You can do dedup on file level. There is a program called rdfind which will create hardlinks for your files. This means that two equals files will refer to the same inode, and if delete one of them it will not ruin the other hardlink. If you remove both of them, then the inode that hardlinks points will be deleted.
     
    #32
    Kybber likes this.
Similar Threads: alternatives deduplication
Forum Title Date
FreeBSD and FreeNAS Does anybody know of a deduplication software for one-time deduplication (and consolidation) ? May 4, 2017
FreeBSD and FreeNAS Deduplication and thin provisioning are not working Mar 16, 2016

Share This Page