How a very large ZFS pools configured ?

Discussion in 'FreeBSD and FreeNAS' started by Fritz, May 13, 2016.

  1. ttabbal

    ttabbal Active Member

    Joined:
    Mar 10, 2016
    Messages:
    719
    Likes Received:
    193
    I haven't done SSD arrays, but with spinners I've done all the same and mix-and-match. Never had a problem with either way, so long as I thoroughly test the drives. On SSD I would be even less concerned about it, particularly used units. The built-in wear leveling is likely randomized at least a little by the drive firmware, and the order any flash cells fail, so write patterns won't be exactly the same even in a mirror. With used units, the existing data use should push them further out of sync, if they ever were.

    All this is based on ZFS. I've read some people say that at least older hardware RAID required identical drives. ZFS doesn't seem to care either way, all that matters is that replacements be >= the old drive size. New arrays use the lowest common size. I've even done mirrors with a 120GB and a 500GB. The 500 was a replacement and just happened to be the only available known good drive I had sitting about when one side of the mirror died. It worked fine.

    Oh... the Seagate 7200.12 debacle ... I feel for you. If it makes you feel any better, I had about 6 of the infamous 75GXPs die in the same month. The others were burned in effigy. :)
     
    #41
  2. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    739
    Likes Received:
    182
    Yeah I see what you are saying Ryc, it just seems like we aren't giving Unwind a chance to defend his opinion. (I see though, he hasn't really tried too do so yet) but everyone's theory can have some validity to it right? I too wish though, that Unwind would support his opinion a little more, at least since he was specifically asked to, as his theory is up for debate.

    Unwind my friend, if you are reading this, try to give a bit of clarity to your position eh? You haven't responded as yet, but we would like to hear about your thoughts if possible.

    I do agree that we are not nowhere near the Freenas climate and for that, I am terribly glad! I deeply apologize if drew too close a comparison to it. Reading through and seeing dregs of tension brought back those memories of trying to be a member of the Freenas forums. Your opinions there were ignored, and there was literally no respect for a person's ideas, and I prayed for the poor soul who asked a question when he/she needed help. I don't think this is good for any open forum.

    However, it seems like many attempts were made to directly inquire on a forum members ideas, so this is a excellent thing. Frustration too, may very well be expressed through writing, and while it may not desired by one or two scarred folks such as meself, it doesn't represent a hatred for new ideas/personal opinions, but a expression of an emotion to understand the thought process of another. This too, is an excellent thing.

    I just don't know though, I just don't see how that particular theory of Unwind could possibly hurt others though. Seems abit odd to me at best. I mean, it would be interesting to know if a hypothesis like that can in fact cause damage to your setup, but I feel like it just can't. I agree though, most setups will not come with a mixed bag of drives. Seems like a personal setup type thing if anything, as Patrick pointed out.

    Maybe, in defense of Unwind, he is thinking of a mutual read/write - longevity - performance benefit from using different manufacturer ssd's?
     
    #42
  3. Terry Kennedy

    Terry Kennedy Well-Known Member

    Joined:
    Jun 25, 2015
    Messages:
    1,016
    Likes Received:
    472
    There were lots of drive-level features to support older RAID controllers, or even non-RAID controllers. First was Rotational Position Sensing, which was an option* on washing-machine-sized drives. It told the controller how far away the first sector was. This was pretty important when the disk was only spinning at 1200 to 1500 RPM. CPU speed increased far more rapidly than disk speed, so even when disk speed was standardized at 2400 and 3600 RPM it was still important.

    Somewhat later, Spindle Sync was introduced. This caused all drives to operate at exactly the same speed. It was beneficial because there were usually multiple drives on the (SMD / SCSI / whatever) data bus and the system could order writes so that a multi-sector write could be performed across the drive array without "blowing a rev". This could mean the difference between being able to write the data in one revolution vs. multiple revolutions.

    The last use for Spindle Sync was to get updated firmware from drive vendors that normally didn't want to make updates available. You could get a drive back from an RMA with a newer firmware revision and you were normally out of luck. But going "I need spindle sync!" was pretty much guaranteed to get you the new firmware for your other drives.

    * You might thing this was something silly to charge for. But "way back when", Direct Seek was an option. If you didn't buy the option, every time the drive did a seek it would recalibarate to Track 0 first, then seek to the desired track. So if you were on track 117 and wanted 118, it actually went 117 - 0 - 118. Totally clobbered performance.
     
    #43
    Fritz likes this.
  4. unwind-protect

    Joined:
    Mar 7, 2016
    Messages:
    119
    Likes Received:
    9
    Well, unfortunately I'm not (yet) in the habit of saving all web links.

    To understand my position on homogenous mirrors on SSDs you first need to understand that from my experience SSDs practically never die when their expected lifetime of writes is exceeded. I have a stack of SSDs that I killed (some by accident, some deliberately) that all had the controllers (the ones on the SSD) die under specific "non-expected" write patterns. Doing that was pretty easy, and every new generation everybody came out that those problems are well-known and now fixed in the current generation. Then the hyped Intel SSDs that were all the huge behaved the same way for me and I had it. The Samsung 850 was the first one that I couldn't kill within a day, but somebody over at Tomshardware did discover a pattern to do that. As I mentioned elsewhere I have non-death problems with the Samsung 850, too.

    So, if you believe, like I do, that SSDs can be killed with specific write patterns, then very obviously you don't want to back a single raw device with a mirror on drives that will die on the same write pattern.

    I do remember that both those patterns and the "don't do raid1 on SSDs" issue were popular enough to be Slashdotted at some point. Today people just say "raid is no backup, what did you expect" when people lose critical mass in arrays, so no real discussion happens anymore.
     
    #44
  5. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,769
    Likes Received:
    1,111
    We actually understand you position. The problem is that your position is based on ideas that are not consistent with how SSD actually work or any any facts or data published elsewhere.

    Without facts or data or empirical evidence to support it all we are left with is a rather odd misinformed idea. Everyone is, of course, entitled to whatever ideas they choose to believe. But when they start advising others, who may be novice and naive, they take on some an obligation to reasonably defend that advice - and 'I think I remember reading somewhere once' is a pretty weak defense indeed. And 'I'm not in the habit of saving Web links' is fairly interpreted as 'I got nothin'.

    Again - always welcome to provide any evidence to support this idea. Until then I continue to call it FUD.

    Sent from my SM-G925V using Tapatalk
     
    #45
    Last edited: Jun 5, 2016
    Fritz likes this.
  6. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,767
    Likes Received:
    1,453
    Agreed.


    (expand above quote to see emphasized selections)

    Since you don't have any other data, articles or proof except what you've done... what about the "easy" way that you, yourself killed these SSD?

    You mention "specific" write patterns... what exactly are your specific write patterns that are taking down "hyped intel" and the other "stack" of SSD that you have sitting there?

    What are the "hyped" Intel SSD? Specific make/model please. Firmware version would be nice too.

    It seems to me that since you have done it yourself, and repeatedly do it that you can:
    - Share how you're killing them exactly
    - Narrow down the issue to something in your setup that's killing them

    I look forward to the additional information on this topic.
     
    #46
  7. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    1,062
    Likes Received:
    353
    Wow, how did I miss more discussion about this enthralling topic?

    It's perfectly safe to run the same discs (either SSDs or platter drives) in RAID arrays as long as you buy one of my patented magic rocks that prevent the transmission of non-expected write patterns through SATA cables*. I'll also give away a litre of snake oil (a $700 value!) if it doesn't also work as a tiger repellent.

    * SAS version is twice the price, obviously.
     
    #47
  8. Sleyk

    Sleyk Active Member

    Joined:
    Mar 25, 2016
    Messages:
    739
    Likes Received:
    182
    Come on, you can't be making a profit on that Snake oil? The price is just too reasonable. You sir, are a good man!

    I am actually saddened to hear that. You make a good point about that. It does seem as if any one mentions any other sort of RAID solution besides what is considered safe or accepted, it does get sort of written off as just you not following accepted advice.

    I am truly curious too, Unwind, have you ever come across any articles or any other evidence of this sort of issue? Did you perform any type of experiment yourself? I am actually gonna have a scenario where I will be looking at grabbing a few different ssd's from different manufacturers to raid. I might be willing to grab two different manufacturers like Corsair and OCZ and swing them through some torture testing to see if any errors arise from using them in this way.
     
    #48
    Last edited: Jun 20, 2016
  9. cookiesowns

    cookiesowns Active Member

    Joined:
    Feb 12, 2016
    Messages:
    142
    Likes Received:
    41
    I personally believe what Unwind said has some truth to it. But... the likely hood of this happening in the real-world is very unlikely. I've ran RAID-0 SSD's in production for a while, back in X25-E, or X25-M days.

    We needed the IOPs, and at the same time raid-0 has 1 very significant benefit, you are halving all writes to SSD's. So if endurance was your concern RAID-0 might actually increase your MTBF, because each drive is getting 50% of writes, so you have 50% extra of life-time. Assuming SSD's all wear & die at the same exact rate.But I'm sure you guys knew that :p

    As for ZFS, personally if it's just for a backup / non production rig, I wouldn't mind going 8-12 drive vdevs. For production stuff in 24 bay chassis's I've stuck with 4x 6 Z2's with 4 & 8TB drives.
     
    #49
    Benten93 likes this.
  10. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    1,062
    Likes Received:
    353
    I still think you might be conflating different issues; you seem to be saying that RAID0 works for you to achieve a certain number of IOPS, and that device reliability has been good enough under that configuration that you've not had any significant catastrophic failures. That's all well and good, I've used RAID0 myself before for scratch databases.

    (As an aside, in the wild SSD failure from flash write exhaustion seems to be incredibly rare... most SSDs die from failure of the controller in my experience)

    unwind-protect seems to be saying that you shouldn't run two of the same SSDs in any sort of RAID array since there are apparently magical Read/Write Patterns Of Doom that will brick your SSD, shave your eyebrows and give your dog Dutch elm disease. This goes from harmless old-wives tale to actively harmful advice IMHO so it needs to be challenged.
     
    #50
    T_Minus likes this.
  11. cookiesowns

    cookiesowns Active Member

    Joined:
    Feb 12, 2016
    Messages:
    142
    Likes Received:
    41
    Yes,

    I should have mentioned I was digressing from the actual topic on hand. Whatever folklore that unwind believes regarding magical write patterns is beyond me.

    That said I mentioned the benefits of raid0 in specific scenarios as it actually reduces writes to each drive, unlike a direct mirror which is 1:1, but you guys all probably knew that.
     
    #51
  12. unwind-protect

    Joined:
    Mar 7, 2016
    Messages:
    119
    Likes Received:
    9
    I have a stack of dead SSDs to prove it.

    Send me a consumer SATA SSD and I'll kill it without exceeding the documented amount of allowed writes total. It isn't even difficult. Apart from what I do at work I found that putting ZFS on top of disk encryption in CBC mode will generally do it. Varying the block mode will do some more.

    In any case, this doesn't have much to do with why homogenous raid1s are a bad idea if this is the time where you start using a then-new model of storage device. Individual models of harddrives or SSDs that have specific defects in all manufactured pieces are not exactly uncommon.
     
    #52
  13. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,767
    Likes Received:
    1,453
    Do you purposely ignore myself and others who ask you specific questions? It's as-if you like to repeat yourself over and over without providing any actual information re: your process, and specific use-case experiences.

    At-least now we know you're doing it with drive encryption + ZFS but you've been very vague again about: apart from what's done at your work.

    We don't know WHAT you're doing at work.
    We don't know HOW you're killing them at work.

    Are we going to wait over a month for you to reply with a different cryptic story?

    I also noticed you went from "hyped intel" to simply "consumer sata ssd"... does this mean you've not accomplished this with an enterprise intel drive?
     
    #53
  14. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,769
    Likes Received:
    1,111
    I'll second @T_Minus: describe your specific method for destroying the consumer SSD so others can verify it.

    Sent from my SM-G925V using Tapatalk
     
    #54
  15. vBuild2BAU

    vBuild2BAU Member

    Joined:
    Feb 27, 2016
    Messages:
    43
    Likes Received:
    9
    I'll Third @T_Minus: I would also like a guide on how to kill a consumer SSD quite easily.
     
    #55
  16. EffrafaxOfWug

    EffrafaxOfWug Radioactive Member

    Joined:
    Feb 12, 2015
    Messages:
    1,062
    Likes Received:
    353
    Here's the command I've been using:
    Code:
    effrafax@wug:~# cat /sys/block/sdk/queue/rotational
    0
    effrafax@wug:~# cat /sys/block/sdk/armour/invulnerabletohammers
    1
    effrafax@wug:~# hit -d /dev/sdk --with-tool /dev/massive_fscking_sledgehammer --magical-write-pattern /dev/zero
    /dev/sdk now proper dead
     
    #56
    cperalt1, cookiesowns and rubylaser like this.
  17. rubylaser

    rubylaser Active Member

    Joined:
    Jan 4, 2013
    Messages:
    842
    Likes Received:
    229
    +1 for awesome pseudo code :)
     
    #57
  18. unwind-protect

    Joined:
    Mar 7, 2016
    Messages:
    119
    Likes Received:
    9
    I think it would be overall better to postphone this until after we meet in person. Maybe it's time for a forum meet? :) I'm not a bad guy, I just have a stack of dead consumer SATA SSDs (actually one at work and one at home). I can't connect my online identity to my work and in any case you wouldn't be able to verify it unless we meet in person.

    FWIW I had several deaths that were simply running my version of bonnie++ on top of a ZFS on top of either FreeBSD/geli or Linux/dmcrypt.

    I think it is important to pick a cipher block mode that does the most unexpected things for the SSD in question.

    Now, you can tell from the vagueness of my writing what the big problem is: how do I test whether this is really reproducible? I don't have an endless supply of spare SSDs around. The last SSDs I have at home I stuffed into my gaming-only windows machine where they didn't die yet. At work things have moved on from having to evaluate storage hardware. In any case, nobody there uses SATA SSDs for anything important.

    As a result I also don't have a command-by-command record and I can't tell you specifically which OS with which block mode killed the things. On dm-crypt I usually use aes-cbc-essiv:sha256. On geli I use XTS. I believe those are the current defaults anyway.


    In addition to all this I repeatedly had consumer SATA SSDs hang for extended periods, or even indefinitely. Not good for long uptime.

    FWIW I cannot killed magnetic drives with any access method I had ever used. Seagates die like flies in my hands but I rate that as working as intended.
     
    #58
  19. PigLover

    PigLover Moderator

    Joined:
    Jan 26, 2011
    Messages:
    2,769
    Likes Received:
    1,111
    FUD

    Sent from my SM-G925V using Tapatalk
     
    #59
  20. wildchild

    wildchild Active Member

    Joined:
    Feb 4, 2014
    Messages:
    394
    Likes Received:
    57
    Actually, purely from a theoretic point of view i kind of agree with Unwind.

    Wether it is write patterns, rotten firmware, other unknows, i dunno, or simply updating firmware to the SSD's while keeping your pool online and running
    If given an option i can see why someone would build up a pool of 2 different brands of disks.

    i know Dell did it for some time with their ZNAS devices

    Actually did it myself with spinners, back in the dying 3 gb disks
     
    #60
Similar Threads: very large
Forum Title Date
FreeBSD and FreeNAS ZFS/FreeNas : Large pools design with a HGST 4U60 rack Jul 28, 2017
FreeBSD and FreeNAS ZFS storage space maximization in large zpools May 24, 2015

Share This Page