Did some write benchmarks of a few SSD's

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Evan

Well-Known Member
Jan 6, 2016
3,346
601
113
Old intel 320's :)
Have a few of them, and 520/530 doing good duty ever since installation.
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,774
2,150
113
960 and 961 are the new NVME or do you mean enterprise?
 

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,774
2,150
113
Those products are on the above graphs is what I was referencing. Sorry for lack of clarity.
 

DouglasteR

Active Member
Dec 19, 2015
135
37
28
So...

A S3500/3700 will feel much more faster than a 960 Pro for example ?

AFAIK, QD1 is the most important depth for perceptive quickness, correct ?
 
Last edited:

CookiesLikeWhoa

Active Member
Sep 7, 2016
112
26
28
35
So...

A S3500/3700 will feel much more faster than a 960 Pro for example ?

AFAIK, QD1 is the most important depth for perceptive quickness, correct ?
What usually makes a desktop "feel" faster, in my experience at least, is latency.

As pointed out above most desktops don't need a ton of iops, but latency makes a noticeable difference.
 
  • Like
Reactions: james23

matt_garman

Active Member
Feb 7, 2011
230
68
28
So do we know why the enterprise drives are so much better? Two things I believe to differentiate consumer versus enterprise: controller and over-provisioning. How much of an effect does each have? And are there other things that make it better?

I'd be curious to see the results these same tests run on consumer SSDs with significant amounts of over-provisioning (say 25-50%).

Edit: Here's a related link, Write Saturation And Over-Provisioning Tests.
 
Last edited:

Stephan

Well-Known Member
Apr 21, 2017
1,033
801
113
Germany
Enterprise SSDs have:

- Better cooling, so no or little need to throttle the controller and with that read/write speeds to stay within thermal envelope
- Faster interface speeds e.g. SAS 12GBps, U.2, PCIe, instead of SATA
- More advanced electronics like power loss protection capacitors, ECC-RAM for the controller
- Firmware engineering is better, e.g. LDPC instead of BCH error correction, latency guarantees, reliable health warnings
- End-to-end data protection, data is protected through additional checksums from its way to/from flash chip to/from CPU
- MLC (2-bit per cell) instead of TLC (3-bit per cell) flash, although advances in 3D-NAND is changing that
- Select A-die flash silicon from premium bins
- More flash channels from controller to flash are used, primarily to increase write speeds
- More space dedicated to over-provisioning to be able to replace bad cells for larger TBW (TeraByte Written) over drive's lifetime
- Self-encrypting drive junkware features (you should not trust this)
- Higher capacities available than with consumer SSDs

Did I forget something...
 

Evan

Well-Known Member
Jan 6, 2016
3,346
601
113
@Stephan there is plenty of enterprise sata and tlc drives out there.
Samsung and Intel make a bunch of sata drives. Samsung especially has a heap of tlc drives, mixed use models as well not just read optimised ones.

You touched in error rates and it's sometimes clear that enterprise have lower error rates 1 x 10 ^17 where some consumer drives it's not stated or less.
 

matt_garman

Active Member
Feb 7, 2011
230
68
28
Enterprise SSDs have:

- Better cooling, so no or little need to throttle the controller and with that read/write speeds to stay within thermal envelope
- Faster interface speeds e.g. SAS 12GBps, U.2, PCIe, instead of SATA
- More advanced electronics like power loss protection capacitors, ECC-RAM for the controller
- Firmware engineering is better, e.g. LDPC instead of BCH error correction, latency guarantees, reliable health warnings
- End-to-end data protection, data is protected through additional checksums from its way to/from flash chip to/from CPU
- MLC (2-bit per cell) instead of TLC (3-bit per cell) flash, although advances in 3D-NAND is changing that
- Select A-die flash silicon from premium bins
- More flash channels from controller to flash are used, primarily to increase write speeds
- More space dedicated to over-provisioning to be able to replace bad cells for larger TBW (TeraByte Written) over drive's lifetime
- Self-encrypting drive junkware features (you should not trust this)
- Higher capacities available than with consumer SSDs

Did I forget something...
Agreed, but not all those things necessarily contribute to improved write performance.

I'm trying to look at write performance on enterprise versus consumer in a "all things being equal" perspective (to the extent possible of course).

In particular:
  • Cooling - is it possible to construct an environment such that the consumer drive's "lesser" cooling becomes a non-issue?
  • Interface - of course there are things better than SATA, but what if we look exclusively at SATA drives to help level the playing field
  • Power loss protection - great feature, but I don't think it helps with write performance
  • MLC vs TLC - there are MLC-based consumer SSDs
  • Better binning - does this make for faster NAND, or just more reliable/long-term stable NAND (honest question)
  • More flash channels - I'd file this under controller, but yes, I can definitely see how this would improve write performance
  • Self-encrypting - I don't think this would affect performance
  • Higher capacities - I'm interested in comparing equivalent capacities
  • More over-provisioning - definitely helps, but you can change the amount of OP on consumer drives
I think, except for the controller, most differences that can affect write performance can be normalized. So then we could see how much difference the controller actually makes.

Let's say, just for sake of argument, you don't care about the non-performance impacting perks of enterprise drives. Now, can you game the over-provisioning to get similar performance and a lower cost per GB? Or maybe you don't need to perfectly match performance; the graphs above show basically a 20x improvement. But what if your application only needs a 10x improvement?
 
Last edited:

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,774
2,150
113
Hmm, maybe I don't get it? Why?

You can easily already see how much a controller makes, it's a huge difference.

Everything you've talked about "normalizing" is pretty much irrelevant other than OP, which just extends the time before the consumer drive does slow down, and it will.

You also can't compare like capacity to like capacity 'on that point' because the NAND may be 16GB 32GB 64GB etc, and having 24x 16 vs 12x32 most def. affects performance as you can see how the newer gen Intel SSD performance dropped in certain work loads due to this. So a 500GB consumer with 16GB on each chip != enterprise with 32gB on chip, etc...


I think the problem is you're going by the "Rated Spec" and a consumer drive don't ever go for 100% drive it's like 8GB span only, so you'll NEVER get the write out of a consumer as an enterprise does. Look at Steady State performance in reviews of SSD it's clear as day to see this, and OP does help here also clear, but it's just a delay to the slow down not a remedy for it, because it's in the firmware of the enterprise SSD.


So in the end, I'm not sure what you're trying to accomplish.
You can't make a consumer SSD write like an enterprise one in terms of performance, consistency and ultimately reliability... sure a consumer COULD burst to the write perf. of an enterprise ssd under certain work loads, and assuming it's not constantly being written/read in the background too.
 

matt_garman

Active Member
Feb 7, 2011
230
68
28
Thanks, I see it's a little more nuanced than I originally assumed.

Hmm, maybe I don't get it? Why?
So in the end, I'm not sure what you're trying to accomplish.
Just to give a little more color to where I'm coming from: originally I was curious about enterprise vs consumer strictly for home use (see this thread). In my case I know my writes are minimal. And am I willing to pay the enterprise premium for the other benefits? In my mind, since I am certain I don't need the write performance, the other benefits are like insurance. Everybody has a choice for how much insurance they want to buy, right? Do you have disability insurance, life insurance, home owner's insurance (not required if you don't have a mortgage), what level of auto insurance, etc. Except for legal requirements, there's not a right or wrong, and everybody makes their own cost vs risk analysis.

So while I started out thinking about this in terms of my home situation, it also applies to what we're doing at work. We have a home directory server for about 40 employees using good old 7200 rpm spinners in a software raid config. It works and is stable, but the speed is lacking. If it's being lightly used, it's not bad, but if (for example) someone kicks off a big compile, then everyone suffers.

Now... at work we also have a few 24-disk consumer (Samsung 850 Pro) SSD arrays, see here. Credit to @dba for the Dirt Cheap Data Warehouse inspiration. The "catch" though, is that these arrays are literally 99.999% reads. It's effectively a WORM workload. We add a tiny amount of incremental data every day, and the rest of the time it's just constant reading. Using this allowed the company to save a ton of money by avoiding going with a big iron storage vendor. Even now, we max out the dual 40gb network interfaces (with jumbo frames!) before the drives run out of steam.

I made the thought mistake that, since the read performance is so wonderful with these drives, how bad could the write performance really be? Seeing the graphs above, and the related discussions, I see the answer is: pretty bad!

So, coming back to the home directory server. While we recognize the current server is slow, the higher-ups are very "financially conservative" (I'm being generous here)... So before I understood the huge consumer vs enterprise write performance discrepancy, I was thinking: we get a 2U chassis with 24 2.5 hotswap bays. Fill it with eight 1tb consumer SSDs. Raid-6 them together for nearly 6tb of storage, and 16-bays left for future requirement increases. I haven't revisited the numbers in a while, but last I checked, the base server would be about $4k and the drives another $4k, so $8k all-in. Now the higher-ups are anchored to that number. I haven't done any cost estimation for enterprise drives yet, but I'm sure the cost is going to be much higher.

Also, to give a little insight to the "ignoring non-performance impacting benefits of enterprise SSDs" mindset: the insurance analogy I used above is apt, because one of the company owners in fact does not have insurance on his house. He can afford to "self insure". Paradoxically, the cost of home owner's insurance is therefore trivial to him, but he believes the chance of a disaster so low that he's willing to make the trade off. So hopefully that gives you a little insight into the thought-process under which I'm working. I've worked here for over a decade; I can say with very high confidence, excepting for performance issues, the owner is definitely willing to trade all the other risks to save some money. The performance issue could be a deal-breaker, but if I can close the performance gap (even partially) with over-provisioning, then that's almost certainly what I'd be instructed to do.
 

ATS

Member
Mar 9, 2015
96
32
18
49
So do we know why the enterprise drives are so much better? Two things I believe to differentiate consumer versus enterprise: controller and over-provisioning. How much of an effect does each have? And are there other things that make it better?

I'd be curious to see the results these same tests run on consumer SSDs with significant amounts of over-provisioning (say 25-50%).

Edit: Here's a related link, Write Saturation And Over-Provisioning Tests.
Power Loss Protection. For things like Sync Writes that is a killer feature. Consumer drives that don't have PLP, can't legitimately Sync until the data is actually on the flash, and enterprise drive with PLP can signal Sync the second it hits the write cache.
 
  • Like
Reactions: Stephan