How to decipher performance rating on both Enterprise SSD and Consumer

[azev]

I am having trouble understanding IOPS rating from different vendor spec sheet.
I noticed that most consumer SSD actually have higher IOPS rating on their spec sheet than the enterprise counterpart that cost significantly more. I understand that most of Enterprise device have much higher endurance level which I associate with the additional cost.

For example, comparing Intel DC S3700 SSD to a measly MX100 SSD,
Intel is rated at 75000 IOPS/36000 IOPS Random
Crucial is rated at 90K IOPS/85K IOPS Random

Does this mean performance wise the Crucial is way better, especially the write ?

Other than NAND endurance what are the value of paying so much more than consumer SSD ?

If you get a 512Gb MX100 and only used 400Gb of it will that bring it closer to the enterprise level endurance ?

Thanks

 

[Patrick]

Based on a good amount of usage and testing (mostly unpublished at this point.)

You are looking at comparable numbers using clean drives. Once drives get "dirty" and have been used/ filled up a good amount, enterprise drives tend to be much faster and more consistent on performance. The NVMe drives, Toshiba SAS drives and etc are extremely consistent. I have seen Crucial consumer drives take over 800ms to complete a write. The enterprise drives do not do that.

 

[azev]

so in theory, if you crazy over provision a consumer ssd, let say 50% over provision them, would you then have a drive that is almost as consistent as an enterprise drive with better performance ?

 

[andrewbedia]

I don't think that's quite how that works...

 

[Patrick]

so in theory, if you crazy over provision a consumer ssd, let say 50% over provision them, would you then have a drive that is almost as consistent as an enterprise drive with better performance ?

Not quite. Overprovisioning does help to a point. For example, the Intel 730 (and why I am so interested in it) has a higher-clocked processor than the S3500. On the other hand the S3700 has better NAND. Those Toshiba drives I recently benchmarked have higher quality NAND and a 12gbps interface.

Now, on the other hand, if you over provision you can get better consistency in many cases, but you still have limits in terms of NAND and controller to contend with.

Thanksgiving morning maybe I am less than coherent, but the bottom line is: OP helps, but it is not necessarily an equalizer.

 

[wlee]

What if 730 is over provisioned at the same level, would it perform better than S3500? NAND in S3700 is better in terms of endurance, also faster?

 

[Entz]

I would expect that a 730 480GB over provisioned would match or beat a S3500 in endurance (already faster due to the overclocked controller). S3700 is faster then the S3500 but not by much, the S3700's key feature is its high endurance (and sustained writing ability).

S3500/S3700 also have low latency and seem to respond to sync writes far faster then any drive I have seen benchmarked (something to the effect of 10x faster), which makes them ideal as SLOG
devices. I would expect the 730 would do the same, something I have meaning to test (drives will be here tomorrow)

Now making say a MX100 as consistent as a S3500 would be significantly harder for example (As Patrick alluded too).

 

[azev]

Entz, please share your experience using 730 as ZIL, it would be interesting to see what kind of performance are you getting from 730 as ZIL.

And I found this interesting article about overprovisioning

AnandTech | Exploring the Relationship Between Spare Area and Performance Consistency in Modern SSDs

 

[Entz]

Performance of the 730 as a SLOG does work very well. I don't have a smaller S3500 to compare it against but using ESXi over NFS to my file server (10gbit link), using sysbench from a Ubuntu 14.04 VM:
-sync=always: 730 48.457 Mb/s , MX100 512 3.8177 Mb/s, 600 Pro 240 1.5994, None 1.5053
-sync=standard: 730 336.31 Mb/s , MX100 512 199.09 Mb/s, 600 Pro 240 120.47, None 50.667

Still doing some more tests but it seems to be very strong in that role. Using a 480GB SSD for a 8-16GB ZIL is a wee bit overkill . The 120GB S3500 is still the safer bet as we do not know how well the power loss protection works (if at all).

 

[wlee]

It would be interesting to see how Samsung 850 Pro performs as a SLOG though doesn't have power loss protection it has great consistency. Also would be nice to compare with a S3500 / S3700 as baseline.
It is great for myself to learn something new and interesting from forum on daily basis.

 

[azev]

This is great information, It would awesome if we can built a chart to compare different SSD "effectiveness" as a SLOG device.

 

[Patrick]

-sync=always: 730 48.457 Mb/s , MX100 512 3.8177 Mb/s, 600 Pro 240 1.5994, None 1.5053
-sync=standard: 730 336.31 Mb/s , MX100 512 199.09 Mb/s, 600 Pro 240 120.47, None 50.667

Awesome! I am actually surprised the MX100 is beating the 600 Pro by such a wide margin. Then again, the 600 Pro is the only one with confirmed power loss protection.

 

[Entz]

Yeah it is odd, as you would think having power loss protection would allow the drives to cheat a bit when it comes to flush commands. The 600 Pro being slower is confirmed in a few other places and the going theory is that they are simply doing the flush as instructed which increases latency where as the Intel drives are ignoring it.MX100 is a bit of an oddball.

600 Pros are good data drives though (especially at the clearout prices this summer). Worth noting that the 600 Pro was the smallest drive in the set but for the tiny amount of data i doubt it matters much.

I might actually pick up a small S3500 as I was planning on using the 730s as data drives. Would be interesting to see the differences.

I think the Samsung 850 Pros will be a worthy successor to the 840s as data drives, not sure how they fare as log device. You don't always need one either...

 

[Biren78]

Performance of the 730 as a SLOG does work very well. I don't have a smaller S3500 to compare it against but using ESXi over NFS to my file server (10gbit link), using sysbench from a Ubuntu 14.04 VM:
-sync=always: 730 48.457 Mb/s , MX100 512 3.8177 Mb/s, 600 Pro 240 1.5994, None 1.5053
-sync=standard: 730 336.31 Mb/s , MX100 512 199.09 Mb/s, 600 Pro 240 120.47, None 50.667

Still doing some more tests but it seems to be very strong in that role. Using a 480GB SSD for a 8-16GB ZIL is a wee bit overkill . The 120GB S3500 is still the safer bet as we do not know how well the power loss protection works (if at all).

Hey @Entz I'd like to try this with one of my SAS SSD and rotating drives. Any more information on what you did here?
Which sysbench test? Did you apt install sysbench?
So you have a ESXi host Ubuntu VM running sysbench 10gbe link to a switch connected to another machine running ZFS? What OS for zfs? What is the underlying store disk, ssd other?

I'm trying to figure out if this would be an upgrade.

 

[Entz]

Which sysbench test? Did you apt install sysbench?

Yes that is correct, apt-get install sysbench

File Server (Dedicated for the test):
Ubuntu 14.04
E3-1230 v1
24GB Ram
Brocade 1020 10gbe card.
Pool: 4x 2TB WD Red drives, connected to the onboard controller.
NFS folder exported directly on the pool.
SSD log testing was done with the drives connected to a LSI 9211-8I card, running in IT mode.


VM Server (Dedicated for the test):
ESXi 5.5 U2
E3-1265L v3
32GB Ram
ConnectX2-EN 10gbe card
Connected directly to the File server for testing (no switch, dedicated subnet). Using a switch would be easier but I didn't have the free ports.

Test VM:
Ubuntu 14.04
512MB Ram
Dedicated 32GB drive added to the VM for testing, stored on the NFS datastore. Formatted to EXT4

Tests:
zfs set sync=always tank/esxi-nfs
on VM: sysbench --test=fileio --file-total-size=16G --file-test-mode=seqwr --max-time=300 run

zfs set sync=standard tank/esxi-nfs
on VM: sysbench --test=fileio --file-total-size=16G --file-test-mode=seqwr --max-time=300 run

files created by sysbench were removed between tests.

Hopefully that helps.

I was unable to find a sysbench test that matches what ESXi is doing over NFS + ZFS locally. There may be one (likely getting the right block sizes). It is well documented that ZFS and NFS and ESXi are not the best for speed due to sync writes.

 

[mrkrad]

The gist of longevity with flash , is the slower the write process and lower the voltage, the longer the nand will not wear out or destabilize. Which is why the intel has horrible write speeds! and the samsung consumer drives have excellent write speeds.

With overprovision of 30% you can achieve much better steady-state write speeds with samsung ssd in raid-1 !

 

[Entz]

As a follow up to this, I picked up a 120GB S3500 and the number are quite interesting. The lack of NAND does seem to hurt it in a straight run compared to the 480gb on 730 but the thing is extremely consistent (within 1% deviation per run).

sync=always: 45.149 Mb/s :: Very similar to the 730, possibly hitting a limit of some kind
sync=standard: 121.99 Mb/s :: It was across the board -- speed, latency, IOPs -- on par with the 600 Pro 240GB

I would feel pretty confident in saying that the 730 is really just a rebadged and overclocked S3500 . If it does have full power loss protection enabled (likely) there is no reason to get a S3500 (which I am sure is the point in hiding it ), especially at these prices (get double the space for the same money).

 

[Biren78]

I'd bet we're gonna see better SSDs in early 2015. If everything today is gonna be obsolete, the 730 is cheap per gb.