SSD read endurance tests

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Lance Joseph

Lance Joseph

Member
Oct 5, 2014
82
40
18
Tech Report did a piece a few months ago about SSD write endurance testing.
This got me thinking that I've not heard of anyone doing read endurance tests on SSDs.

It'd be great to hear of any experiences on this front.
I'm going to be doing very read-heavy production workloads with some "Pro-sumer" grade SSDs.
These drives are anticipated to be assembled in a RAID-10 configuration for SQL databases in read-only mode.

Has anyone done or heard of any tests being done like this before? Either publicly or privately?
I'm very tempted to experiment with some older and mostly unused Samsung 840 Pro drives.

Reading a raw device over and over again in a loop while periodically dumping the SMART data could be interesting.

Who wants to try to burn up some SSDs?
 
D

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
  • Like
Reactions: T_Minus and Chuckleb
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
OP if you go with an 840 Pro.

I'd urge a S3500 for read-heavy usage.
 
  • Like
Reactions: b3nz0n8
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I've been eyeballin' those for a couple weeks, they were listed @ $259 now $249, not a bad deal at all for 600Gb s3500's, not a fantastic deal either. I have not seen the 800gb s3500's avail for a month or two at least w/ any general availability in the $325-375 range but they were there a few months back for a good lil bit, had a buddy of mine get a dozen of them @ $300 each.
 
  • Like
Reactions: T_Minus
Lance Joseph

Lance Joseph

Member
Oct 5, 2014
82
40
18
FWIW, I started doing a series of full reads on a Samsung 840 Pro as of yesterday.

What I've noticed about the SMART counters is mildly interesting.
The SMART attribute for "199 CRC_Error_Count" has incremented by 1 each day and is now at 3.
It's not entirely clear what this means at the moment, but I'm going to let it run through the weekend and see where this counter stands.

This is what I'm currently running:
#for f in [1..300] ; do dd if=/dev/sda | pv | dd of=/dev/null bs=1024K ; done
Each cycle takes about 15 minutes so this round of testing should be done by Monday.

The CRC_Error_Count counter has piqued my curiosity about the consistency of results that are being returned on each read cycle.
To test for consistency, I'm going to run the following command to see if end-to-end reads return the same MD5 hash:
# for f in [1..100] ; do md5sum /dev/sda >> /root/sda_md5sum ; done

I'm cautiously not pointing any fingers at the drive as this "issue" could very well be caused by the SATA cable or SAS card.

Let's see what happens...
 
  • Like
Reactions: Chuntzu and T_Minus
Lance Joseph

Lance Joseph

Member
Oct 5, 2014
82
40
18
Quick update:
I've run through approximately 1000 full-read cycles of the Samsung 840 Pro.
Checksums return the same result each time which is expected.
Each cycle takes approximately 15 minutes to complete.

SMART counters are mostly the same.
The Wear_Leveling_Count has incremented from 10 to 11.
I've not done _any_ writes to the drive, as the Total_LBAs_Written counter hasn't budged.

I'll continue testing until more noteworthy behavior occurs.
 
Patriot

Patriot

Moderator
Apr 18, 2011
1,451
792
113
Lance Joseph said:
Quick update:
I've run through approximately 1000 full-read cycles of the Samsung 840 Pro.
Checksums return the same result each time which is expected.
Each cycle takes approximately 15 minutes to complete.

SMART counters are mostly the same.
The Wear_Leveling_Count has incremented from 10 to 11.
I've not done _any_ writes to the drive, as the Total_LBAs_Written counter hasn't budged.

I'll continue testing until more noteworthy behavior occurs.
Click to expand...
depending on the filesystem... noatime... also known as... a write on read.
Unless you are tracing your workload... you don't know that you are not writing anything to disk...
 
M

matt_garman

Active Member
Feb 7, 2011
212
41
28
Lance, would you be willing to elaborate on the details of your platform? E.g., distribution, kernel version, filesystem, filesystem mount options, any tuning, etc. Also, the hardware specs: specific drive model numbers and controller/HBA?

Can you also continuously run iostat in parallel with your dd tests (say at a five minute interval)?

I ask all these questions because we are doing something very similar, and seeing the read performance of the SSDs degrade over time. Our workload/testing is also effectively read-only. Our test is conceptually similar to yours, although a bit fancier. First we generate a few TB of random data with a size distribution (on the order of a few MB to a few GB, normally distributed, with the median around 500 MB). Then we have multiple parallel dd reads taking place, continuously. The dd read jobs pick files (from the generated file set) at random. We also run iostat in parallel, logging output to a file.

What we see is that all drives start off with a read rate that is near their advertised theoretical max (400 MB/s or better). We get the device read rate from iostat. Over time (on the order of hours) this drops down to 1/10th of the initial rate (30ish MB/s). Given enough time (say a weekend run), performance may go back up... and then down again. Overall read throughput has the shape of a sine wave, though the period isn't consistent. That is, it may be five hours peak-to-peak, and the next peak comes after seven hours, and the next after three, etc.

We are using CentOS 6.7, kernel version 2.6.32-573.1.1. Drives are eight Samsung 850 EVO 2TB (MZ-75E2T0B/AM). They are connected to an Avagotech-branded LSI MegaRAID SAS-3 3108 (9361-8i) RAID controller. We have tried both hardware RAID-5 through the controller, and Linux software RAID-5 (using the controller as an HBA). So far we've tried ext4 and xfs filesystems, with the default mount options. I know there's a lot of tuning to be done, but we wanted to get a consistent baseline first. This fluctuating behavior is disconcerting.

We've also tried using disks individually, rather than in RAID. We see similar behavior.

Another thing that is weird: the bad performance is persistent through reboots. That is, we can be testing, hit a performance trough (double-digit read throughput), reboot, give the machine some time to "cool off" (in case heat is an issue), then re-start the tests and performance is still miserable. The only way to get performance back is to let the test run for hours, or re-create the filesystem (i.e. format).

Note that as someone mentioned above, even though our tests are read-only, we are in fact implicitly doing some small writes due to access time updates. We're using the default mount options, which implicitly turns on atime and diratime. We're going to do some testing with disabling those options (i.e. eliminate all writes) to see if that fixes the issue.

I can't see SMART data, as this controller hides that.
 
  • Like
Reactions: Lance Joseph
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
I have read numerous performance woes threads about using LSI HBA/ctrls + samsung ssd's. Maybe another case. Put a true enterprise class ssd in there and repeat same tests and I bet the result is different.
 
M

matt_garman

Active Member
Feb 7, 2011
212
41
28
whitey said:
I have read numerous performance woes threads about using LSI HBA/ctrls + samsung ssd's. Maybe another case. Put a true enterprise class ssd in there and repeat same tests and I bet the result is different.
Click to expand...
Well, the goal here is low-price, and enterprise-class drives remove the low cost aspect. :)

Our workload is literally 99% reads, and my understanding is the major increase in cost for enterprise class drives is due to more robust NAND that can tolerate more writes over its lifetime.

Anyway: we have removed the LSI controller as the culprit. Our system originally had the eight Samsung drives connected to the LSI controller, and two Kingston SSD drives connected to the motherboard's SATA controller (RAID-1 for OS installation). We broke the RAID-1 on the Kingston drives, and swapped a Kingston with a Samsung. That put one Samsung on the motherboard's SATA controller and one Kingston on the LSI.

Our test has been dramatically simplified, and we still see the same degradation on the Samsung drives only (even on the motherboard's SATA controller). The Kingston drive does not exhibit this behavior.

This is a super-simple test, with no RAID involved. We format the drive, create a single big file (about 100GB, at least 2x RAM size), then continuously re-read the file in a tight loop:
Code: 
while [ 1 ] ; do dd bs=64K if=/path/to/bigfile of=/dev/null ; done
While the above is running, we run iostat continuously, logging to a file:
Code: 
iostat -tmx 300 > /path/to/logfile.txt 2>&1 &
What we see is that iostat initially shows the read performance at 500 MB/s (or greater). This is right in line with the manufacturer's advertised specs. But, after some amount of time, this read performance degrades to 30-50 MB/s---literally 1/10 the rated spec. It appears this happens after a little over 5 TB has cumulatively been read from the SSD. Performance will stay in this abysmal state for a while, then recover back to 500 MB/s. Once in the "good" state, it lasts only as long as roughly 5 TB has been read, then back to the "bad" state again.

We've run this test numerous times now, it's easily repeatable. We've repeated it on CentOS 6.7 and now CentOS 7. We're planning to try FreeBSD and Windows today, but I'd be quite surprised if the result is any different.

Edit: I should add, when we mount the partition to do the read test, we use "ro,noatime" options. This should prevent any writes from taking place. And if the SMART attribute Total_LBAs_Written is true, then we are in fact not writing any data to the drive (i.e. that SMART attribute is unchanged before and after the test).
 
Last edited:
  • Like
Reactions: Lance Joseph, Chuntzu and Chuckleb
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
Have you tried this with other drive reads than samsung?
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
Sorry, I meant any other drives than those 2 that had the identical slow down at 5TB?
 
M

matt_garman

Active Member
Feb 7, 2011
212
41
28
There are 8 total identical Samsung 850 EVO 2TB drives in the system, and two Kingston 128 GB SSDs. One Kingston is used for the OS. The other Kingston has been lumped in with the eight Samsung SSDs for comparison purposes. Both the Kingston and a Samsung have been tried on both the LSI controller and the motherboard's SATA controller (Intel C612 chipset). All eight Samsung drives exhibit the same phenomenon, regardless of being attached to the Intel SATA controller or the LSI controller. This happens in both hardware and software RAID mode, and also with the controller in JBOD mode, using drives individually.

We haven't tested any other SSDs at this point, but are going to order a Samsung 850 PRO to see if it has the same behavior.
 
Lance Joseph

Lance Joseph

Member
Oct 5, 2014
82
40
18
Quick update #2: The Wear_Leveling_Count went from 11 up to 12. Total_LBAs_Written have remained the same.
The CRC_Error_Count has also stayed at 3 which suggest that the cable was probably jostled previously.
These workloads are being generated on a system running Centos 7 that doesn't do anything else.

@matt_garman The drive is connected to a Dell T5600 motherboard. Dual E5-2620, 16G RAM & an Intel C600 3Gb/s SATA.
When checking in on it today, I noticed that a pair of tests had run for 23 minutes each rather than the usual 15 minutes 13-16 seconds.
I'm not sure why that happened, but I like your idea of using iostat to track counters. I think I'll start logging this and spit out some graphs.
I'll also run my tests with it connected to a LSI SAS3008-8i card and see if I note any performance drops over an extended period of long reads.

Funny you should mention the 2T Samsung Evo 850 drives. I just bought 42 of those to be spread across three systems.
Each system has 3x LSI SAS3008-8i cards. They originally came with IR firmware however I reflashed one system to IT mode (P10).
The 14 drive RAID-0 that I benched between two systems showed the same results, regardless of whether I was using IR or IT firmware.
I think this may suggest that until reasonably priced 12G SATA drives hit the market, it shouldn't matter which firmware is used with 6Gb drives.

Quick edit: @matt_garman try enabling LSI Fastpath to bypass the RAID controllers onboard cache: https://forums.servethehome.com/ind...y-what-does-it-do-and-how-does-it-do-it.1492/
 
Last edited:
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,640
2,057
113
Are you still mainly using these in read-only SQL Database setup? (MS SQL? MySQL? Or What?)
 
Lance Joseph

Lance Joseph

Member
Oct 5, 2014
82
40
18
T_Minus said:
Are you still mainly using these in read-only SQL Database setup? (MS SQL? MySQL? Or What?)
Click to expand...
@T_Minus: That's correct. Each system has 14x 850 Evo drives for read-only MS SQL database workloads. Additionally it has 4x 1.2T Intel 750 NVMe for SQL tempdb workloads. 2x Intel S3510 SSDs for a mirrored root. Brings the total to 20 drives with room to add 4 more down the road. We're using the Supermicro 2028U-TNRT+ chassis.

@matt_garman: Is it possible the drives are getting too hot and throttling performance back until the heat dissipates? Perhaps it would be worth dumping SMART data (using the 'megaraid' flags) over the course of the five hours. If the LSI adapter isn't cool enough, it may also be throttling. You can also get card's temperature data from the LSI MegaRAID utility.
 
You must log in or register to reply here.
Top Bottom