Slow Samsung 850 Pro performance

[PnoT]

I seem to be having slow performance on my SSDs in my SM chassis and am unable to pinpoint the source of the problem. I'm having severe performance issues with my SSDs in the sub 200MB/sec range for the entire array but today those numbers have increased a bit but are still below what I would expect from a RAID 0 of all SSDs with this setup. I've pieced together a bunch of information and have tried my best to supply everything I can think of in this post to describe my issue in the hopes that someone can help me diagnose the problem.

One thing I couldn't find is a reliable way to determine link speed on each drive as most of the commands I found, on the net, came back with "<unknown>".

Current Setup

X8SIL-F
Xeon 3450
16GB ECC RAM
SuperMicro SC846 Chassis
SAS2 backplane
IBM M1015 flashed to an LSI 9211-8i in IT mode and running R19 firmware.
Single cable from M1015 P0 to PRI_J0 on the backplane
5592.2 Update 3

Drive / RAID layout

8 x 4TB WD RED + 2 x 5TB WD RED in SHR
4 x 256GB Samsung 850 Pro in RAID 0

Drive Info:

/dev/sdq:

ATA device, with non-removable media
        Model Number:       Samsung SSD 850 PRO 256GB
        Serial Number:      S1SUNSAFC81422B
        Firmware Revision:  EXM02B6Q
        Transport:          Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Used: unknown (minor revision code 0x0039)
        Supported: 9 8 7 6 5
        Likely used: 9
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors:  500118192
        Logical  Sector size:                   512 bytes
        Physical Sector size:                   512 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:      244198 MBytes
        device size with M = 1000*1000:      256060 MBytes (256 GB)
        cache/buffer size  = unknown
        Nominal Media Rotation Rate: Solid State Device
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 1   Current = 1
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
                SET_MAX security extension
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    64-bit World wide name
                Write-Read-Verify feature set
           *    WRITE_UNCORRECTABLE_EXT command
           *    {READ,WRITE}_DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Gen3 signaling speed (6.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Phy event counters
           *    unknown 76[15]
           *    DMA Setup Auto-Activate optimization
                Device-initiated interface power management
           *    Asynchronous notification (eg. media change)
           *    Software settings preservation
                unknown 78[8]
           *    SMART Command Transport (SCT) feature set
           *    SCT LBA Segment Access (AC2)
           *    SCT Error Recovery Control (AC3)
           *    SCT Features Control (AC4)
           *    SCT Data Tables (AC5)
           *    reserved 69[4]
           *    DOWNLOAD MICROCODE DMA command
           *    SET MAX SETPASSWORD/UNLOCK DMA commands
           *    WRITE BUFFER DMA command
           *    READ BUFFER DMA command
           *    Data Set Management TRIM supported (limit 8 blocks)
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
                supported: enhanced erase
        2min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50025388a08df889
        NAA             : 5
        IEEE OUI        : 002538
        Unique ID       : 8a08df889
Checksum: correct

{ /volume3}-> dmesg | grep "Write cache"
[    8.714585] sd 0:0:15:0: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.714698] sd 0:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.715425] sd 0:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.716042] sd 0:0:10:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.716334] sd 0:0:12:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.716425] sd 0:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.716564] sd 0:0:14:0: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.769484] sd 0:0:13:0: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.772838] sd 0:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.773529] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.773881] sd 0:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.775287] sd 0:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.778207] sd 0:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.778531] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.779942] sd 0:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[    8.790654] sd 0:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[   66.494905] sd 7:0:0:0: [synoboot] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[236816.271293] sd 0:0:17:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA
[236830.021295] sd 0:0:18:0: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA
[236838.521456] sd 0:0:19:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA
[236927.771667] sd 0:0:20:0: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA

The hdparm results per drive and finally the overall array look right on the money but why are tests with dd so horrible

Disk /dev/sdq: 256GB
Disk /dev/sdr: 256GB
Disk /dev/sds: 256GB
Disk /dev/sdt: 256GB

hdparm -tT --direct /dev/sdr
/dev/sdr:
Timing O_DIRECT cached reads:   946 MB in  2.00 seconds = 472.71 MB/sec
Timing O_DIRECT disk reads: 1468 MB in  3.00 seconds = 489.13 MB/sec

hdparm -tT --direct /dev/sds
/dev/sds:
Timing O_DIRECT cached reads:   966 MB in  2.00 seconds = 482.10 MB/sec
Timing O_DIRECT disk reads: 1476 MB in  3.00 seconds = 491.79 MB/sec

hdparm -tT --direct /dev/sdt
/dev/sdt:
Timing O_DIRECT cached reads:   962 MB in  2.00 seconds = 480.54 MB/sec
Timing O_DIRECT disk reads: 1464 MB in  3.00 seconds = 487.95 MB/sec

hdparm -tT --direct /dev/sdq
/dev/sdq:
Timing O_DIRECT cached reads:   964 MB in  2.00 seconds = 481.62 MB/sec
Timing O_DIRECT disk reads: 1466 MB in  3.00 seconds = 488.58 MB/sec

hdparm -tT --direct /dev/vg3/volume_3
/dev/vg3/volume_3:
Timing O_DIRECT cached reads:   2880 MB in  2.00 seconds = 1439.44 MB/sec
Timing O_DIRECT disk reads: 4570 MB in  3.00 seconds = 1523.11 MB/sec

Here is the query on my LSI controller to determine link speed which look like 8x:

lspci -vvv -d 1000:0072
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

If you look at the performance of volume2 which consists of NAS REDS (SHR) it's not bad when using dd and the speeds are very consistent. Keep in mind that this image only shows 7 / 9 drives in the array due to the limitation of the resource monitor tool so if you add another 2 drives @ 90MB/sec that's well over 1GB/sec.

dd if=/dev/zero of=/volume2/test.bin bs=1M count=500M

The same command on the RAID0 Samsung 850 Pros nets some pretty crappy results. If you look closely you can see huge swings in performance from 50MB/sec to almost 200MB/sec per drive which is completely the opposite of how the SHR and spinning disks are performing.

dd if=/dev/zero of=/volume3/test.bin bs=1M count=500M

 

[RTM]

You are not the only one who have had problems with that SSD and LSI controllers, I found this thread, that covers the same problem:
https://forums.servethehome.com/ind...ung-850-pros-not-lsi-9361-8i-compatible.5804/

 

[EffrafaxOfWug]

Been a while since we had that combo available, but we had an issue where samsung drive would run like superglue off a shovel if the LSI cache wasn't turned off. IIRC there's a setting to turn the cache to write-through which improves the performance some...? I don't use the option ROM or the LSI tools myself (suspect you might be in the same boat as me) so not easy for me to test at the moment, and I think the last 830 Pro drive (which also didn't get on with the LSI) went into someone's laptop.

IIRC it spanned out from some incompatibility where the LSI thought the SSD didn't have a cache or something like that, and refused to let the drive use it. There was some combination of gubbins you could set in the option ROM or the CLI to get it working again but it was more bother than it was worth for us.

 

[PnoT]

It looks like most of these are using IR mode if I'm not mistaken and enabled write-cache but how does that work in IT mode since it's just passing drives through?

I found this interesting

Well there is no easy way to do that with the 9361. You have to create the Raid then uninstall the raid card + hook up the SSDs to the onboard SATA. You can enable the SSD write cache using the samsung tools now and you need to do it for every single SSD used in the Raid. After that, reinstall the 9361 with the SSDs. The 9361 thinks its disabled but its not.

Maybe it helps someone but I would not recommend it.

Anyone tried that through the tools as I can't seem to find that option in the Magician tool.

I also ran dd on just one of the drives in the array as requested by another member.

dd if=/dev/zero | pv | dd of=/dev/sdq bs=1M count=1000000
0+1000000 records in4MiB/s] [       <=>                                                                                                                                                                                ]
0+1000000 records out
1.26GiB 0:00:02 [ 544MiB/s] [           <=>

Looks legit

 

[wildchild]

Maybe a dumb question, but does performance improve after a reboot ?

 

[PnoT]

Maybe a dumb question, but does performance improve after a reboot ?

My initial findings were that the array was doing ~200MB/sec total but after a reboot last night and update 3 applied it has increased in which the screenshots above represent.

Is there something specific you have in mind or just wanted to make sure something wasn't going on with the OS and a reboot fixed it?

 

[wildchild]

Maybe a trim/op issue..

Depending on os that can have a dramatic decrease in performance

While i use zfs i know the lsi cards in general also have a big problem with trim.

Secure erase and op to 70% and test again... especially if performance goes down if time progresses

 

[Chuckleb]

So I have zfs, LSI, Samsung... Didn't over provision. Can I get the same effect by setting a zfs quota at 70%?

 

[wildchild]

Hi chuckleb, unfortunatly you cant to my knowledge.
Cant be any data on the empty part or firmware simply wont mark it as usable for the clean up routines.

I'm sure you can find plenty of weblinks confirming my findings.

Btw i also run lsi, zfs and a mixture of 840/850.. for both the pro model.

For zil i run op'ed intel 3700's

 

[T_Minus]

From everything I've read on WHT over the years I'd stay away from Samsung Home drives and any sort of continued usage... sure the drives may last a long long time but they suffer from latency issue, GC slow-down/catch-up, LSI issue, etc...

 

[PnoT]

Just before I tore these down from my SOFS build they were performing exceptionally well on an LSI with the proper settings and RAID 10. I'm hoping someone can shed some light on making these puppies perform.

 

[TechIsCool]

My Problem was with 840 Pros not 850 but I think the same issues are still relevant. Scroll to the bottom for a TL;DR

 

[wildchild]

Just before I tore these down from my SOFS build they were performing exceptionally well on an LSI with the proper settings and RAID 10. I'm hoping someone can shed some light on making these puppies perform.

Tested op/ secure erase yet ?

 

[PnoT]

My Problem was with 840 Pros not 850 but I think the same issues are still relevant. Scroll to the bottom for a TL;DR

Very informative article and I'm still working my way through it so thank you for the link.

Tested op/ secure erase yet ?

I found a couple of articles to get the job done this afternoon if everything goes as planned.

SSD Secure Erase - Thomas-Krenn-Wiki

SSD Over-provisioning using hdparm - Thomas-Krenn-Wiki

 

[wildchild]

Thomas krenb method i use my self.. works well... i think 70-80% op should be plenty

 

[wildchild]

Tried it yet ?

 

[PnoT]

Tried it yet ?

Unfortunately I was @ the data center all day yesterday so didn't get a chance to work on my issue.

 

[mephisto]

Hey guys,

I'm also having problems with the 850 PRO.

I have a dell R515 server with a PERC H700 (LSI) and I noticed now it is running damn slow for writes. I'm not sure if the "garbage" collection is rubbish (sorry, I learnt British English) or the drive simply is not copying. I have an exchange server running on ESXi and 2x 850 PRO 1TB in RAID1. The Exchange 2013 VM is about 450GB in size and it doesn't really generate much disk activity.

On the beginning it was running really fast for writes, about 450MB sequential. Now, it is about 50/60MB.

I'm been playing with disk cache and raid controller cache. With DISK and RAID controller cache disable, I get about 40MB, with write back enabled, still around 40MB sec, then enabling write back + disk cache I get pretty much the same result.

I'll move the Exchange server VM to another host and try to wipe these drives and re create the RAID array to see if there is any improvements, but I'm a bit disappointed.

I have a few other hosts running 845DC EVO 480GB and I noticed one host (Dell R710 with PERC H700) was also a bit slow, I think 150MB sec, and the server is just used for Citrix server, so no write intensive. I remember they being 400MB+ when new.

Next I'll have to test a RAID-5 array of 845DC PRO I have with 4x800GB and see the results, but so far I remember those are fine, they are running on a Dell R720 with PERC H710.

I'll post some more results later

 

[mephisto]

So I was testing the RAID-5 array of 4x 845DC PRO 800GB, they are performing veyr well, 1700MB read and 1200MB write sequential write back enabled.

The 850PRO is definitely behaving weirdly, 50MB/sec with any configuration thrown at it, very disappointing!

 

[mephisto]

So, I've deleted the RAID1 container on the PERC H700 controller, then recreated and initialised it. No changes, still about 40MB sec.

I'm planning now to take the disks out of the server and use on a normal workstation as I believe this is due to TRIM not available through the RAID controller.

The other 845DC EVOs and 845DC PROs I have are running perfectly fine, so I wonder if there is a firmware artificial limitation with garbage collection when the 850PRO is used in hardware RAID mode.