HP DL380 G6 P410i SSD issue

[Andyreas]

Hi again all

I've given up trying to find the answer via google so I really hope someone here knows. I've updated my HP DL380 G6 (8 bay SFF standard stuff) with the embedded P410i 512MB BBWC. I've both updated the P410i and the HP SAS Backplane. When I attach a Samsung 256GB Pro SSD it only connects at 1.5 Gbps. So ofcourse i tried to find what the maximum Connection speed was but some say 6 Gbps for SAS and 3 Gbps for SATA and that is pretty much where I ran in to the wall. If that is true, why does the SSD connect at 1.5? And if 1.5 SATA 3 SAS is the correct speed....dammit. Any easy way to upgrade the backplane cause I guess that is what is the bottleneck then?

Can anyone help me with this rather simply but strangely hard question.

 

[Andyreas]

I'd be so grateful if someone could explain this to me. If I am limited to 1.5 Gbps on normal SATA disks I wonder if the Array would perform better by dropping the SSD and going for the SAS mechanical harddrives instead. Sure accesstime is faster for SSD but not sure it is Worth it. SAS SSD seems out of my budget i Believe.

 

[Andyreas]

This feels like a weird conversation with myself. I updated the firmware of backplane and controllercard yesterday, no solution. But today I pull the powercoords to the server for a minute and rebooted and suddenly it connects at 3 Gbps. So problem solved. Now I only need to decide on going with 1TB SSD (M500) or a combo of ssd and hdd in storage spaces....

 

[Jeggs101]

Can/ did you set speeds manually? Sometime you get lower autonegotiation speeds and need to set manually.

 

[Andyreas]

I actually looked for that setting, both by logging in via the "rom"based ACU and then by installing the Array config utility in win 2012 r2 but I couldnt adjust it anywhere. Just deleted the Array and added 2 256 GB samsung Pros in raid0 (only to check performance) and it scales good now at 3 Gbps, doing 450 MB/s writes. I am not sure how to set the read/write ratio when i turn on the controller cards 512MB cache thou. I guess turning it on only will do good but benchmarking after doing so is pretty hard.

Now let's hope Patrick gets his performancescript to autoload the results somewhere before I continue with the rest of my windows installation.

 

[mrkrad]

You could always throw a P420 or LSI 9266 in that DL380. I've got both and the P420/1gb FBWC was super cheap ($269) new but very tempermental due to heat issues (hence why the P430/2gb FBWC is out now!). The LSI 9266 is also a good match!

 

[Andyreas]

Thank you for the response mrkrad. Wouldnt the backplane still be the bottleneck in that situation or am I misunderstanding that? Is it the p410i that is the current 3 Gbps-limit?

 

[Lost-Benji]

I personally would have thrown the SSD at a dedicated HBA or on-board port capable of AHCI and even SATA-III, leave the baby RAID for HDD's.

 

[mrkrad]

You know I have a DL380 G7, but my 9266-8i runs the SSD at 6gbps PHY rate without errors, so unless your backplane(s) are poorer quality, I think it is the controller.

Ideally with the extra riser, and extra 8SFF cage, you could run 4 9266-4i controllers to 16 drives in one of these beasts - not cheap but ideal. I've got 8 per 9266-8i, plus a couple of 10gbe nic's in my dl380 G7.

 

[Andyreas]

Lost-Benji, thanks for the suggestion but I had the budget to fill it with M500 960GB SSDs so really had to use the harddrivebays. They scale quite well so far thou. I know I will be limited by iops soon thou.

mrkrad, thanks again. I really need to read up on the technical stuff regarding the backplane and since it isnt an expander or anything similar it might just be an "extension" of the electrical wires, so no bottleneck at all. I really gotta try that. Decent price on the P420s too. My former/current setup was the HP DL380 G7 (thou only 4 core cpus in that one) but that will surely get the upgrade since it doesnt even have a BBU today. (Which I got seriously aware of when the raid5 Array running my esxi crashed cause of me being stupid (managed to recover it thou which feelt rather great and didnt cost me the expected 20k$)).

Regarding the riser and an extra 8SFF Cage I'd doubt I will run in to issues regarding speed or size of the storage Before i run into it with the cpu and/or RAM. My current DL380G7 runs 2 E5620, 100GB Ram at both 90-95% now but the storage is no where near it's limits and with 8 M500 1TB in Raid10 (decided to go that route instead of storage spaces) they should swallow what I throw at them. I hope.

 

[mrkrad]

let me know how those M500's work out. I've heard little of using them in raid! Are you doing heavy overprovisioning yourself?

Sometimes having the PHY rate run slower than maximum can be a blessing

 

[Patrick]

The M500s are not fast, but the 480GB and 960GB models have a decent amount of built-in OP, MLC, more RAM and supercaps.

Thus far I probably have 15-20 of them and will be getting another 480GB or 960GB model today. The 480GB model is within $1 of its lowest price on Amazon right now at $261 of $0.54/ usable GB. The 960GB model is down to $455 or $0.47/ usable GB. RAID 1 wise the 960GB model has been fine for some time.

Edit: Just got the 480GB model. $257 from a fulfilled by Amazon seller that give Prime and no tax in CA!

 

[Lost-Benji]

Lost-Benji, thanks for the suggestion but I had the budget to fill it with M500 960GB SSDs so really had to use the harddrivebays. They scale quite well so far thou. I know I will be limited by iops soon thou.

Now I see, first up, I would like to suggest maybe a decent A/M RAID card that is designed for the application. The other area that may come up is to do with the fact you are using SATA in a system designed for SAS. Some might be asking WTF I make this suggestion but when you look at the basic differences, it soon becomes clear.

SATA signalling is rather low when compared with SAS. You have cables and connections that are all adding losses that would normally be fine with SAS but might now be pushing your luck with SATA. This would end up seeing drives negotiate at slower speeds when the controllers are trying to compensate for losses.

There is also quite a few items online from people looking for "SSD Firmware" for the P410. This would lead me to believe that there are others who are finding the limits of a small card, yours being the embedded limits things even more.

 

[Andyreas]

Thanks for the feedback again guys. As Patrick said the M500 OP pretty well by themselves but I'll probably add a little bit more just in case. The speed of these are good but not great but considering it's prize compared to other 1TB solutions and the fact that I am limited by the 3 gbps speed of the controller I don't find that as an issue.

As I said before I setup 2 Samsung 256 GB Pro in Raid0 and 1 M500 and did alot of benchmarking. Everything went great, no dips in performance however I tested it. I did it without using the 512MB cache as write cache, without using the drives own built in cache etc etc. The more options I turned on the slightly better it performed. I had the controllers cards cache set to 25% read 75% write. And now to the problem.

I reinstalled Win2012R2DC on the samsungs but this time setup as Raid1. I still had the M500 as "Raid 0". Writes from the M500 to the Samsungs tags along great first 7-800MB then it crashed down to ridiculous speeds. I tried turning of the internal cache of the drives, turning on cache of the controller etc etc. It doesnt change. I probably found some something with the Samsung drives and the controller that doesnt work well together.

Next week I'll dump the Raid1 and add 4 M500 as Raid10 and redo all the tests. That will be interesting to see how that works.

 

[mrkrad]

I found that disabling BBWC by setting to 100% READ,0% WRITE, enable DRIVE WRITE CACHE (MUST!), the main point is to tweak the STRIP size (16K p/E crucial m500, 8kb samsung 840 pro) - the default strip size is ridiculous and would require severe overwriting to the SSD!

Samsung drives require 30% OP to function well sustained.

I'd check the strip size - you'll find it is far greater than the recommended size! I've seen 8 drives 128KB strip size - every byte written would induce 8 drives to write 4 * 128KB in raid-10 !!

I think the minimum size might be a good place to test and benchmark, unless you are going after full linear bandwidth. I'm more interested in random i/o latency!

 

[Andyreas]

Thanks mrkrad! You are spot on, strip size is, not sure I dare to tell, 256 kb. How do I fifgure out the optimal size, ie why is it 16k on the m500 and 8kb on the samsung? Any way of calculating? Time for a reinstall =)

 

[Andyreas]

Sumup of some benchmarks

Hey all, i promised to give some feedback after i received the M500s and here it goes, a rather lengthy post and can't really understand some of the numbers, they make Little sense to me.

• Server: HP DL380 G6
• Controller-card: P410i (internal) 512MB BBU connect to HP standard backplane
• C-drive: 2 x Samsung Pro 256 GB in Raid0 (stripe 256KB) connected at 3Gbps
• D-drive: 4 x M500 960GB in Raid10 (stripe 32/64KB) connected at 3Gbps
• Large folder copy consists off: 6 files total size 5,87 GB
• Small folder copy consists off: 25947 files total size 3,25 GB
• All benchmarks done with default settings, times in seconds and done on a Win Server 2012R2 DC Clean install with latest SSP from HP from Februari 2014

I did three different config of caches:

1 - Controller cache (512MB) 25%Read/75% Write, SSD internal write cache on
2 - Controller cache (512MB) Off, SSD internal write cache on
3 - Controller cache (512MB) Off, SSD internal write cache Off

1: Largefilecopy C->D 28
2: Largefilecopy C->D 27
3: Largefilecopy C->D 19

1: Largefilecopy D->C 1,16
2: Largefilecopy D->C 1,41
3: Largefilecopy D->C 44

1: Largefilecopy C->C 1,46
2: Largefilecopy C->C 2,19
3: Largefilecopy C->C 1,58

1: Largefilecopy D->D 24
2: Largefilecopy D->D 24
3: Largefilecopy D->D 26

1: Smallfilecopy C->D 27
2: Smallfilecopy C->D 40
3: Smallfilecopy C->D 56

1: Smallfilecopy D->C 1,01
2: Smallfilecopy D->C 46
3: Smallfilecopy D->C 54

1: Smallfilecopy D->D 27
2: Smallfilecopy D->D 32
3: Smallfilecopy D->D 51

[HR][/HR]On top of these manual tests i did some CrystalDiskMark benchmarks

1: Cdrive
Sequential Read : 547.559 MB/s
Sequential Write : 181.902 MB/s
Random Read 512KB : 518.181 MB/s
Random Write 512KB : 239.252 MB/s
Random Read 4KB (QD=1) : 29.472 MB/s [ 7195.2 IOPS]
Random Write 4KB (QD=1) : 56.124 MB/s [ 13702.1 IOPS]
Random Read 4KB (QD=32) : 196.395 MB/s [ 47947.9 IOPS]
Random Write 4KB (QD=32) : 85.730 MB/s [ 20930.1 IOPS]
Test : 1000 MB [C: 7.5% (36.0/476.5 GB)] (x5)

2: Cdrive
Sequential Read : 443.748 MB/s
Sequential Write : 180.633 MB/s
Random Read 512KB : 412.743 MB/s
Random Write 512KB : 172.690 MB/s
Random Read 4KB (QD=1) : 22.903 MB/s [ 5591.6 IOPS]
Random Write 4KB (QD=1) : 33.757 MB/s [ 8241.4 IOPS]
Random Read 4KB (QD=32) : 206.378 MB/s [ 50385.1 IOPS]
Random Write 4KB (QD=32) : 188.713 MB/s [ 46072.5 IOPS]
Test : 1000 MB [C: 8.8% (41.8/476.5 GB)] (x5)

3: Cdrive
Sequential Read : 442.998 MB/s
Sequential Write : 127.719 MB/s
Random Read 512KB : 410.077 MB/s
Random Write 512KB : 80.137 MB/s
Random Read 4KB (QD=1) : 22.071 MB/s [ 5388.5 IOPS]
Random Write 4KB (QD=1) : 2.283 MB/s [ 557.5 IOPS]
Random Read 4KB (QD=32) : 206.455 MB/s [ 50404.1 IOPS]
Random Write 4KB (QD=32) : 20.871 MB/s [ 5095.5 IOPS]
Test : 1000 MB [C: 8.8% (41.9/476.5 GB)] (x5)

1: Ddrive
Sequential Read : 784.862 MB/s
Sequential Write : 418.037 MB/s
Random Read 512KB : 688.133 MB/s
Random Write 512KB : 565.760 MB/s
Random Read 4KB (QD=1) : 22.569 MB/s [ 5509.9 IOPS]
Random Write 4KB (QD=1) : 42.613 MB/s [ 10403.5 IOPS]
Random Read 4KB (QD=32) : 170.178 MB/s [ 41547.4 IOPS]
Random Write 4KB (QD=32) : 51.680 MB/s [ 12617.1 IOPS]
Test : 1000 MB [D: 1.3% (23.7/1788.4 GB)] (x5)

2: Ddrive
Sequential Read : 699.891 MB/s
Sequential Write : 337.923 MB/s
Random Read 512KB : 599.136 MB/s
Random Write 512KB : 336.283 MB/s
Random Read 4KB (QD=1) : 18.135 MB/s [ 4427.5 IOPS]
Random Write 4KB (QD=1) : 28.773 MB/s [ 7024.8 IOPS]
Random Read 4KB (QD=32) : 181.460 MB/s [ 44301.8 IOPS]
Random Write 4KB (QD=32) : 74.721 MB/s [ 18242.4 IOPS]
Test : 1000 MB [D: 2.4% (42.1/1788.4 GB)] (x5)

3: Ddrive
Sequential Read : 698.120 MB/s
Sequential Write : 184.918 MB/s
Random Read 512KB : 599.984 MB/s
Random Write 512KB : 133.169 MB/s
Random Read 4KB (QD=1) : 18.153 MB/s [ 4431.9 IOPS]
Random Write 4KB (QD=1) : 1.714 MB/s [ 418.6 IOPS]
Random Read 4KB (QD=32) : 182.963 MB/s [ 44668.6 IOPS]
Random Write 4KB (QD=32) : 3.983 MB/s [ 972.3 IOPS]
Test : 1000 MB [D: 2.4% (42.1/1788.4 GB)] (x5)

[HR][/HR]And then some screendumps from Atto

1: Cdrive

2: Cdrive

3: Cdrive

1: Ddrive

2: Ddrive

3: Ddrive

[HR][/HR]And then some Read screendumps from HD Tune

1: Cdrive

2: Cdrive

3: Cdrive

1: Ddrive

2: Ddrive

3: Ddrive

[HR][/HR]And my conclusion of all this?...comes later since I have to run right now.

 

[dba]

The default settings on most (if not all) SSD benchmarking tools will give you less than useful results. ATTO, for example, was reading and writing in just one 256MB area of the array in your tests. To get a better picture of real performance, increase this significantly, say to 10GB or even larger.

 

[mrkrad]

Seems like the tests are hitting the BBWC cache - as DBA said you need to increase the size!

I'd play around with strip size, 256K is way too big for the samsung 840 pro's.

But overall for that generation of raid controller (they are up to P430/2gb FBWC now!) it seems inline. The P420 has enhanced "fast path" like mode for SSD that bypasses all the cache to give you more speed! not sure if they have that for the p410 !

I did not get much better performance between the P420/1gb FBWC and the LSi 9260-8i honestly! even with the M5014 (LSI 9260-8i oem IBM) the performance was nowhere near as good as a true intel SATA port on the motherboard!

 

[Andyreas]

Hey dba. Thanks for the feedback and I know they are not perfect for benchmarking. My main goals was to benchmark the individual setups to each other and not for correct real World speed.

2GB seems to be the largest ATTO does but there really wasnt much difference in the charts.

Mrkrad, thanks again, this feedback means alot. The only reason for 256K was that the CMD/DOS program couldnt adjust it when i set it up. I only have the basic Win2012R2 install on it.

But I am pretty sure now that the only way to get any real performance improvements is to switch the controller card for a better one.

The main thing that bugs me thou is that I dont seem to get any performance improvements when I use the 512MB cache (for writes) on the controller card. That bugs me some actually.

To give some more info regarding benchmarks when I am doing all of these tests anyway I am adding some numbers regarding storages spaces below. Bear in mind this is Win 2012 R2. Then disk were setup as single raid0 to the OS and I hade the same settings as I did in setup 1 above (75% write/25% Read on Raid Card and write cache enabled on disks).

So first screendump, Storage Spaces, 4 M500 960TB in Mirror config (default settings):

Then in Simple config:

And last in parity:

And since mirror is the setup i will use some figures from CrystalD comparing Raid10 to SS

Crystal D on Raid10
Sequential Read : 784.862 MB/s
Sequential Write : 418.037 MB/s
Random Read 512KB : 688.133 MB/s
Random Write 512KB : 565.760 MB/s
Random Read 4KB (QD=1) : 22.569 MB/s [ 5509.9 IOPS]
Random Write 4KB (QD=1) : 42.613 MB/s [ 10403.5 IOPS]
Random Read 4KB (QD=32) : 170.178 MB/s [ 41547.4 IOPS]
Random Write 4KB (QD=32) : 51.680 MB/s [ 12617.1 IOPS]
Test : 1000 MB [D: 1.3% (23.7/1788.4 GB)] (x5)

CrystalD on Storage Spaces Mirror
Sequential Read : 488.315 MB/s
Sequential Write : 428.223 MB/s
Random Read 512KB : 441.561 MB/s
Random Write 512KB : 479.638 MB/s
Random Read 4KB (QD=1) : 19.592 MB/s [ 4783.3 IOPS]
Random Write 4KB (QD=1) : 32.915 MB/s [ 8035.9 IOPS]
Random Read 4KB (QD=32) : 187.476 MB/s [ 45770.5 IOPS]
Random Write 4KB (QD=32) : 41.221 MB/s [ 10063.8 IOPS]
Test : 1000 MB [D: 0.0% (0.2/1785.9 GB)] (x5)