Benchmrks! : Adaptec 8405 -> RES3TV360 Expander + benchmarks/info

[james23]

Hey,

So i just got my adaptec 8405 in a week ago and hooked it to my Intel res3tv360 36pt expander. all is working great! the maxview adaptec software sees all the correct info from the expander (its make, model, firmware, even temp!, correct port #s)...there are ALOT of complaints about maxview software around the web, but so far ive found it to be pretty good.

i'll post some quick, initial bench results below, but my main question is: What is my max ceiling speed wise due to port speed limitation?.
ie both are 12/gbs sff8643 ports (each sff8643 technically is 4x 12/gbs channels) but the speed of my actual drives is 3.0gb/s (sataI) so does that lock all the ports (on the expander and thus also to the 8405 HBA) to 3.0gb/s and thus my MAX is 4x 3.0gb/s (1200mb/s max theoretical).


I even hooked my older intel res2tv240 24 pt sff-8087 expander, into a port of the res3tv360 and that worked great too (was able to see a drive hooked into the res2tv240 as well as the expander itself in maxview)

btw: the only reason i had to ditch LSI cards is bc they do not allow you to create multiple raid types PER DISK, ie have a set of 5 disks - half of space is a 5x raid6 and the other half of the space is a 5x raid0. All raid cards, EXCEPT LSI allow you to do this, adaptec sales calls this feature:
"Optimized Disk Utilization (Multiple Arrays per disk)"

NOTE: one thing to be aware of, i noticed that on the expander, you MUST hook the sff8643->4x sata fan-out cables in the correct order (in the expanders manual there is a chart, that has A,B,C,D,E,F.... next to each of the expanders sff8643 ports). if you dont hook the drives up in the proper order, they will not show up to the HBA (the adaptec 8405 in my case)... luckily i figured this out very early on.

benches: (crystal mark 4.01 , win 2008r2 fresh install) - note all of these drives below are pretty old, both sets over 5 years of power on hours - ill post more benches when i grab my centos 6 USB stick on which i have tools strictly for benchmarks.

5x 150gb WD Velociraptor drives, Raid6 1024 stripe size:
Sequential Read (Q= 32,T= 1) : 434.986 MB/s
Sequential Write (Q= 32,T= 1) : 361.270 MB/s
Random Read 4KiB (Q= 32,T= 1) : 7.726 MB/s [ 1886.2 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 2.217 MB/s [ 541.3 IOPS]
Sequential Read (T= 1) : 412.759 MB/s
Sequential Write (T= 1) : 366.595 MB/s
Random Read 4KiB (Q= 1,T= 1) : 1.167 MB/s [ 284.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 1.998 MB/s [ 487.8 IOPS]
Test : 8192 MiB [G: 20.7% (10.4/50.0 GiB)] (x2)


12x 2TB Samsung F4 drives RD0/1024 stripe:
Sequential Read (Q= 32,T= 1) : 623.417 MB/s
Sequential Write (Q= 32,T= 1) : 950.166 MB/s
Random Read 4KiB (Q= 32,T= 1) : 5.247 MB/s [ 1281.0 IOPS]
Random Write 4KiB (Q= 32,T= 1) : 10.488 MB/s [ 2560.5 IOPS]
Sequential Read (T= 1) : 598.603 MB/s
Sequential Write (T= 1) : 957.282 MB/s
Random Read 4KiB (Q= 1,T= 1) : 1.216 MB/s [ 296.9 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 8.480 MB/s [ 2070.3 IOPS]
Test : 32768 MiB [H: 0.0% (0.4/22319.9 GiB)] (x2)


6x 2TB Samsung 8405 drives RD0/1024 stripe:
Sequential Read (T= 1) : 651.200 MB/s
Sequential Write (T= 1) : 776.677 MB/s
Test : 8192 MiB [M: 0.0% (0.4/11159.9 GiB)] (x2


6x 2TB Samsung 8405 drives RD0/256k stripe:
Sequential Read (T= 1) : 466.419 MB/s
Sequential Write (T= 1) : 739.759 MB/s
Test : 8192 MiB [K: 0.0% (0.4/11159.9 GiB)] (x2)

 

[Chuntzu]

I have been curious about these SAS expanders. How well do they work using SAS/data 6.0gbps and pooling their speeds? They supposedly can use all available bandwidth from 12gbps controllers with 6.0gbps drives.

 

[james23]

I was wondering about that, when i made my initial post above, i only had 3.0gbps drives connected to it (and they showed up as 3.0gbps on card and expander), today i was able to connect 8x 3tb hitachi HUA drives which are 6.0gbps, and the expander and the card both show the ports at 12.0gbps!

the performance is AMAZING! check out these inital, quick benchmarks:

8x hitachi 3tb "HUA723030ALA640" drives in raid6 , 1024 stripe size, adaptec 8405 to intel expander (see my first post for more info): note these drives get 140-160 MB/s sequential when i benchmark one of them connected directly to my motherboard. (so im maxing all 8 drives!)

raid 6:

-----------------------------------------------------------------------

CrystalDiskMark 4.0.1 x64 (C) 2007-2015 hiyohiyo

Crystal Dew World : Crystal Dew World

-----------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes


Sequential Read (Q= 32,T= 1) :  1104.555 MB/s

Sequential Write (Q= 32,T= 1) :  930.888 MB/s

Random Read 4KiB (Q= 32,T= 1) :  7.936 MB/s [  1937.5 IOPS]

Random Write 4KiB (Q= 32,T= 1) :  2.549 MB/s [  622.3 IOPS]

Sequential Read (T= 1) :  1087.935 MB/s

Sequential Write (T= 1) :  931.959 MB/s

Random Read 4KiB (Q= 1,T= 1) :  0.738 MB/s [  180.2 IOPS]

Random Write 4KiB (Q= 1,T= 1) :  1.805 MB/s [  440.7 IOPS]


Test : 32768 MiB [L: 0.0% (0.1/499.9 GiB)] (x2)

Date : 2015/05/07 20:31:38

OS : Windows Server 2008 R2 Enterprise (Full installation) SP1 [6.1 Build 7601] (x64)

raid 0 (1024k stripe, 8x same drives) :

-----------------------------------------------------------------------

CrystalDiskMark 4.0.1 x64 (C) 2007-2015 hiyohiyo

Crystal Dew World : Crystal Dew World

-----------------------------------------------------------------------

* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]

* KB = 1000 bytes, KiB = 1024 bytes


Sequential Read (Q= 32,T= 1) :  1227.825 MB/s

Sequential Write (Q= 32,T= 1) :  1259.811 MB/s

Random Read 4KiB (Q= 32,T= 1) :  7.965 MB/s [  1944.6 IOPS]

Random Write 4KiB (Q= 32,T= 1) :  11.862 MB/s [  2896.0 IOPS]

Sequential Read (T= 1) :  1225.205 MB/s

Sequential Write (T= 1) :  1268.702 MB/s

Random Read 4KiB (Q= 1,T= 1) :  0.712 MB/s [  173.8 IOPS]

Random Write 4KiB (Q= 1,T= 1) :  12.115 MB/s [  2957.8 IOPS]


Test : 32768 MiB [P: 0.0% (0.1/1023.9 GiB)] (x2)

Date : 2015/05/07 19:48:32

OS : Windows Server 2008 R2 Enterprise (Full installation) SP1 [6.1 Build 7601] (x64)

 

[james23]

Hey, just to update, i can now confirm that the 8405 -> RES3TV360 does allow multiple different sata speeds on the same expander / same RAID card port. (some one asked about this above). the 8405 only has one sas-HD port, in my case going to the expander.

What i mean is it shows my 12x Samsung F4 drives at 3.0/gbs and my 8x hitachi HUA drives at 12.0GB/s (they are SATA 6.0GB/s but i read that the 12.0 protocol allows devices at 6.0 speed to connect at 12.0 so it is functioning properly, even though the max speed per sata port really is really 6.0 on those showing 12.0).

I def see the port speed difference in my benchmarks, as even in a RAID 6 array, the 8x HUA drives max out each drives max sequential R/W. i.e each of those HUA drives alone does ~140 MB/s R/W sequential, so when i have 8x of them in a raid6 (or even raid0 for comparison) array, im seeing 8x 140MB/s (1200-1300 MB/s RW) on the raid6 array sequential benchmarks!.

whereas on the 12x samsung F4 each drives sequential is ~120 MB/s yet im only seeing 1000-1200 mb/s R/W on a 12x raid0 array of those. so it seems the sata 3.0 vs sata 6.0 connection really helps.

note; the Hitachi HUA drives are 7200rpm, the Samsung F4s im running are 5400rpm (and F4s are over 5 years old of 24/7).

Im still super impressed with this 8405 and RES3TV360 expander combo! if you recall from my other posts ive gone trough several Adaptec and LSI raid controllers and have now decided/choose this 8405 (it, along with a adaptec 51645 and 5405z running on this same machine... lol i have 50+ HDDs running on this one 4u supermicro server )