Need advice to improve DAS performance

[charlesnadeau]

I need some advice on how to improve the performance of my DAS.
I have a SansDigital 8-bay eSATA enclosure connected to my workstation based on a Supermicro H8DG6-F motherboard via a StarTech PEXESAT322I SATA 3 controller and the 2 eSATA cables that came with the enclosure. The enclosure is filled with a mix of Seagate and WD disks of various capacities. My operating system is Ubuntu 12.04.5. This machine is my main workstation and is also used as a Bacula storage director. Although the controller, most drives and enclosure are all SATA3 capable, my disks only work at SATA2 speed at best and the performance when used in a pair of 4 disks ZFS array is abysmal. Here is what Linux reports on the speed of the drives and on the controller:

charles@beijing:~$ dmesg | grep -i sata | grep 'link up'; lsscsi
[ 2533.647525] ata8.15: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 2533.967672] ata8.00: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
[ 2534.287689] ata8.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2534.607730] ata8.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[ 2534.927747] ata8.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17441.228498] ata7.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17441.548666] ata7.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[17441.868662] ata7.01: SATA link up 1.5 Gbps (SStatus 113 SControl 320)
[17442.188856] ata7.02: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[17442.508870] ata7.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17478.177844] ata7.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17478.525982] ata7.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[17479.050074] ata7.01: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[17479.370132] ata7.02: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[17479.690226] ata7.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17515.228560] ata7.15: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17515.547471] ata7.00: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[17515.871486] ata7.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17516.195469] ata7.02: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[17516.515546] ata7.03: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[17517.155562] ata7.05: SATA link up 1.5 Gbps (SStatus 113 SControl 320)
[0:0:0:0]    cd/dvd  TSSTcorp DVDWBD SH-B123L  SB03  /dev/sr0
[2:0:0:0]    disk    ATA      C300-CTFDDAC128M 0007  /dev/sda
[6:0:0:0]    disk    ATA      WDC WD20EARS-00M 50.0  /dev/sdk
[6:1:0:0]    disk    ATA      WD30EFRX-68AX9N0 80.0  /dev/sdl
[6:2:0:0]    disk    ATA      ST31000528AS     CC34  /dev/sdm
[6:3:0:0]    disk    ATA      WDC WD30EFRX-68A 80.0  /dev/sdn
[7:0:0:0]    disk    ATA      KINGSTON SV300S3 505A  /dev/sdo
[7:1:0:0]    disk    ATA      ST3000DM001-9YN1 CC4B  /dev/sdp
[7:2:0:0]    disk    ATA      ST3750330AS      SD15  /dev/sdq
[7:3:0:0]    disk    ATA      ST3750640AS      3.AA  /dev/sdr
[8:0:0:0]    disk    ATA      WDC WD2002FAEX-0 1D05  /dev/sdf
[8:0:1:0]    disk    ATA      WDC WD3000HLHX-0 5G04  /dev/sdg
[8:0:2:0]    disk    ATA      WDC WD20EARX-008 AB51  /dev/sdh
[8:0:3:0]    disk    ATA      MKNSSDCR60GB-DX  BBF0  /dev/sdi
[8:0:4:0]    disk    ATA      WDC WD20EARX-00P AB51  /dev/sdj
charles@beijing:~$ lspci | grep -i sata
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

I get a maximum of 30MB/s from each array.
I am thinking of replacing the controller by a LSI SAS 9200-8e and a SFF-8088 to 4 eSATA cable (like one of these.
Would this HBA work with my enclosure, i.e. can this HBA works with a port multiplier like the one in my enclosure? Would it improve the performance of my arrays? If not, what are my options? The cables between the HBA and the enclosure are 1m long: is it too long? Since the enclosure is right next to the workstation, I could use a shorter cable.
Thanks for your advices!

Charles

 

[Patrick]

This is a bit worse than normal, but eSATA does have that issue. The best way to change the game in terms of performance would be to move to a real SAS/ SATA setup.

Here is a good 2 part series: http://www.servethehome.com/external-sassata-disk-chassis-wiring-part-1/

You have 8 drives running over 2 eSATA ports and a relatively weak RAID controller. All of your disks are running at SATA II speeds but you are going to be low on bandwidth per drive. Once you start striping data across hard drives, you are going to start seeing decent bottlenecks.

 

[charlesnadeau]

Patrick,

I changed the eSata cables over the week-end, shortening them from 1m to 30cm. No changes in performance. Since I know that kernel 3.14 support the HighPoint RocketRAID 644L that came with the enclosure, I'll try it this week-end and report back.
Do you think the LSI HBA and cable I mentioned above would work with my enclosure? I know at one point I'll have to move to an external chassis but for now I'd like to make only minimal change to my set-up.
Thanks again for your help!

Charles

 

[MiniKnight]

@charlesnadeau I think you are basically at the point where you want more performance than you are going to get in the setup. Cables are the cheap fix.

Did they change the
dmesg | grep -i sata | grep 'link up'; lsscsi
Output to all 3.0 gig?

The next step is basically a re-do. I mean - you have a DAS box that is performing worse than a gigabit NAS box. You are going to spend lots of time + money + energy figuring out if it is a software or hardware issue.

Sorry to be so blunt here, but I've done this before and it would have been cheaper to re-do.

My only other longshot idea is I wonder if there is a more compatible controller? There you spend $35 on an esata controller if it works great, or you save the $35 towards the eventual fix and save hours.

 

[Darkytoo]

I have to agree with MiniKnight, I ran into similar issues with my setup. I started with lower end RAID controllers and moved up to my current LSI 9271 which is about the best. You have a random assortment of drives, some are negotiating at SATA1, and some are negotiating at SATA2. I had the same issue with WD Green drives on every RAID controller I tried, and if you throw a nice hardware RAID controller at those drives, it still won't fix it because the green drives can have TLER issues which cause them to underperform and randomly drop out of arrays. I would making an array with just the drives that are running at SATA2 and see what happens. With what you have now, the only options you have are:
1. Start from scratch, replace WD Green drives with at least toshiba or better drives (basically, any hard drive you can't get at bestbuy) and purchase something like an LSI 9260.
2. Stick with what you have, and try doing some sort of SSD caching for your data. This will help in the short term, but you will very quickly learn the limits of SSD caching, and go back to #1.

I'm no RAID expert, but i've been on this site long enough, and wasted enough money trying to fix the same issues you're seeing that sometimes you just have to throw a bunch of money at a problem.

 

[Mike]

Since you have a bunch of random disks I would create a few small sized arrays and create a spanning volume on top of that, bound to increase the performance at least a little bit. Otherwise, rip out the backplane of that enclosure and run direct sata and do the same with one software array. Md, btrfs or even zfs if that's your thing.

 

[charlesnadeau]

Thank you all for your comments. Over the week-end I moved 3 SATA3 disks from the external enclosure to the main computer case so they could be connected to the LSI 2008 HBA that is on my motherboard. I noticed a big increase in read speed: I get about 140MB/sec from my main ZFS array. However the write speed is still around 25MB/sec. What puzzles me is all my Sata3 drives connected to my LSI2008 negotiate a link speed of 3Gb/s instead of 6Gb/s (I only connected Sata3 drives to this HBA). I am using Ubuntu 12.04.4. I already checked the jumpers and the cables. Should I use the libata parameter at boot time to force the speed to 6Gb/s?

Regarding the SAS enclosure, I am leaning toward this Supermicro one, paired with a LSI 9207-8e controller. Is it a combination that will give me good speed?
Thanks again.

Charles