C6100 with SAS1068E poor disk i/o performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Gromph

New Member
Jul 8, 2013
9
0
0
I'm experiencing poor disk i/o performance on a Dell C6100 with a SAS1068E. Running stock CentOS 6.5 with all the latest updates
/dev/sda is a 4 disk IME array of 15k sas seagate ST9146852SS drives
/dev/sdb is a 2 disk IM array of 2 Samsung 840 Pro SSD drives
I'm using sysbench to test: sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndwr run
/dev/sda gets about 7 MB/sec
/dev/sdb gets about 3.5 MB/sec
I would expect much higher performance, especially from the SSDs. I have tried many things such as:
1. Enabling write caching using lsiutil
2. echo deadline >/sys/block/sdb/queue/scheduler
3. set options: nodiratime,discard in /etc/fstab
4. enabled TRIM for LVM by setting issue_discards in /etc/lvm/lvm.conf
Does anyone have any other suggestions to improve performance? Thanks.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
What 1068e based card are you using ?.

Is it via the PCIe slot or mezzanine slot ?.

Whilst the 1068e IO controller chipset is limited to SATA II speeds, those speeds you are seeing are still extremely slow.

RB
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Have you tried benchmarking the drives individually to make sure you can get decent speeds from the single drives first. Make sure you test all the cables just in case you have a bad connection / cable in there slowing the whole array down.

Have you also confirmed the arrays are sync'd and are not building / rebuilding as this is likely to slow down performance.

The 1068e based cards are entry level but I would expect faster speeds than that.

If you are running Linux then have you checked the PCIe connection speed for that device.
Code:
lspci -vv
You should get something back like
LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCap gives the ports capacity.
LnkSta gives the connected cards current state.

Look at the Speed and Width settings. Speed 2.5GT/s is PCIe 2.0 and Width x8 means all 8 PCIe lanes are being used.

RB
 

Gromph

New Member
Jul 8, 2013
9
0
0
Nope I haven't tried benchmarking them individually. The server is in colo, so I'll have to see if I can use lsiutil to pull a drive out of the raid and mount by itself.

lspci looks good:
Capabilities: [68] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-


On the node I posted about originally with 15k 146gb SAS drives in a 4 disk IME raid. :
They get about 56mb/s doing a seqwr: sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=seqwr run
And about 3.6mb/s doing a rndwr: sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndwr run

On 2 other nodes I have the same 15k 146gb SAS drives in a 4 disk IME raid.
They get about 80mb/s doing a seqwr: sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=seqwr run
And about 8mb/s doing a rndwr: sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndwr run


Does that sound about right? Seems low to me.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Well writing is not the strong point of a Raid 1E array as you are duplicating the writes over a couple of disks.

What are the read benchmarks like ?.

The problem with Colo is that in this case, node 1s differences could very well be due to a hardware problem (faulty sas cable, 1 drive underperforming, faulty sas card connector etc). Not being able to play with the hardware is going to make it hard to find. Doing an individual drive benchmark may narrow it down though.

RB
 

Gromph

New Member
Jul 8, 2013
9
0
0
Here are some read benchmarks:
In between tests I ran this to clear the linux file cache: sync && echo 1 > /proc/sys/vm/drop_caches

node 1
sda is a 4 disk IME array of 15k sas seagate ST9146852SS drives
sdb is a 2 disk IM array of 2 Samsung 840 Pro SSD drives

sda 21Mb/sec sysbench --num-threads=16 --test=fileio --file-total-size=55G --file-test-mode=rndrd run
sda 295MB/sec sysbench --num-threads=16 --test=fileio --file-total-size=55G --file-test-mode=seqrd run

sdb 157Mb/sec sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=rndrd run
sdb 137Mb/sec sysbench --num-threads=16 --test=fileio --file-total-size=3G --file-test-mode=seqrd run

node 4
sda is a 4 disk IME array of 15k sas seagate ST9146852SS drives
sda 25Mb/Sec sysbench --num-threads=16 --test=fileio --file-total-size=20G --file-test-mode=rndrd run
sda 291Mb/Sec sysbench --num-threads=16 --test=fileio --file-total-size=20G --file-test-mode=seqrd run
 

dba

Moderator
Feb 20, 2012
1,477
184
63
San Francisco Bay Area, California, USA
I have done some rough testing of the old LSI 1068e chip. It's not fast, but it's faster than what you are seeing by far! There is an issue somewhere. As mentioned before, I'd try single disk benchmarks to confirm that each disk is working as expected and that the card can push data as expected when no RAID is being used.
 

Gromph

New Member
Jul 8, 2013
9
0
0
Thanks for the suggestions. I went ahead and bought a LSI2008 mezzanine card (XX2X2), but the random write performance isn't much better. I tried lots of tests and the best was the sata on the motherboard, but I really want to use the LSI2008 card to improve performance of the SAS drives too.


motherboard SATA using a single 840 EVO 250GB
random write: 29 MB/s

using new LSI2008 card with 840 EVO 120GB single drive no raid, no lvm
random write: usually 6MB/s but sometimes 10MB/s
sequential write: 85Mb/s
random read: 300MB/s
sequential read: 478MB/s


# lspci -vv
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
Subsystem: Inventec Corporation Device 6019
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 24
Region 0: I/O ports at d000
Region 1: Memory at faf3c000 (64-bit, non-prefetchable) [size=16K]
Region 3: Memory at faf40000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at faf80000 [disabled] [size=512K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

I noticed when I ran lspci -vv dmesg said:
mpt2sas 0000:02:00.0: vpd r/w failed. This is likely a firmware bug on this device. Contact the card vendor for a firmware update.

# dmesg | grep LSISAS2008
mpt2sas0: LSISAS2008: FWVersion(06.00.00.00), ChipRevision(0x03), BiosVersion(07.07.00.00)


I setup the OMSA tools on this page: Index of /repo/hardware/OMSA_7.4.0
And ran update_firmware:
Running system inventory...

Searching storage directory for available BIOS updates...
Checking BIOS - 1.44.00
Did not find a newer package to install that meets all installation checks.
Checking ST9146852SS Firmware - ht66
Did not find a newer package to install that meets all installation checks.
Checking SAS 2008 FW v1.4 Controller 0 Firmware - 00.00.00.00
Did not find a newer package to install that meets all installation checks.
Checking Samsung SSD 840 Firmware - 4b0q
Did not find a newer package to install that meets all installation checks.
Checking System BIOS for XS23-TY3 - 1.44.00
Did not find a newer package to install that meets all installation checks.


I tried different cables, different slots, played with a few settings mentioned in my first post. None had much effect. I keep thinking there has got to be something I'm missing, but doing lots of searching on LSI2008 and linux performance doesn't bring up much. It seems it should be plug and play. I installed CentOS 6.5 and sysbench and not much else.

Thanks.