LSI SAS9200-16E -- Slow Performance Using 8+ Drives

Oddworld

Member
Jan 16, 2018
57
28
18
120
Overview: My LSI card in IT mode tends to slow when more than 8 drives are being utilized. Is this normal behavior?


Hardware:



Description of Problem:

  • When running badblocks -wsv /dev/sd[X], I see sustained write speeds of 150 MB/s per drive
  • This holds true for up to 6-8 drives concurrently
  • If I run more than 6-8 concurrently, then cumulative write performance drops
  • If I run all 12 drives concurrently, then all 12 have lower sustained write speeds of 82 MB/s (total of 1020 MB/s)
  • Is this normal behavior?

SAS2IRCU Output:

oddworld@e5labuntu:~/SAS2IRCU_P20/sas2ircu_linux_x86_rel$ sudo ./sas2ircu 0 DISPLAY
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS2116_1
BIOS version : 0.00.00.00
Firmware version : 20.00.07.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 1023
Concurrent commands supported : 32455
Slot : Unknown
Segment : 0
Bus : 6
Device : 0
Function : 0
RAID Support : No
------------------------------------------------------------------------

LSPCI Output:


oddworld@e5labuntu:~/SAS2IRCU_P20/sas2ircu_linux_x86_rel$ sudo lspci -s 06:00.0 -vvv
06:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (rev 02)
Subsystem: LSI Logic / Symbios Logic SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor]
Physical Slot: 2
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 24
NUMA node: 0
Region 0: I/O ports at d000

Region 1: Memory at 50280000 (64-bit, non-prefetchable) [size=16K]

Region 3: Memory at 50080000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at 50000000 [disabled] [size=512K]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend+
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
Not readable
Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
Vector table: BAR=1 offset=00002000
PBA: BAR=1 offset=00003800
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] Power Budgeting <?>
Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 0064
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000050284000 (64-bit, non-prefetchable)
Region 2: Memory at 00000000500c0000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas

oddworld@e5labuntu:~/SAS2IRCU_P20/sas2ircu_linux_x86_rel$

 
Last edited:

UhClem

Member
Jun 26, 2012
55
22
8
NH, USA
It's a 3Gb/s expander. Using a single port/cable/connection limits your bandwidth to 4x(3Gb/s) [theoretical]-- real-world = 4x 250-275 MB/s = 1000-1100 MB/s. Add a second cable/node to double the #s.
 
  • Like
Reactions: altmind

Oddworld

Member
Jan 16, 2018
57
28
18
120
Thanks. Sorry for being thick, but which part is a 3gb/s expander - the DAS or the LSI card? The DAS is supposed to have two 6gb/s ESMs.
 

Oddworld

Member
Jan 16, 2018
57
28
18
120
These are SAS drives. Below is SMART:





smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: IBM-ESXS
Product: ST31000424SS
Revision: BC2D
Compliance: SPC-3
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Logical block size: 512 bytes
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000c5004131c38b
Serial number: #####################
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Feb 12 10:11:45 2020 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature: 37 C
Drive Trip Temperature: 65 C

Elements in grown defect list: 0

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 105159310 0 0 105159310 105159310 5290.445 0
write: 0 0 0 0 0 15905.987 0
verify: 40753 0 0 40753 40753 4.078 0

Non-medium error count: 16

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 10859 - [- - -]
# 2 Background long Completed - 285 - [- - -]
# 3 Background short Completed - 262 - [- - -]
# 4 Background long Completed - 222 - [- - -]
# 5 Background short Completed - 178 - [- - -]

Long (extended) Self-test duration: 13596 seconds [226.6 minutes]
 

UhClem

Member
Jun 26, 2012
55
22
8
NH, USA
From: IBM - IBM System Storage EXP3000 Express -

IBM System Storage EXP3000 Express

Affordable, scalable and performance oriented SAS/SATA storage expansion unit
Highlights
  • 3GBps SAS drive expansion technology
  • Combination of 12 SAS—up to 7.2TB—or SATA—up to 24.0TB—drives per enclosure
  • SAS or SATA hard disk drive intermix supported with select servers using the DS3000 systems, MegaRAID or ServeRAIDâ„¢ host bus adapters
  • Support for three EXP3000 enclosures on a DS3000 system for up to 28.8 TB with SAS or up to 96.0TB SATA
  • Flexible power options for Telco applications and NEBS and ETSI compliance
The IBM System Storage® EXP3000 Express® is a high-density 2U, 19" rack mount driven closure designed for supporting a total of twelve 3.5" disk drives.
====
Good / Cheap / Fast (Choose any 2)
 

Oddworld

Member
Jan 16, 2018
57
28
18
120
Now I'm really scratching my head. Any clues? EBay listing shows 6GB/s (which obviously isn't trustworthy). However IBM's website contemplates both 3 and 6GB/s speeds.

Also from IBM's website: IBM - IBM System Storage EXP3000 Express - Specifications -


Model 1727-01X—IBM System Storage® EXP3000 Express®
1727-02T—IBM System Storage EXP3000 Express with DC power supplies
Environmental services module Single ESM with dual ESM upgrade
SAS interface: Mini-SAS connectors
Drives supported
3GBps SAS: 146GB, 300GB, and 450GB SAS drives at 15,000 rpm speeds
6GBps SAS: 300GB, 450GB, 600GB SAS drives at 15,000 rpm speeds

3GBps SATA: 500GB, 750GB, 1.0 TB, 2.0 TB SATA drives at 7,200 rpm speed
Solid State Disk: 50GB SATA (System x direct attach only via the ServeRAID MR10M SAS/SATA controller adapter)
Fans and power supplies Two hot-swappable 515 watt (115 – 230 V AC) power supplies
Rack support 19", industry-standard rack
Warranty Three-year parts and labor warranty
EXP3000 physical characteristics
Dimensions 8.7 cm x 44.7 cm x 55.0 cm (3.43 in x 21.6 in x 17.6 in) (H x W x D)​


Here's some data from FreeBSD:

Link Speed is reported to be 300.000MB/s

Code:
➜  ~ dmesg | grep -i transfers
ses0: 300.000MB/s transfers
ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes)
da1: 300.000MB/s transfers
da2: 300.000MB/s transfers
da3: 300.000MB/s transfers
da5: 300.000MB/s transfers
da11: 300.000MB/s transfers
da8: 300.000MB/s transfers
da7: 300.000MB/s transfers
da10: 300.000MB/s transfers
da6: 300.000MB/s transfers
da9: 300.000MB/s transfers
da0: 300.000MB/s transfers
da4: 300.000MB/s transfers
➜  ~ sudo camcontrol inquiry da0
pass0: <IBM-ESXS ST31000424SS BC2D> Fixed Direct Access SPC-3 SCSI device
pass0: Serial Number 9WK5QEH60000C2156E9D
pass0: 300.000MB/s transfers, Command Queueing Enabled
➜  ~ sudo smartctl -a /dev/da0
smartctl 7.0 2018-12-30 r4883 [FreeBSD 12.1-RELEASE amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               IBM-ESXS
Product:              ST31000424SS
Revision:             BC2D
Compliance:           SPC-3
User Capacity:        1,000,204,886,016 bytes [1.00 TB]
Logical block size:   512 bytes
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000c5004131c38b
Serial number:        9WK5QEH60000C2156E9D
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Feb 12 11:12:06 2020 CST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature:     37 C
Drive Trip Temperature:        65 C

Elements in grown defect list: 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:   105159314        0         0  105159314   105159314       5290.445           0
write:         0        0         0         0          0      16316.461           0
verify:    41173        0         0     41173      41173          4.116           0

Non-medium error count:       16

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background short  Completed                   -   10859                 - [-   -    -]
# 2  Background long   Completed                   -     285                 - [-   -    -]
# 3  Background short  Completed                   -     262                 - [-   -    -]
# 4  Background long   Completed                   -     222                 - [-   -    -]
# 5  Background short  Completed                   -     178                 - [-   -    -]

Long (extended) Self-test duration: 13596 seconds [226.6 minutes]
 

UhClem

Member
Jun 26, 2012
55
22
8
NH, USA
Don't confuse yourself with drive characteristics. Your throughput limitation stems from the (first-generation/3Gbps) SAS expander technology of the EXP3000.

Can anyone point the OP to an Introduction to SAS Expanders write-up?

Meanwhile, you should verify (from IBM tech docs for the EXP3000) that your expander (itself) supports Dual Cables. If so, a second SAS cable (from the LSI card) would double your throughput.

EDIT: From a cursory look at ftp://ftp.software.ibm.com/systems/support/system_x_pdf/43w7852.pdf , the EXP3000 does not support DualCables for the purpose of doubling throughput. (It looks like the 2 SAS connections you mentioned [btw, labeled In & Out] are for Cascading (multiple EXP3000s and/or ESMs [which would further dilute your throughput, by sharing it among even more drives]).)

That manual is dated 2007! 1000-1100 MB/s was more than sufficient for 12 sata drives at Max. (fastest sas drives ~110ea)
 
Last edited:

Oddworld

Member
Jan 16, 2018
57
28
18
120
Thanks for your patience. I am obviously still learning, and have much more to learn.

The DAS has two ESM modules. Each ESM has two ports (one IN and one OUT, like you mentioned).

Is it possible to run two cables to the DAS to double throughput?
  • HBA Port #1 ==> DAS ECM #1 (IN)
  • HBA Port #2 ==> DAS ECM #2 (IN)
I'm not sure if this is what you mean by Dual Cables, or not. If it helps I am attaching a picture of the back of the DAS.


Imgur

 

UhClem

Member
Jun 26, 2012
55
22
8
NH, USA
... The DAS has two ESM modules. Each ESM has two ports (one IN and one OUT, like you mentioned).

Is it possible to run two cables to the DAS to double throughput?
Not according to page 13 of the above-linked PDF. That "Dual-ESM Configuration" is intended for redundant drive paths, with a separate controller connected to the second ESM, whereby, if supported, you'd have a hot-spare controller (I think that's called failover).
I'm not sure if this is what you mean by Dual Cables ...
Pardon my incorrect terminology, that capability is (correctly) called Dual-Link--apparently lacking in the EXP3000's expander.
 

Stereodude

Active Member
Feb 21, 2016
412
65
28
USA
Thanks for your patience. I am obviously still learning, and have much more to learn.

The DAS has two ESM modules. Each ESM has two ports (one IN and one OUT, like you mentioned).

Is it possible to run two cables to the DAS to double throughput?
  • HBA Port #1 ==> DAS ECM #1 (IN)
  • HBA Port #2 ==> DAS ECM #2 (IN)
I'm not sure if this is what you mean by Dual Cables, or not. If it helps I am attaching a picture of the back of the DAS.
Only if it supports dual linking. You look to be maxing out the SAS link.