9400-16e bottlenecking at 4GB/s?

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
Welp, this is looking much better. All I did was swap the two cables I was using from port 1 to port 2 and vice versa. On top of that I upgraded my HBA firmware to the latest P22 (previously was on P20).

Code:
# ./readtest.sh

h 229.97 MB/sec
o 248.21 MB/sec
m 250.91 MB/sec
v 229.30 MB/sec
w 242.84 MB/sec
p 247.66 MB/sec
r 243.08 MB/sec
c 238.93 MB/sec
l 238.63 MB/sec
f 244.10 MB/sec
t 247.52 MB/sec
g 233.57 MB/sec
b 244.49 MB/sec
a 257.27 MB/sec
n 233.47 MB/sec
u 236.98 MB/sec
x 238.29 MB/sec
k 238.72 MB/sec
d 226.05 MB/sec
e 223.39 MB/sec
s 254.68 MB/sec
i 234.16 MB/sec
q 218.85 MB/sec
j 236.76 MB/sec
 
  • Like
Reactions: UhClem and itronin

UhClem

Active Member
Jun 26, 2012
258
119
43
NH, USA
Welp, this is looking much better. ...
Nice!
And, with that out of the way ...
... Also, I have a second one of these units connected the exact same way (2 miniSAS-HD cables from 9400-16e to JB4242). In that system is 24 SATA disks and 8 x SAS3 SSD's. I ran your script on that system and it comes back with the below output which adds up to roughly 7,126MiB (7.47GB/s) which is pretty close to the theoretical bandwidth of of PCIe 3.0 x8.
Damn! That's the best I've ever seen (for PCIe 3 x8).
Given the device mix on Sys2, it might even go higher by not using DIRECT I/O.
You can remove the --direct from the script.
Also, if you pipe the output from the script through the following script, it will sort the output and give you a Total line (in MdB [vs MiB]):
Code:
#!/bin/bash
echo "scale=2" > /tmp/ttot_$$
echo -n "(" >> /tmp/ttot_$$
sort | tee /dev/tty | sed -n -e "s/^.* \([0-9\.]*\) .*$/\1+\\\/p" >> /tmp/ttot_$$
echo "0)*10.24*10.24/100" >> /tmp/ttot_$$
echo "Total =" `bc < /tmp/ttot_$$` MdB/sec
rm /tmp/ttot_$$
What motherboard on Sys2? (also a X11SCH ?)
On that system, please do:
Code:
lspci -vv -d 1000:00ac
(Some gory details about the 9400-16) and
Code:
lspci -nn -d 10b5:
(might produce nothing)
and
Code:
lsscsi -v | grep -B1 expander
(details about the devices, which could give rise to something "interesting")
Thanks.
 

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
Nice!
And, with that out of the way ...

Damn! That's the best I've ever seen (for PCIe 3 x8).
Given the device mix on Sys2, it might even go higher by not using DIRECT I/O.
You can remove the --direct from the script.
Also, if you pipe the output from the script through the following script, it will sort the output and give you a Total line (in MdB [vs MiB]):
Code:
#!/bin/bash
echo "scale=2" > /tmp/ttot_$$
echo -n "(" >> /tmp/ttot_$$
sort | tee /dev/tty | sed -n -e "s/^.* \([0-9\.]*\) .*$/\1+\\\/p" >> /tmp/ttot_$$
echo "0)*10.24*10.24/100" >> /tmp/ttot_$$
echo "Total =" `bc < /tmp/ttot_$$` MdB/sec
rm /tmp/ttot_$$
What motherboard on Sys2? (also a X11SCH ?)
On that system, please do:
Code:
lspci -vv -d 1000:00ac
(Some gory details about the 9400-16) and
Code:
[CODE]# lspci -nn -d 10b5:
#
[/code]
(might produce nothing)
and
Code:
lsscsi -v | grep -B1 expander
(details about the devices, which could give rise to something "interesting")
Thanks.

Well, at the moment I now have both JB4242 units connected to the same 9400-16e on System 2. System 2 is a SuperMicro H12SSL-I with PCIe 4.0 slots and an EPYC 7443p. See below for the outputs you requested.

Code:
# lspci -vv -d 1000:00ac
44:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3416 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) (rev 01)
        Subsystem: Broadcom / LSI HBA 9400-16e
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 81
        IOMMU group: 66
        Region 0: Memory at 2807f900000 (64-bit, prefetchable) [size=1M]
        Region 2: Memory at 2807f800000 (64-bit, prefetchable) [size=1M]
        Region 4: Memory at b3500000 (32-bit, non-prefetchable) [size=1M]
        Region 5: I/O ports at 2000 [size=256]
        Expansion ROM at b3600000 [virtual] [disabled] [size=256K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x8 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [148 v1] Power Budgeting <?>
        Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [168 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [254 v1] Dynamic Power Allocation <?>
        Capabilities: [284 v1] Vendor Specific Information: ID=0002 Rev=1 Len=100 <?>
        Capabilities: [384 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
Code:
# lsscsi -v | grep -B1 expander
[1:0:0:0]    disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sda
  dir: /sys/bus/scsi/devices/1:0:0:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:0/end_device-1:1:0/target1:0:0/1:0:0:0]
[1:0:1:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdb
  dir: /sys/bus/scsi/devices/1:0:1:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:0/end_device-1:0:0/target1:0:1/1:0:1:0]
[1:0:2:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdc
  dir: /sys/bus/scsi/devices/1:0:2:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:1/end_device-1:0:1/target1:0:2/1:0:2:0]
[1:0:3:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdd
  dir: /sys/bus/scsi/devices/1:0:3:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:2/end_device-1:0:2/target1:0:3/1:0:3:0]
[1:0:4:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sde
  dir: /sys/bus/scsi/devices/1:0:4:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:3/end_device-1:0:3/target1:0:4/1:0:4:0]
[1:0:5:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdf
  dir: /sys/bus/scsi/devices/1:0:5:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:4/end_device-1:0:4/target1:0:5/1:0:5:0]
[1:0:6:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdg
  dir: /sys/bus/scsi/devices/1:0:6:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:5/end_device-1:0:5/target1:0:6/1:0:6:0]
[1:0:7:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdh
  dir: /sys/bus/scsi/devices/1:0:7:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:6/end_device-1:0:6/target1:0:7/1:0:7:0]
[1:0:8:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdi
  dir: /sys/bus/scsi/devices/1:0:8:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:7/end_device-1:0:7/target1:0:8/1:0:8:0]
[1:0:9:0]    disk    HGST     HUH721010AL4200  A9G0  /dev/sdj
  dir: /sys/bus/scsi/devices/1:0:9:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:8/end_device-1:0:8/target1:0:9/1:0:9:0]
[1:0:10:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdk
  dir: /sys/bus/scsi/devices/1:0:10:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:9/end_device-1:0:9/target1:0:10/1:0:10:0]
[1:0:11:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdl
  dir: /sys/bus/scsi/devices/1:0:11:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:10/end_device-1:0:10/target1:0:11/1:0:11:0]
[1:0:12:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdm
  dir: /sys/bus/scsi/devices/1:0:12:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:11/end_device-1:0:11/target1:0:12/1:0:12:0]
[1:0:13:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdn
  dir: /sys/bus/scsi/devices/1:0:13:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:12/end_device-1:0:12/target1:0:13/1:0:13:0]
[1:0:14:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdo
  dir: /sys/bus/scsi/devices/1:0:14:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:13/end_device-1:0:13/target1:0:14/1:0:14:0]
[1:0:15:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdp
  dir: /sys/bus/scsi/devices/1:0:15:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:14/end_device-1:0:14/target1:0:15/1:0:15:0]
[1:0:16:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdq
  dir: /sys/bus/scsi/devices/1:0:16:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:15/end_device-1:0:15/target1:0:16/1:0:16:0]
[1:0:17:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdr
  dir: /sys/bus/scsi/devices/1:0:17:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:16/end_device-1:0:16/target1:0:17/1:0:17:0]
[1:0:18:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sds
  dir: /sys/bus/scsi/devices/1:0:18:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:17/end_device-1:0:17/target1:0:18/1:0:18:0]
[1:0:19:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdt
  dir: /sys/bus/scsi/devices/1:0:19:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:18/end_device-1:0:18/target1:0:19/1:0:19:0]
[1:0:20:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdu
  dir: /sys/bus/scsi/devices/1:0:20:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:19/end_device-1:0:19/target1:0:20/1:0:20:0]
[1:0:21:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdv
  dir: /sys/bus/scsi/devices/1:0:21:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:20/end_device-1:0:20/target1:0:21/1:0:21:0]
[1:0:22:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdw
  dir: /sys/bus/scsi/devices/1:0:22:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:21/end_device-1:0:21/target1:0:22/1:0:22:0]
[1:0:23:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdx
  dir: /sys/bus/scsi/devices/1:0:23:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:22/end_device-1:0:22/target1:0:23/1:0:23:0]
[1:0:24:0]   disk    HGST     HUH721010AL4200  A9G0  /dev/sdy
  dir: /sys/bus/scsi/devices/1:0:24:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:23/end_device-1:0:23/target1:0:24/1:0:24:0]
[1:0:25:0]   enclosu QCT      JB4242 SIM0      1600  -
  dir: /sys/bus/scsi/devices/1:0:25:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:0/expander-1:0/port-1:0:24/end_device-1:0:24/target1:0:25/1:0:25:0]
[1:0:26:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdz
  dir: /sys/bus/scsi/devices/1:0:26:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:1/end_device-1:1:1/target1:0:26/1:0:26:0]
[1:0:27:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdaa
  dir: /sys/bus/scsi/devices/1:0:27:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:2/end_device-1:1:2/target1:0:27/1:0:27:0]
[1:0:28:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdab
  dir: /sys/bus/scsi/devices/1:0:28:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:3/end_device-1:1:3/target1:0:28/1:0:28:0]
[1:0:29:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdac
  dir: /sys/bus/scsi/devices/1:0:29:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:4/end_device-1:1:4/target1:0:29/1:0:29:0]
[1:0:30:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdad
  dir: /sys/bus/scsi/devices/1:0:30:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:5/end_device-1:1:5/target1:0:30/1:0:30:0]
[1:0:31:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdae
  dir: /sys/bus/scsi/devices/1:0:31:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:6/end_device-1:1:6/target1:0:31/1:0:31:0]
[1:0:32:0]   disk    ATA      WDC WD101KRYZ-01 1H02  /dev/sdaf
  dir: /sys/bus/scsi/devices/1:0:32:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:7/end_device-1:1:7/target1:0:32/1:0:32:0]
[1:0:33:0]   disk    ATA      WDC WD100EMAZ-00 0A83  /dev/sdag
  dir: /sys/bus/scsi/devices/1:0:33:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:8/end_device-1:1:8/target1:0:33/1:0:33:0]
[1:0:34:0]   disk    ATA      WDC WD100EMAZ-00 0A83  /dev/sdah
  dir: /sys/bus/scsi/devices/1:0:34:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:9/end_device-1:1:9/target1:0:34/1:0:34:0]
[1:0:35:0]   disk    ATA      WDC WD100EMAZ-00 0A83  /dev/sdai
  dir: /sys/bus/scsi/devices/1:0:35:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:10/end_device-1:1:10/target1:0:35/1:0:35:0]
[1:0:36:0]   disk    ATA      WDC WD100EMAZ-00 0A83  /dev/sdaj
  dir: /sys/bus/scsi/devices/1:0:36:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:11/end_device-1:1:11/target1:0:36/1:0:36:0]
[1:0:37:0]   disk    ATA      WDC WD100EMAZ-00 0A83  /dev/sdak
  dir: /sys/bus/scsi/devices/1:0:37:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:12/end_device-1:1:12/target1:0:37/1:0:37:0]
[1:0:38:0]   disk    ATA      WDC WD101EMAZ-11 0A81  /dev/sdal
  dir: /sys/bus/scsi/devices/1:0:38:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:13/end_device-1:1:13/target1:0:38/1:0:38:0]
[1:0:39:0]   disk    ATA      WDC WD140EMFZ-11 0A81  /dev/sdam
  dir: /sys/bus/scsi/devices/1:0:39:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:14/end_device-1:1:14/target1:0:39/1:0:39:0]
[1:0:40:0]   disk    ATA      WDC WD140EDFZ-11 0A81  /dev/sdan
  dir: /sys/bus/scsi/devices/1:0:40:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:15/end_device-1:1:15/target1:0:40/1:0:40:0]
[1:0:41:0]   disk    ATA      WDC WD100EMAZ-00 0A83  /dev/sdao
  dir: /sys/bus/scsi/devices/1:0:41:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:16/end_device-1:1:16/target1:0:41/1:0:41:0]
[1:0:42:0]   disk    ATA      WDC WD140EMFZ-11 0A81  /dev/sdap
  dir: /sys/bus/scsi/devices/1:0:42:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:17/end_device-1:1:17/target1:0:42/1:0:42:0]
[1:0:43:0]   disk    ATA      WDC WD140EMFZ-11 0A81  /dev/sdaq
  dir: /sys/bus/scsi/devices/1:0:43:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:18/end_device-1:1:18/target1:0:43/1:0:43:0]
[1:0:44:0]   disk    ATA      WDC WD140EDFZ-11 0A81  /dev/sdar
  dir: /sys/bus/scsi/devices/1:0:44:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:19/end_device-1:1:19/target1:0:44/1:0:44:0]
[1:0:45:0]   disk    ATA      WDC WD140EMFZ-11 0A81  /dev/sdas
  dir: /sys/bus/scsi/devices/1:0:45:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:20/end_device-1:1:20/target1:0:45/1:0:45:0]
[1:0:46:0]   disk    ATA      WDC WD140EMFZ-11 0A81  /dev/sdat
  dir: /sys/bus/scsi/devices/1:0:46:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:21/end_device-1:1:21/target1:0:46/1:0:46:0]
[1:0:47:0]   disk    ATA      WDC WD140EMFZ-11 0A81  /dev/sdau
  dir: /sys/bus/scsi/devices/1:0:47:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:22/end_device-1:1:22/target1:0:47/1:0:47:0]
[1:0:48:0]   disk    ATA      WDC WD140EDFZ-11 0A81  /dev/sdav
  dir: /sys/bus/scsi/devices/1:0:48:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:23/end_device-1:1:23/target1:0:48/1:0:48:0]
[1:0:49:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdaw
  dir: /sys/bus/scsi/devices/1:0:49:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:24/end_device-1:1:24/target1:0:49/1:0:49:0]
[1:0:50:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdax
  dir: /sys/bus/scsi/devices/1:0:50:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:25/end_device-1:1:25/target1:0:50/1:0:50:0]
[1:0:51:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sday
  dir: /sys/bus/scsi/devices/1:0:51:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:26/end_device-1:1:26/target1:0:51/1:0:51:0]
[1:0:52:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdaz
  dir: /sys/bus/scsi/devices/1:0:52:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:27/end_device-1:1:27/target1:0:52/1:0:52:0]
[1:0:53:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdba
  dir: /sys/bus/scsi/devices/1:0:53:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:28/end_device-1:1:28/target1:0:53/1:0:53:0]
[1:0:54:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdbb
  dir: /sys/bus/scsi/devices/1:0:54:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:29/end_device-1:1:29/target1:0:54/1:0:54:0]
[1:0:55:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdbc
  dir: /sys/bus/scsi/devices/1:0:55:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:30/end_device-1:1:30/target1:0:55/1:0:55:0]
[1:0:56:0]   disk    HGST     HBTAC2DH2SUN3.2T B170  /dev/sdbd
  dir: /sys/bus/scsi/devices/1:0:56:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:31/end_device-1:1:31/target1:0:56/1:0:56:0]
[1:0:57:0]   enclosu QCT      JB4242 SIM0      1600  -
  dir: /sys/bus/scsi/devices/1:0:57:0  [/sys/devices/pci0000:40/0000:40:03.1/0000:44:00.0/host1/port-1:1/expander-1:1/port-1:1:32/end_device-1:1:32/target1:0:57/1:0:57:0]
 

UhClem

Active Member
Jun 26, 2012
258
119
43
NH, USA
Well, at the moment I now have both JB4242 units connected to the same 9400-16e on System 2. System 2 is a SuperMicro H12SSL-I with PCIe 4.0 slots and an EPYC 7443p.
That's a cool set-up, being able to "re-configure" such large sub-hierarchies (each shelf) between systems.
See below for the outputs you requested.
Thanks very much.
The lspci -vv -d 1000:00ac confirms what I expected was the reason you were able to get that extra throughput. It is due to an incremental reduction in PCIe bus overhead.
Code:
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
         ==>           MaxPayload 512 bytes, MaxReadReq 512 bytes
With the effective [DevCtl (vs DevCap)] MaxPayloadSize of 512 (vs 256), overhead is reduced, and your bus throughput ceiling is raised to (best-case) 7560 MB/s (from 7250). Don't know whether its the HBA, the Mobo/Gen4, or a combination. (I'd guess that the X11SCH-connected 9400 will show 256, but ...)

The lsscsi output suggests (from the WD model#s) that there should be adequate combined throughput from the 24x SATA drives to show you "something interesting" (warning: it will feel like "deja vu all over again";)). I can't tell whether the present config has the two shelves in cascade ("series"), or "parallel" (using all 4 HBA ports); and it would be best to do this test with the SATA-containing shelf directly connected, dual-link, to the 9400. Then, pls run the readtest using the drive letters for all 24 SATA drives (but no SAS). And, don't worry--It's not a/another bug/glitch--more like a feature with an implementation limitation (documented by LSI [yes, I determined that the JB4242 actually does use LSI expander chip(s)--and made my amend[ment]s in an above [POST]]).
 
  • Like
Reactions: Brian Puccio

IamSpartacus

Well-Known Member
Mar 14, 2016
2,463
615
113
That's a cool set-up, being able to "re-configure" such large sub-hierarchies (each shelf) between systems.

Thanks very much.
The lspci -vv -d 1000:00ac confirms what I expected was the reason you were able to get that extra throughput. It is due to an incremental reduction in PCIe bus overhead.

With the effective [DevCtl (vs DevCap)] MaxPayloadSize of 512 (vs 256), overhead is reduced, and your bus throughput ceiling is raised to (best-case) 7560 MB/s (from 7250). Don't know whether its the HBA, the Mobo/Gen4, or a combination. (I'd guess that the X11SCH-connected 9400 will show 256, but ...)

The lsscsi output suggests (from the WD model#s) that there should be adequate combined throughput from the 24x SATA drives to show you "something interesting" (warning: it will feel like "deja vu all over again";)). I can't tell whether the present config has the two shelves in cascade ("series"), or "parallel" (using all 4 HBA ports); and it would be best to do this test with the SATA-containing shelf directly connected, dual-link, to the 9400. Then, pls run the readtest using the drive letters for all 24 SATA drives (but no SAS). And, don't worry--It's not a/another bug/glitch--more like a feature with an implementation limitation (documented by LSI [yes, I determined that the JB4242 actually does use LSI expander chip(s)--and made my amend[ment]s in an above [POST]]).
Interesting info for sure.

Right now, I do indeed have both systems dual linked directly to two ports on the 9400-16e. Read test results of all 24 SATA drives is below. It's not a perfect test case because this server is being used for media streaming as we speak. Also there is a mix of SATA drives in here. 8 of them are WD Gold 7200pm drives and the rest are white label 5400rpm drives.

Code:
af 251.81 MB/sec
at 191.73 MB/sec
ap 204.10 MB/sec
ag 197.37 MB/sec
av 214.63 MB/sec
aj 209.05 MB/sec
am 210.59 MB/sec
an 210.39 MB/sec
ak 207.00 MB/sec
ai 201.62 MB/sec
as 209.94 MB/sec
aq 213.01 MB/sec
ac 254.46 MB/sec
au 199.23 MB/sec
ae 242.14 MB/sec
al 191.94 MB/sec
ar 189.75 MB/sec
ab 237.27 MB/sec
ao 206.54 MB/sec
ad 235.02 MB/sec
a 228.55 MB/sec
y 240.20 MB/sec
aa 237.34 MB/sec
ah 195.76 MB/sec
 
  • Wow
Reactions: UhClem

UhClem

Active Member
Jun 26, 2012
258
119
43
NH, USA
Interesting info for sure.

Right now, I do indeed have both systems dual linked directly to two ports on the 9400-16e. Read test results of all 24 SATA drives is below. It's not a perfect test case ...
It is quite OK.
And I think the yolk's on me!

I was expecting that test to be vulnerable to a limitation in LSI's Databolt implementation, which I have seen examples of. Likely, the JB4242 is not an "ordinary" SAS-expander.

The test resulted in a Total of 5431 MdB/sec. If the 9400-16 had been dual-linked to an ordinary expander, e.g. a LSI SAS3x36, the total should have been ~5200 (see below; note that the hdparm test does reads of 2MB (which the kernel driver breaks into (4x) 512KB reads to the controller; hence, the lower "tail" of the blue SeqRead plot).
LSI-Databolt-95.jpg

Or, maybe 5200 vs 5400 is "close enough for guvmnt work"? As a final test, could you run the test again, this time removing the "&" from the hdparm line. That will cause each drive's speed to be tested stand-alone (no concurrency/contention), sequentially. By comparing the two results, we'll be able to see if concurrency caused drive speeds to be throttled. Thanks.
=========
"If you push something hard enough, it will fall over." -- Fudd's First Law:)