SAS 3008 PCIe3.0 x4 only, why not x8 lanes???

berndh

New Member
Mar 11, 2021
9
0
1
Got my self hands on a Asrock X570D4U board with a Ryzen 5 3600 CPU and attached some peripherals to it:

M2 slot filled with PCIE3.0x4 NVME SSD
PCIE4 slot filled with Intel X520-2DA (dual 10gbits NIC) PCIE2.0x8
PCIE6 slot filled with a SAS 3008 PCIE3.0x4, hooked up only 4 SATA disks (for now)

Now the thing is that I see that the SAS 3008 only runs on PCIE3.0 with x4 lanes (downgraded) and not on a x8 lanes bandwidth as according to the SAS 3008 chipset datasheet.

If I count up all the PCIE lanes and knowing that the Ryzen 5 3600 has 24 PCIE lanes to distribute:
x8 for Intel NIC
x4 for M2 SSD
x4 for the Upstream link X570 chipset

I still have x8 lanes left for the SAS controller. Why is it only taking (downgrade) to x4? Swapping the SAS 3008 from PCIE slot 4 to Slot 6 does not make any difference.

If I remove the Intel NIC board completely, still the SAS 3008 on x4 lanes.

Thanks for some explanation/insights you could give me.

Dump of the lspci -vv

2d:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
Subsystem: Broadcom / LSI SAS9300-8i
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 61
IOMMU group: 28
Region 0: I/O ports at f000 [disabled]
Region 1: Memory at fc540000 (64-bit, non-prefetchable) [size=64K]
Region 3: Memory at fc500000 (64-bit, non-prefetchable) [size=256K]
Expansion ROM at fc400000 [disabled] [size=1M]
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x4 (downgraded)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
 
Last edited:

llowrey

Active Member
Feb 26, 2018
100
67
28
I have a similar config with the same board and CPU.

PCIE6 is an x8 Mellanox card
PCIE5 is an x8 SAS 3008 running at x4 since the slot is only x4
PCIE4 is an x8 SAS 3008 running at x8

My guess is you might have bifurcation set up incorrectly. It should be x8x8 but if you have it set to x8x4x4 you would see what you are observing since PCIE4 would be configured for two x4 devices instead of a single x8 device.

Also note that one of the m.2's is PCIe3 x2 and the other is PCIe2 x4 and both are connected to the X470 chipset and therefore share the same PCIe3 x4 bandwidth to the CPU.
 

berndh

New Member
Mar 11, 2021
9
0
1
OK, so it should be possible to run it in a x8 config. Are you sure you have the X570 board as well? and confirmed that these cards actually run at the right speed and lanes bandwidth?

I tried the bifurcation setting indeed, as I thought that would do the trick:
I had the slot configured as x16 and also tried the x8x8, both show the SAS 3008 card show up as x4.

The X570D4U board actually scales down PCIE6 from x16 to x8 when there is a card inserted into PCIE4, that is why I also tried to remove the NIC and see if the lanes usage went up for PCIE6. No change.

My thinking now, is that the card SAS itself negotiates a x4 link, is there a bios setting somewhere?

Thanks
 

i386

Well-Known Member
Mar 18, 2016
2,464
677
113
31
Germany
The block diagram shows a switch between cpu <-> bios & bmc, not sure if it is using 4 pcie lanes...
 

berndh

New Member
Mar 11, 2021
9
0
1
Correct. That is indeed shown in the picture. However, when I remove the NIC card an only use the SAS card, it still keeps running at x4. Therefor I'm expecting that there is an setting incorrect that keeps the card running on x4.
 

llowrey

Active Member
Feb 26, 2018
100
67
28
Dang... I misread your post I thought you specified the X470D4U. Sorry for adding confusion. I do have an X570D4U waiting for a CPU so can't test this just yet.

I still recommend checking if bifurcation is x8x8 and not x8x4x4
 

berndh

New Member
Mar 11, 2021
9
0
1
Dang... I misread your post I thought you specified the X470D4U. Sorry for adding confusion. I do have an X570D4U waiting for a CPU so can't test this just yet.

I still recommend checking if bifurcation is x8x8 and not x8x4x4
No problem. the X470D4U indeed has a different PCIE layout. I was already confused for SLOT5 being x4... on the X570 it is an x1 :(

I did check the bifurcation setting. Set it to x8x8 an still the SAS gets x4 lanes assigned. However bifurcation is actually only applicable for PCIE6 if you use in x16 mode, and want to split that x16 into smaller chunks. It has nothing to do with PCIE4 unless you actually have a card in PCIE4, then PCIE6 only gets x8. Perhaps then bifurcation is not supported anymore.

Is perhaps the SAS 3008 depending on how many HDD/SDD are attached for the link negotiation? I only have 4 HDD connected. Perhaps I don't need the additonal bandwidth, unless I have 8 HDD/SDD attached. But I can give it a try with some spare HDD lying around.
 
Last edited:

berndh

New Member
Mar 11, 2021
9
0
1
Ok, did another test. I removed the HBA and added another Intel NIC PCIE2.0 x8 into the PCIE6 slot. Booted the server and now both the card end up as x8 (no downgrades to x4). So it must be the HBA card that actuallly negotiates a downgrade to x4.

Is there a setting somewhere that I can change to make it a x8 again in the HBA board itself?

I see this in the dmesg:
[ 0.566149] pci 0000:2d:00.0: reg 0x10: [io 0xf000-0xf0ff]
[ 0.566160] pci 0000:2d:00.0: reg 0x14: [mem 0xfc540000-0xfc54ffff 64bit]
[ 0.566170] pci 0000:2d:00.0: reg 0x1c: [mem 0xfc500000-0xfc53ffff 64bit]
[ 0.566183] pci 0000:2d:00.0: reg 0x30: [mem 0xfc400000-0xfc4fffff pref]
[ 0.566189] pci 0000:2d:00.0: enabling Extended Tags
[ 0.566290] pci 0000:2d:00.0: supports D1 D2
[ 0.566332] pci 0000:2d:00.0: 31.504 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x4 link at 0000:00:03.1 (capable of 63.008 Gb/s with 8.0 GT/s PCIe x8 link)

Apparently the link is limited by the bridge at 00:03.1, which indeed is downgraded to x4, but why??? There is no other device attached to this bridge so the bridge is propably configured to the HBA, but it is capable of doing x8.

Btw, running the SAS 3008 P16 firmware. Someone have experience with this firmware? I also tried to flash it back to P13, but got the same x4 result. Seems to be really specific to the HBA.
 
Last edited:

berndh

New Member
Mar 11, 2021
9
0
1
Update: I removed the HBA and put it into a Gigabyte B550i Aurus Pro AX with Ryzen 4650g processor in the only available x16 slot that system has and it also turns up as a PCIE3.0 x4 card. It was immediately recognized by Windows 10, but again with an x4 PCIE link width. It also showed the x8 capability.

I guess my card is just broken, or this is normal behaviour. Could someone check their link width for their SAS 3008/LSI 9300 8i IT card?

Thanks in advance
 

llowrey

Active Member
Feb 26, 2018
100
67
28
This is in my X470D4U board:

Code:
2c:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
        Subsystem: Broadcom / LSI SAS9300-8i
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 112
        Region 0: I/O ports at e000 [size=256]
        Region 1: Memory at f6d40000 (64-bit, non-prefetchable) [size=64K]
        Region 3: Memory at f6d00000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at f6c00000 [disabled] [size=1M]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 512 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x8 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, NROPrPrP-, LTR-
                         10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt-, EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-, TPHComp-, ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                         AtomicOpsCtl: ReqEn-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
                         EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [c0] MSI-X: Enable+ Count=96 Masked-
                Vector table: BAR=1 offset=0000e000
                PBA: BAR=1 offset=0000f000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 04000001 0000200f 2c010000 4d25e7ba
        Capabilities: [1e0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn-, PerformEqu-
                LaneErrStat: 0
        Capabilities: [1c0 v1] Power Budgeting <?>
        Capabilities: [190 v1] Dynamic Power Allocation <?>
        Capabilities: [148 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
Firmware: 16.00.10.00-IT
BIOS: 8.11.00.00

You might want to check/clean the contacts. Perhaps one or more is dirty or a trace has been cut.
 
  • Like
Reactions: berndh

berndh

New Member
Mar 11, 2021
9
0
1
Btw, my config, a little different from yours (the bios seems to be different, the FW seems to be the same) :

------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
Controller type : SAS3008
BIOS version : 8.37.00.00
Firmware version : 16.00.10.00
Channel description : 1 Serial Attached SCSI
Initiator ID : 0
Maximum physical devices : 1023
Concurrent commands supported : 9856
Slot : 0
Segment : 0
Bus : 45
Device : 0
Function : 0
RAID Support : No

What software did you use to flash to IT mode?

What ever I try I don't have a HBA bios to manipulate settings, like sometimes it is mentioned to hit CTRL+H to get into the HBA bios. I don't get that during system boot up. Perhaps there I can change a setting.

Thanks
Bernd
 

berndh

New Member
Mar 11, 2021
9
0
1
Update: After getting a local HBA it works nicely with PCIE3.0 x8. So the card is indeed defective. I've gotten my money back and the seller even paid for the return costs. Very good seller.

@llowrey, thanks for all the great support.

Topic can be closed.