PMC PM8001 based HBA

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
I needed another inexpensive low profile HBA with 2 external SFF-8088 ports so I went and and picked up the cheapest one I could find for like $15 shipped. Based on the price alone I totally expected it to be just another LSI SAS2008 clone. But it happened to be Oracle branded and based on PMC PM 8001 chip. I had some experience with Adaptec and HP branded PMC based RAID controllers but never a PMC HBA.

I installed and it works
Code:
lspci -Dnnvvd ::0107
0000:01:00.0 Serial Attached SCSI controller [0107]: PMC-Sierra Inc. Device [11f8:8001] (rev 05)
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at f7b30000 (64-bit, non-prefetchable) [size=64K]
        Region 2: Memory at f7b20000 (64-bit, non-prefetchable) [size=64K]
        Region 4: Memory at f7b10000 (32-bit, non-prefetchable) [size=64K]
        Region 5: Memory at f7b00000 (32-bit, non-prefetchable) [size=64K]
        Expansion ROM at f7a00000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM L0s Enabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [ac] MSI-X: Enable+ Count=16 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00004000
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr+ BadTLP- BadDLLP+ Rollover- Timeout+ NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: pm80xx
        Kernel modules: pm80xx
My question is whether I should keep it or just replace with something more common?
For starters I would like to keep current with firmware updates and in this case I do not even know where to get the updates from. Or what the proper tools to use to flash and configure the card. HP and Adaptec RAID utilities do not seem to recognize this controller. Probably because it uses generic PMC pm80xx driver.

On the positive side I noticed that it seems to run cooler than the LSI cards I have.
 
Last edited:

DanielWood

Member
Sep 14, 2018
44
17
8
My question is whether I should keep it or just replace with something more common?
For starters I would like to keep current with firmware updates and in this case I do not even know where to get the updates from. Or what the proper tools to use to flash and configure the card. HP and Adaptec RAID utilities do not seem to recognize this controller. Probably because it uses generic PMC pm80xx driver.

On the positive side I noticed that it seems to run cooler than the LSI cards I have.
I've not had good luck with the NetApp PMC8003 HBAs. They work, until you have a lot of stress on the SAS link, then once a drive drops out, it seems to cascade lock up the whole bus under both FreeNAS and CentOS.

I've had much better luck with LSI 9206-16E controllers and using SFF-8640 to QSFP cables on my DS4243s. I get drive losses with Unrecovereable Read Errors with using the SAS interposers for multipath, but FreeNAS recovers almost immediately and resilvers a few MBs and continues on the scrub. I am often left with hanging Active/Failed multipath, but I wrote a dead simple script to repair that condition:

Code:
#!/bin/bash

for mpathdisk in `gmultipath status | grep DEGRADED | grep -Eo 'disk[0-9]+'`
do
   gmultipath restore $mpathdisk `gmultipath status $mpathdisk | grep FAIL | grep -Eo 'da[0-9]+ | head -n1'`
   echo "Restored $mpathdisk"
done

Here is what a URE causing dropout looks like, 9-seconds from start to finish:
Code:
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): READ(16). CDB: 88 00 00 00 00 01 44 e2 f3 20 00 00 00 f8 00 00
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): CAM status: SCSI Status Error
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): SCSI status: Check Condition
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): Info: 0x144e2f395
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): Field Replaceable Unit: 1
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): Command Specific Info: 0x4140
Apr 15 00:48:09 freenas (da17:mps0:0:50:0): Error 5, Unretryable error
Apr 15 00:48:09 freenas GEOM_MULTIPATH: Error 5, da17 in disk23 marked FAIL
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): READ(16). CDB: 88 00 00 00 00 01 44 e2 f3 20 00 00 00 f8 00 00
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): CAM status: SCSI Status Error
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): SCSI status: Check Condition
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): Info: 0x144e2f395
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): Field Replaceable Unit: 1
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): Command Specific Info: 0x4140
Apr 15 00:48:11 freenas (da46:mps0:0:79:0): Error 5, Unretryable error
Apr 15 00:48:11 freenas GEOM_MULTIPATH: Error 5, da46 in disk23 marked FAIL
Apr 15 00:48:11 freenas GEOM_MULTIPATH: all paths in disk23 were marked FAIL, restore da17
Apr 15 00:48:12 freenas ZFS: vdev state changed, pool_guid=4748048525577549863 vdev_guid=9092319160336420981
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): READ(16). CDB: 88 00 00 00 00 01 44 e3 1f 58 00 00 01 00 00 00
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): CAM status: SCSI Status Error
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): SCSI status: Check Condition
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): Info: 0x144e31ff6
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): Field Replaceable Unit: 1
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): Command Specific Info: 0x4140
Apr 15 00:48:14 freenas (da17:mps0:0:50:0): Error 5, Unretryable error
Apr 15 00:48:14 freenas GEOM_MULTIPATH: Error 5, da17 in disk23 marked FAIL
Apr 15 00:48:14 freenas GEOM_MULTIPATH: all paths in disk23 were marked FAIL, restore da46
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): READ(16). CDB: 88 00 00 00 00 01 44 e3 1f 58 00 00 01 00 00 00
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): CAM status: SCSI Status Error
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): SCSI status: Check Condition
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): Info: 0x144e31ff6
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): Field Replaceable Unit: 1
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): Command Specific Info: 0x4140
Apr 15 00:48:16 freenas (da46:mps0:0:79:0): Error 5, Unretryable error
Apr 15 00:48:16 freenas GEOM_MULTIPATH: Error 5, da46 in disk23 marked FAIL
Apr 15 00:48:16 freenas GEOM_MULTIPATH: all paths in disk23 were marked FAIL, restore da17
Apr 15 00:48:18 freenas ZFS: vdev state changed, pool_guid=4748048525577549863 vdev_guid=9092319160336420981
 
Last edited:

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
I have got 2 Xyratex HB-1235 enclosures (i.e. 24 drives) connected to the HBA and I have been running stress tests for 48 hours now. Either I am not stressing them enough (just running rsync from my other server, both connected at 10Gbe to the switch) or my card is stable - no errors in the kernel log. Both hosts are running Ubuntu 18.04.
 

DanielWood

Member
Sep 14, 2018
44
17
8
I have got 2 Xyratex HB-1235 enclosures (i.e. 24 drives) connected to the HBA and I have been running stress tests for 48 hours now. Either I am not stressing them enough (just running rsync from my other server, both connected at 10Gbe to the switch) or my card is stable - no errors in the kernel log. Both hosts are running Ubuntu 18.04.

I dont start seeing issues typically until 10-20 hours into a scrub which is completely saturating the bus. Then again, I'm dealing with larger datasets (80+ TB of data, each set).
 

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
I'm dealing with larger datasets
I have 24x 4Tb in these 2 enclosures configured as 4 RAIDZ2 vdevs which gives me about 60Tb usable. It's less than yours but it is big enough for the full scrub process to take more than 24 hours. So I filled the volume up to 90% capacity and run the scrub twice... still no errors.

I've done some more digging and found this old but still quite interesting thread on the Illumos developer's mailing list where Keith Wesolowski (one of the guys who designed the HBA in question) mentioned that the card's firmware might be missing SATL (SCSI to ATA Translation Layer) code. Which would explain a lot - I am using the card with SAS drives and it seems to perform just fine.

I think I might keep the HBA for now.
 
Last edited:

BeTeP

Well-Known Member
Mar 23, 2019
653
429
63
Adaptec includes Adp80xxapp utility with firmware upgrades for their 6 and 7 series HBAs which recognizes the card. I was able to see the current firmware version (which was 1.11.0). Unfortunately it does not do much else. Finally I gave up all hope of being able to flash this card in Linux.

Then I just downloaded Solaris 11.4 for x86 and installed it on a spare disk. During very first boot after the installation it recognized the card and automatically updated the firmware to version 1.12.09 (the latest available).
 

xephael

New Member
Jun 4, 2019
7
0
1
Adaptec includes Adp80xxapp utility with firmware upgrades...Then I just downloaded Solaris 11.4 for x86 and installed it on a spare disk. During very first boot after the installation it recognized the card and automatically updated the firmware to version 1.12.09 (the latest available).
I have not downloaded Solaris but I'd presume it has the correct firmware in it's distribution to update the controller? I have found 8088.bin and 8081.bin from Adaptec and would presume this card requires 8001.bin.

Was there any special utility you had to run in Solaris?


I'm dealing with a What I presume is a Netapp/IBM PM8003 Rev3.0 labeled card. Has Quad QSFP SAS ports 6Gbs.
Code:
=============================================
Adp80xxapp Copyright 2014 PMC-Sierra
=============================================
Adaptec HBA Adp80xxapp 1.2.0.10624
(c) 1998-2014 PMC-Sierra, Inc. All Rights Reserved

Number of controllers = 2
CONTROLLER_ADDRESS | VENDOR_ID | DEVICE_ID | FIRMWARE_REVISION | BIOS_VERSION
0000:05:00.0            0x11f8        0x8001       01.12.06.00    1.2.0.�����
0000:04:00.0            0x11f8        0x8001       01.12.06.00    1.2.0.�����
Code:
04:00.0 Serial Attached SCSI controller: PMC-Sierra Inc. Device 8001 (rev 05)
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 64 bytes
   Interrupt: pin A routed to IRQ 17
   Region 0: Memory at fe6c0000 (64-bit, non-prefetchable) [size=64K]
   Region 2: Memory at fe6d0000 (64-bit, non-prefetchable) [size=64K]
   Region 4: Memory at fe6e0000 (32-bit, non-prefetchable) [size=64K]
   Region 5: Memory at fe6f0000 (32-bit, non-prefetchable) [size=64K]
   Expansion ROM at fe700000 [disabled] [size=1M]
   Capabilities: [40] Power Management version 3
       Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
       Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
       Address: 0000000000000000  Data: 0000
   Capabilities: [70] Express (v2) Endpoint, MSI 00
       DevCap:   MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
           ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 116.000W
       DevCtl:   Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
           RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
           MaxPayload 128 bytes, MaxReadReq 512 bytes
       DevSta:   CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:   Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
           ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
       LnkCtl:   ASPM Disabled; RCB 64 bytes Disabled- CommClk-
           ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:   Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
       DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
       DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
       LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
            Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
            Compliance De-emphasis: -6dB
       LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
            EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
   Capabilities: [ac] MSI-X: Enable+ Count=16 Masked-
       Vector table: BAR=0 offset=00002000
       PBA: BAR=0 offset=00004000
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:   DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UEMsk:   DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:   DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:   RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:   RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
       AERCap:   First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
   Kernel driver in use: pm80xx
   Kernel modules: pm80xx

05:00.0 Serial Attached SCSI controller: PMC-Sierra Inc. Device 8001 (rev 05)
   Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
   Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
   Latency: 0, Cache Line Size: 64 bytes
   Interrupt: pin A routed to IRQ 16
   Region 0: Memory at fe4c0000 (64-bit, non-prefetchable) [size=64K]
   Region 2: Memory at fe4d0000 (64-bit, non-prefetchable) [size=64K]
   Region 4: Memory at fe4e0000 (32-bit, non-prefetchable) [size=64K]
   Region 5: Memory at fe4f0000 (32-bit, non-prefetchable) [size=64K]
   Expansion ROM at fe500000 [disabled] [size=1M]
   Capabilities: [40] Power Management version 3
       Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
       Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
   Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
       Address: 0000000000000000  Data: 0000
   Capabilities: [70] Express (v2) Endpoint, MSI 00
       DevCap:   MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
           ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
       DevCtl:   Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
           RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
           MaxPayload 128 bytes, MaxReadReq 512 bytes
       DevSta:   CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
       LnkCap:   Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
           ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
       LnkCtl:   ASPM Disabled; RCB 64 bytes Disabled- CommClk-
           ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
       LnkSta:   Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
       DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
       DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
       LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
            Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
            Compliance De-emphasis: -6dB
       LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
            EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
   Capabilities: [ac] MSI-X: Enable+ Count=16 Masked-
       Vector table: BAR=0 offset=00002000
       PBA: BAR=0 offset=00004000
   Capabilities: [100 v1] Advanced Error Reporting
       UESta:   DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UEMsk:   DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
       UESvrt:   DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
       CESta:   RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
       CEMsk:   RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
       AERCap:   First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
   Kernel driver in use: pm80xx
   Kernel modules: pm80xx
 
Last edited: