Hello all, first post here, but a casual builder of home servers for video editing and personal storage.
For a long time now, I've run a 10GbE DAC setup between my Windows workstation and an older Nas4Free (XigmaNAS) server. However, I've decided to give 40GbE a try. To do so, I've put together a new build TrueNAS server as an experiment, direct connected to the Windows workstation with a Chelsio T580-LP-CR each, connected with 56GbE-capable QSFP+ cables between them.
However, I'm only getting speeds nearing the max bandwidth of these cards in one direction, as follows:
Results from Windows to server:
Results from server to Windows (same iperf settings, three individual tests conducted):
(P.S.: I went with iperf 2.0.9 (Windows) here, as iperf3 was apparently unable to saturate the connection in excess of ~25GbE or so).
Box specs are as follows, for the record:
Windows:
Both NICs are set with MTU 9014.
I originally thought I was potentially looking at a PCIe bottleneck, but I also verified that both Windows and the TrueNAS server are running the cards at PCIe 3.0 x8:
Windows workstation:
Server - LnkCap and LnkSta look good for both ports:
In the course of this mess, I've booted the server up with both the BSD-based TrueNAS CORE and have also tried the Linux-based SCALE as well, with little change between these results (or various DD speed tests of the drive pools themselves, though that's an entirely separate discussion).
Also, while I'll admit the E5-2620 v3 isn't the hottest processor in the world, it does have 40 lanes and is supposed to provide full x8 speed to the PCIe slots, and not once was any core pegged during tests in TrueNAS.
However, I decided to see if I could isolate this from TrueNAS entirely and so booted a live Ubuntu USB to the server, and re-did the iperf tests. Virtually identical.
At this point, I'm stumped, and I'm also concerned that I'm falling into some confirmation bias about this being hardware related and not tuning. Greatly appreciate any assistance or suggestions where to look next.
-Kurt
P.S.: Full disclosure, a fair amount of what I've done here is from massive Google-fu to re-kickstart everything I'd since forgotten when I built my NAS4Free boxes. I'm 100% self taught on this, so please bear with me if I'm slow on some suggestions.
For a long time now, I've run a 10GbE DAC setup between my Windows workstation and an older Nas4Free (XigmaNAS) server. However, I've decided to give 40GbE a try. To do so, I've put together a new build TrueNAS server as an experiment, direct connected to the Windows workstation with a Chelsio T580-LP-CR each, connected with 56GbE-capable QSFP+ cables between them.
However, I'm only getting speeds nearing the max bandwidth of these cards in one direction, as follows:
Results from Windows to server:
Code:
iperf -c 192.168.2.20 -P 8
------------------------------------------------------------
Client connecting to 192.168.2.20, TCP port 5001
TCP window size: 208 KByte (default)
------------------------------------------------------------
[ 10] local 192.168.2.11 port 55568 connected with 192.168.2.20 port 5001
[ 9] local 192.168.2.11 port 55567 connected with 192.168.2.20 port 5001
[ 5] local 192.168.2.11 port 55563 connected with 192.168.2.20 port 5001
[ 3] local 192.168.2.11 port 55561 connected with 192.168.2.20 port 5001
[ 8] local 192.168.2.11 port 55566 connected with 192.168.2.20 port 5001
[ 4] local 192.168.2.11 port 55562 connected with 192.168.2.20 port 5001
[ 6] local 192.168.2.11 port 55564 connected with 192.168.2.20 port 5001
[ 7] local 192.168.2.11 port 55565 connected with 192.168.2.20 port 5001
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 4.00 GBytes 3.43 Gbits/sec
[ 9] 0.0-10.0 sec 4.08 GBytes 3.51 Gbits/sec
[ 5] 0.0-10.0 sec 6.90 GBytes 5.92 Gbits/sec
[ 8] 0.0-10.0 sec 4.42 GBytes 3.80 Gbits/sec
[ 4] 0.0-10.0 sec 5.26 GBytes 4.52 Gbits/sec
[ 6] 0.0-10.0 sec 4.01 GBytes 3.45 Gbits/sec
[ 7] 0.0-10.0 sec 5.35 GBytes 4.59 Gbits/sec
[ 3] 0.0-10.0 sec 4.16 GBytes 3.57 Gbits/sec
[SUM] 0.0-10.0 sec 38.2 GBytes 32.8 Gbits/sec
Code:
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 208 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57200
[ 5] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57202
[ 6] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57204
[ 7] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57206
[ 8] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57198
[ 9] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57208
[ 10] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57210
[ 11] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57212
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.70 GBytes 1.46 Gbits/sec
[ 5] 0.0-10.0 sec 1.71 GBytes 1.46 Gbits/sec
[ 6] 0.0-10.0 sec 1.68 GBytes 1.44 Gbits/sec
[ 7] 0.0-10.0 sec 1.70 GBytes 1.46 Gbits/sec
[ 8] 0.0-10.0 sec 1.72 GBytes 1.48 Gbits/sec
[ 9] 0.0-10.0 sec 1.72 GBytes 1.47 Gbits/sec
[ 10] 0.0-10.0 sec 1.72 GBytes 1.48 Gbits/sec
[ 11] 0.0-10.0 sec 1.79 GBytes 1.53 Gbits/sec
[SUM] 0.0-10.0 sec 13.8 GBytes 11.8 Gbits/sec
[ 4] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57214
[ 5] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57216
[ 6] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57218
[ 7] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57220
[ 8] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57222
[ 9] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57224
[ 10] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57226
[ 11] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57228
[ 4] 0.0-10.0 sec 1.68 GBytes 1.44 Gbits/sec
[ 5] 0.0-10.0 sec 1.65 GBytes 1.41 Gbits/sec
[ 6] 0.0-10.0 sec 1.64 GBytes 1.40 Gbits/sec
[ 7] 0.0-10.0 sec 1.68 GBytes 1.44 Gbits/sec
[ 8] 0.0-10.0 sec 1.66 GBytes 1.43 Gbits/sec
[ 9] 0.0-10.0 sec 1.65 GBytes 1.42 Gbits/sec
[ 10] 0.0-10.0 sec 1.64 GBytes 1.41 Gbits/sec
[ 11] 0.0-10.0 sec 1.67 GBytes 1.44 Gbits/sec
[SUM] 0.0-10.0 sec 13.3 GBytes 11.4 Gbits/sec
[ 4] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57232
[ 5] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57234
[ 6] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57236
[ 7] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57230
[ 8] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57238
[ 9] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57240
[ 10] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57242
[ 11] local 192.168.2.11 port 5001 connected with 192.168.2.20 port 57244
[ 4] 0.0-10.0 sec 1.65 GBytes 1.42 Gbits/sec
[ 5] 0.0-10.0 sec 1.66 GBytes 1.42 Gbits/sec
[ 6] 0.0-10.0 sec 1.74 GBytes 1.49 Gbits/sec
[ 7] 0.0-10.0 sec 1.67 GBytes 1.43 Gbits/sec
[ 8] 0.0-10.0 sec 1.72 GBytes 1.48 Gbits/sec
[ 9] 0.0-10.0 sec 1.62 GBytes 1.39 Gbits/sec
[ 10] 0.0-10.0 sec 1.66 GBytes 1.42 Gbits/sec
[ 11] 0.0-10.0 sec 1.73 GBytes 1.48 Gbits/sec
[SUM] 0.0-10.0 sec 13.5 GBytes 11.5 Gbits/secBox specs are as follows, for the record:
Windows:
- HP Z8 G4
- x2 Xeon Silver 4110's
- 128 GB DDR4
- 1TB NVMe boot drive
- Chelsio T580-LP-CR NIC installed in Slot 1 (3.0 x8)
- Supermicro X10SRH-CF
- E5-2620 v3
- 16GB Micron PC4-2400 ECC DDR4
- 6x SATA+SAS 3TB RAID-Z2 + 3x SATA SSD RAID-Z1 + 2x 1TB NVMe
- Dell H310 x2 (crossflashed)
- Chelsio T580-LP-CR NIC installed in Slot 4 (3.0 x8)
Both NICs are set with MTU 9014.
I originally thought I was potentially looking at a PCIe bottleneck, but I also verified that both Windows and the TrueNAS server are running the cards at PCIe 3.0 x8:
Windows workstation:
Code:
Name Segment Bus Device Function Slot NumaNode PcieLinkSpeed PcieLinkWidth Version
---- ------- --- ------ -------- ---- -------- ------------- ------------- -------
Ethernet 0 0 31 6 0 Unknown
Chelsio Left 0 166 0 4 1 1 8.0 GT/s 8 1.1
Chelsio Right 0 166 0 4 1 1 8.0 GT/s 8 1.1
Local Area Connection 0 4 0 0 0 2.5 GT/s 1 1.1
Ethernet 3 0 4 0 2 0 2.5 GT/s 1 1.1Server - LnkCap and LnkSta look good for both ports:
Code:
05:00.0 Ethernet controller: Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 40
Region 0: Memory at f9900000 (64-bit, non-prefetchable)
Region 2: Memory at f9880000 (64-bit, non-prefetchable)
Region 4: Memory at f998c000 (64-bit, non-prefetchable)
Expansion ROM at f9800000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2 Retimers- DRS-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedC ompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
EqualizationPhase2+ EqualizationPhase3+ LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00
Status: NegoPending- InProgress-
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 1
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Depen dency Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X table
05:00.1 Ethernet controller: Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 44
Region 0: Memory at f9780000 (64-bit, non-prefetchable)
Region 2: Memory at f9700000 (64-bit, non-prefetchable)
Region 4: Memory at f998a000 (64-bit, non-prefetchable)
Expansion ROM at f9680000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 2
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Depen dency Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X table
05:00.2 Ethernet controller: Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 45
Region 0: Memory at f9600000 (64-bit, non-prefetchable)
Region 2: Memory at f9580000 (64-bit, non-prefetchable)
Region 4: Memory at f9988000 (64-bit, non-prefetchable)
Expansion ROM at f9500000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 3
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Depen dency Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X table
05:00.3 Ethernet controller: Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin D routed to IRQ 46
Region 0: Memory at f9480000 (64-bit, non-prefetchable)
Region 2: Memory at f9400000 (64-bit, non-prefetchable)
Region 4: Memory at f9986000 (64-bit, non-prefetchable)
Expansion ROM at f9380000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=8 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 4
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Depen dency Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X table
05:00.4 Ethernet controller: Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 40
Region 0: Memory at f9300000 (64-bit, non-prefetchable)
Region 2: Memory at f8000000 (64-bit, non-prefetchable)
Region 4: Memory at f9984000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 4s to 13s, TimeoutDis- LTR- OBFF Di sabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 5
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Depende ncy Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X table
05:00.5 SCSI storage controller: Chelsio Communications Inc T580-LP-CR Unified W ire Storage Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 44
Region 0: Memory at f9280000 (64-bit, non-prefetchable)
Region 2: Memory at f9200000 (64-bit, non-prefetchable)
Region 4: Memory at f9982000 (64-bit, non-prefetchable)
Expansion ROM at f9180000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=40 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 6
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Depende ncy Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X table
05:00.6 Fibre Channel: Chelsio Communications Inc T580-LP-CR Unified Wire Storag e Controller
Subsystem: Chelsio Communications Inc Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Step ping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 45
Region 0: Memory at f9100000 (64-bit, non-prefetchable)
Region 2: Memory at f9080000 (64-bit, non-prefetchable)
Region 4: Memory at f9980000 (64-bit, non-prefetchable)
Expansion ROM at f9000000 [disabled]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot -,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimite d, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowe rLimit 25.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransP end-
LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s (ok), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- L TR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerRe ductionInit-
FRS- TPHComp+ ExtTPHComp-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizatio nRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [b0] MSI-X: Enable- Count=40 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00001000
Capabilities: [d0] Vital Product Data
Not readable
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalEr r+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkC ap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [170 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 7
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [1a0 v1] Secondary PCI Express
LnkCtl3: LnkEquIntrruptEn- PerformEqu-
LaneErrStat: 0
Capabilities: [1c0 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Depende ncy Link: 00
VF offset: 8, stride: 4, Device ID: 5810
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
Region 4: Memory at 0000000000001000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [200 v1] Transaction Processing Hints
Interrupt vector mode supported
Steering table in MSI-X tableAlso, while I'll admit the E5-2620 v3 isn't the hottest processor in the world, it does have 40 lanes and is supposed to provide full x8 speed to the PCIe slots, and not once was any core pegged during tests in TrueNAS.
However, I decided to see if I could isolate this from TrueNAS entirely and so booted a live Ubuntu USB to the server, and re-did the iperf tests. Virtually identical.
At this point, I'm stumped, and I'm also concerned that I'm falling into some confirmation bias about this being hardware related and not tuning. Greatly appreciate any assistance or suggestions where to look next.
-Kurt
P.S.: Full disclosure, a fair amount of what I've done here is from massive Google-fu to re-kickstart everything I'd since forgotten when I built my NAS4Free boxes. I'm 100% self taught on this, so please bear with me if I'm slow on some suggestions.
Last edited:
