X99 w/Chelsio T420-BT troubles (PCI-E lanes)

[zhoulander]

Running into some weird problems with Chelsio T420-BT cards and X99 setups. I'm not getting full speed in a PCI-E 3.0 x8 lane slot even though the card is a PCI-E 2.0 x8 card. The same card in a Z170 x8 slot works fine.

E5-1650 V3 (40 lanes)
ASROCK X99 Taichi
+ Graphics card
+ Chelsio T420-BT
+ Any other PCI-E add-in card

Motherboard spec page shows:
- 3 x PCI Express 3.0 x16 Slots (PCIE2 @ x16 mode; PCIE4 @ x16 mode; PCIE5 @ x0 mode) (PCIE2 @ x16 mode; PCIE4 @ x8 mode; PCIE5 @ x8 mode)*
*If you install CPU with 28 lanes, PCIE2/PCIE4/PCIE5 will run at x16/x0/x8 or x8/x8/x8.

In this configuration I get full line speed as expected
PCIE2 (x16) : GPU
PCIE4 (x16): Chelsio T420-BT
PCIE5 (x0): Empty

In this configuration I get 33% line speed (iperf around 3.35 Gbps)
PCIE2 (x16): GPU
PCIE4 (x8): Chelsio T420-BT
PCIE5 (x8): Any other card. Even a 1x Gigabit nic.

PCIE2 (x16): GPU
PCIE4 (x8): Any other card. Even a 1x Gigabit nic.
PCIE5 (x8): Chelsio T420-BT

In this configuration, I get full line speed
PCIE2 (x16) : Chelsio T420-BT
PCIE4 (x8): GPU
PCIE5 (x8): Any other card. Even a 1x Gigabit nic.

The Chelsio T420-BT is supposed to be PCI-E 2.0 x8 card. Why does it not like PCI-E 3.0 x8 lanes?
I tried forcing the slot to be 2.0 in BIOS and no help.

I tested the same in an ASUS X99 Deluxe and ran into the same type of performance problems when I manually switched the SLI/CF PCI-E selector to make it x8/x8/x8 or x16/x16/x8.

Similar weird performance issues with a ConnectX-2 in my E5-2670 V1 S2600CP2J Freenas server.
All of the slots in the S2600CP2J are x8 electrically. What is strange is in my Freenas server, I get 50% line speed, iPerf around 4.7 Gbps.

To throw another wrench in the mix, I have a ConnectX-2 in my E5-2670 V1 R2312 S2600GZ4GC server and can get 100% speed off of an x8 slot. The only difference is the R2312 has the latest BIOS installed, whereas the Freenas server as 1.06 to keep the PCI-E 3.0 from being restricted to whitelisted devices.


Now, look at this testing on Z170 platform!

i7 6700K
Gigabyte GA-Z170X-Gaming 7
PCIEX16: GPU
PCIEX8: 1 gig nic
PCIEX4: Chelsio T420-BT
iperf shows around 6.6-8.5 Gbps with the T420-BT on a x4 slot!

PCIEX16: GPU
PCIEX8: Chelsio T420-BT
PCIEX4: 1 gig nic
Full line speed. 9.4 Gbps, 99.93% network utilization.

This is really frustrating, since I wanted X99 for >4 cores, PCI-E lanes for NVME storage and also 10GBe networking.

I ordered some ConnectX-3 PCI-E 3 cards to do more testing. Chelsio T520/other 10GBase-T cards are a bit cost prohibitive, so this may be an incentive to finally pull fibre through the walls.

 

[zhoulander]

I booted up a live Ubuntu distro and discovered that it's only negotiating 1 lane on the X99 Taichi.

ubuntu@ubuntu:~$ sudo lspci -vv -s 04:00.0
04:00.0 Ethernet controller: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller
    Subsystem: Chelsio Communications Inc T420-BT Unified Wire Ethernet Controller
    Physical Slot: 6-1
    Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 45
    Region 0: Memory at fb380000 (64-bit, non-prefetchable) [size=256K]
    Region 4: Memory at fb44c000 (64-bit, non-prefetchable) [size=8K]
    Expansion ROM at fb180000 [disabled] [size=512K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
        Address: 0000000000000000  Data: 0000
        Masking: 00000000  Pending: 00000000
    Capabilities: [60] MSI-X: Enable- Count=8 Masked-
        Vector table: BAR=4 offset=00000000
        PBA: BAR=4 offset=00001000
    Capabilities: [6c] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <256ns, L1 <1us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <8us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
            Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
            Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
            EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

 

[zhoulander]

Tested a ConnectX-2 in the ASROCK X99 Taichi with the Chelsio T420 in the other slot. ConnectX-2 behaves properly linking at x8 width. Chelsio keeps linking at x1 width. Bye bye Chelsio.

 

[zir_blazer]

Why bye bye? I think that you have a solid case with enough tests performed to actually ask their support to try to reproduce it and figure out if they can fix it via Firmware or Drivers. Worth checking if it also happens with the other card you said. Heck, maybe is possible than is not the cards themselves, but that the LGA 2011/2011-3 Chipsets have a bug/errata that negotiates 8 lanes as 1 lane.
Bring in more info.


BTW, are you satisfied with the AsRock X99 Taichi? If I had money to burn, it was pretty much my top choice. Sadly, I don't have money, nor really need a new computer.