Identifying some curious Xilinx/Solarflare cards.

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

astronomy8

New Member
Sep 2, 2017
12
3
3
123
So, I wanted some 10Gb NICs at home. We've had a pandemic raging here for a few years and I've been rather bored. After balking at the premium X710-DA2/DA4 and X520-DA2 cards still command on ebay, I remembered how a few years ago I worked with some SolarFlare SFN8522 and SFN8542 cards on a project for work. They worked rather well. I found someone selling what appeared to be 8522's for $8.99 each. If I ordered more than 5, I got free shipping. So, I did. Thing is, they're not SFN8522's. Clearly, I didn't look very closely before purchasing. Does anyone recognize these?

I removed the passive heatsink on one of the cards to confirm it was fitted with an SFC9250. The board has eight 1GB chips of Micron DDR4 RAM on the back. I haven't removed the large, heatsink in the center as it appears to be epoxied in place. I now suspect there's a Xilinx FPGA or FPGA-ish part underneath.

As you can see from the PCB markings in the attached photos, it's Copyright 2019 Xilinx. One identifier claims to be an SF30-050210 and another label claims, "Regulatory part number SR242." While it's possible I'm just poor at googling, searches for combinations of solarflare/xilinx and sr242/sf30 have not revealed anything of note.

I think they are or are very similar to the Xilinx Alveo U25, but without the pretty, red heatsink. Note the keyed card-edge connector at the top, center of the PCB identified as J8 and another connector at the bottom of the PCB on the opposite end from the mounting bracket identified as J5. The Xilinx document, "Alveo Programming Cable: The User Guide" UG1377 shows how to use the top, J8 connector. I have no idea what the card-bottom, J5 is for.

Installed in a machine, they're not recognized by any driver, niether the in-kernel sfc.ko nor the out-of-tree driver from solarflare/xilinx. I see they identify as vendor 0x1924, SolarFlare, with a device id of 0x0b13. Note, 0x0b13 is only one bit different from device id 0x0b03 for the SFC9250 PF in the sfc kernel driver. They do not appear to have any bootrom despite having two 128MB micron flash chips on board. Without a working sfc driver, sfboot, sfupdate, don't identify the card.

I did add 0x0b13 to the efx_pci_table and rebuilt sfc.ko just to see if it might magically, you know, just work. They did not.

So, in summary, does anyone know what these are? If they are Alveo U25's, then why don't they have the cool, modern alveo u25 red heatsink?

It'd be novel if I could use these and I welcome any suggestions to that end. If not, they're not a total loss - I could always turn them in to taco holders or laundry room wall accents.
 

Attachments

Last edited:

astronomy8

New Member
Sep 2, 2017
12
3
3
123
Thank you for your reply!

So this
Code:
#define    EFX_PCI_DEVID_MEDFORD2_PF_UNINIT    0x0B13
#define    EFX_PCI_DEVID_MEDFORD2            0x0B03    /* SFC9250 PF */
#define    EFX_PCI_DEVID_MEDFORD2_VF        0x1B03    /* SFC9250 VF */
is in dpdk source code: drivers/common/sfc_efx/base/efx.h (v21.11)
Dunno if that helps any. Seems like it's a family called a Medford 2?
Actually, that does help. Specifically, the _UNINIT part of the #define which lead me to:
Code:
#if EFSYS_OPT_MEDFORD2
        case EFX_PCI_DEVID_MEDFORD2_PF_UNINIT:
            /*
             * Hardware default for PF0 of uninitialised Medford2.
             * manftest must be able to cope with this device id.
             */
        case EFX_PCI_DEVID_MEDFORD2:
        case EFX_PCI_DEVID_MEDFORD2_VF:
            *efp = EFX_FAMILY_MEDFORD2;
            *membarp = EFX_MEM_BAR_MEDFORD2;
            return (0);
#endif /* EFSYS_OPT_MEDFORD2 */
in dpdk source code: drivers/common/sfc_efx/base/efx_nic.c (v21.11)

I don't know what manftest refers to in the comment. Possibly a binary tool or just a shorthand for a process or subsystem inside the card. Searching for manftest elsewhere in the code finds some interesting clues, such as: https://code.dpdk.org/dpdk/v21.11/source/drivers/common/sfc_efx/base/efx_regs_mcdi.h#L9433

A hypothesis would be that the card has a corrupt or empty NVRAM of some sort. I noticed the kernel dmesg errors appeared after using lspci, suggesting further uninitialized values.
Code:
[  243.858867] pci 0000:04:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
[  243.990854] pci 0000:04:00.1: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
Makes me wonder if:
1. Perhaps the one card I installed is defective and I should try the others.
2. This is a prototype or semi-custom board that relied on host software to bring it up?
3. These cards are expected to be programmed externally to be used? Maybe I should install the Xilinx VIRTIS PDK? (That sounds like a lot of fun....)
4. Perhaps the DPDK driver can get the card bootstrapped up far enough to investigate further? I'm curious what, if anything, is on those flash chips.

I think I'll start by just swapping in the other cards I received to see if they all are uninitialized.

Either way, thank you for that hint, it looks very promising!
 

astronomy8

New Member
Sep 2, 2017
12
3
3
123
So, some more observations. I realized the center heatsink wasn't glued/epoxied down and removed it to discover a Xilinx Zynq XCZU19EG-FFVC1760AAZ, an UltraScale+ FPGA. This FPGA features 522,720 CLB LUTs and the Alveo U25 specs claim 523K Look-Up Tables. Also, I can't find any other Xilinx device with this card-edge programming connector. Either these are some weirdo development or prototype cards or they're just naked U25's with a fan and no programmed NVRAM.

One thing that throws me is that there is definitely 8GB of RAM DDR4 RAM chips on the back, but U25 spec sheet claims the U25 is only equipped with 6GB in a strange 2GB + 4GB configuration (1x 2GB x 40 DDR4-2400, 1x 4GB x 72 DDR4-2400). That said, I suppose it's entirely possible they're doing some kind of weirdo ECC or parity? Ie, the 2GB group is two 1GB chips with an additional parity and the 4GB group is four 1GB chips with an additional parity. That conjecture is somewhat buttressed by the physical arrangement of the RAM on the back of the board: three on one side and five on the other.

I tried a few other boards and they all come up with the device id 0x0b13 which I now believe means the UNINIT configuration. Also, as can be seen from the PCI-E differential pairs from the card-edge connector, 8x lanes are routed to the SFC9250 and the other 8x are to the Zynq. I noticed that lspci -vvv shows the buswidth as 8x rather than 16x despite being plugged in to a known 16x slot. The U25 datasheet claims it can operate as either Gen3 x16 or 2xGen3 x8 bifurcated. As I'm not certain the X10SRL-f I was using to check these cards out could recognize or do the bifurcation, I installed them in a machine I know can handle it. I was hoping to see a second device accompanying the SFC9250 on the bus, but I was sadly disappointed.

Worse, it seems the U25 doesn't have any facility for programming itself and I'll need to buy the Alveo USB programming adapter.

My next step will be to install the Xilinx XRT and Vitis software. I'm hoping either of those might be able to further initialize the card? It would appear from reading the sfc.ko and solarflare dpdk driver source that there is a facility for the host to at least configure and bring up the SFC9250.

Yes, all I wanted when I found these was a cheap, dual SFP+ ethernet card and instead it seems I bought an adventure.

The list prices for FPGAs always amuse me: XCZU19EG-1FFVC1760I Xilinx Inc. | Integrated Circuits (ICs) | DigiKey

Cheers.
 

Attachments

victhor393

New Member
Feb 13, 2022
26
11
3
Hi, I finally decided to go and sign up to STH forums after I saw this post. I got one of these cards too, except I bought it deliberately after realizing it was a U25. I wonder if we bought from the same seller...

I don't know if you decided to do this, but I went through the effort of signing up for access to the U25 product page, and unfortunately there is nothing there regarding initializing a card that's in this state.

Unfortunately, my testing system's motherboard (currently) doesn't support bifurcation, so I only have access to the NIC, which is not functional, like yours. I tried bridging the "NO FLASH" jumper but that didn't do anything, other than make one of the functions of the Solarflare "0b13" device not show up on the PCI bus.

I thought of posting on the Xilinx forums about it, but without bifurcation, I figured any help from them would be a dead end anyway - need to bring up the PL in the Zynq to get the ports working, so even if the NIC was flashed we still would have no connectivity.

If I'm not mistaken, there is a facility in the hardened PCIe IP of Xilinx FPGAs that lets them power up quickly with a VID/PID, while the host is booting, hopefully before the host firmware enumerates the bus. This lets the enumeration occur before the bitstream can be fully loaded into the fabric, or even to allow the host to load bitstream directly. I imagine it must be similar on the Zynq PL, but I can't know for sure, as I never used it. If there is no enumeration from the Zynq during boot, then I'm going to guess that its flash (or flashes, as there are 2 ICs on board) is empty too.

I wonder if re-enumerating PCIe devices after boot would help clear this up, maybe the PCIe IP is initializing late, for some reason?

Another good idea would be to poke around for a serial port. There is an edge connector right next to the Zynq. I haven't taken a look at it. It's likely 1.8V (don't have anything compatible handy) and probing the card while it's loose on a slot (I only have a low profile bracket, my cases are all normal height, and I don't have a PCIe extension cable) sounds like a very bad idea.

There is a 16MB serial NOR flash on the back, right next to the NIC. It's in a SO-8 package so it shouldn't be too difficult to remove or clip and flash directly, if recovery through PCIe turns out to be impossible. Though I imagine it would be useful to dump the flash of a "normal" working card from Solarflare with the same ASIC, to figure out if the firmware requires some sort of partitioning scheme.

Regarding the amount of memory on the card: I agree with your guess, the extra memory is for ECC, those bus configurations are familiar to me, typical arrangements for ECC on 32-bit and 64-bit wide memory buses.

Just throwing some ideas into the wind here, also it's good to know I'm not alone in this, heh.
 
Last edited:

anything1233

Just a Member
Dec 29, 2020
41
18
8
Dumb question, but have you tried using the Xilinx programming port yet — the one with 14+12 pins? It should let you write PL regardless of the PCIe connectivity…
 

astronomy8

New Member
Sep 2, 2017
12
3
3
123
The programmer just arrived, I've got one of the cards installed in a second machine with the ribbon cable exiting through an empty expansion slot. I'm installing the Xilinx Vitis software now. I doubt I'll have much opportunity to play with it until this weekend, unfortunately.

Worse, it seems like the U25 files aren't publically available. I suppose I'll try applying to their program.
 

astronomy8

New Member
Sep 2, 2017
12
3
3
123
So, I tried enumerating all the devices on the scan chain and it finds the xczu19, but complains that it's not programmed and that the PL (programmable logic) power status is off. I'll have to try more tomorrow.
 

astronomy8

New Member
Sep 2, 2017
12
3
3
123
The bitstream files aren't distributed there, either, I just checked. Only some libraries to interface with the card. I think you may have to compile one of their examples from the Github page? GitHub - Xilinx/Vitis_Accel_Examples: Vitis_Accel_Examples
Unfortunately, you have to agree to an NDA to get not only some of the documentation, but some of the other files as well. There's a specific blob required to build any of the components.


I'll submit a request and see where we get.
 

victhor393

New Member
Feb 13, 2022
26
11
3
If he has the time and interest to look into this, why not? Seems like he's done a lot of work figuring out those Mellanox cards.
 

mwrnd

New Member
Jul 5, 2022
25
18
3
I will try to help out as best I can.

@victhor393
> my testing system's motherboard (currently) doesn't support bifurcation

It is incredibly useful to have some PCIe Riser boards. Consider purchasing two PCIe Riser boards and a PCIe Power Connector 6-Pin Extension Cable. That will allow you to test PCIe devices at a distance from your motherboard. All PCIe devices should work at x1.

PCIe_Riser_x1_x16.png

If you have two Riser boards and are comfortable soldering, you can create a DIY bifurcation adapter. The PCIe x1 riser cable connects PERSTn(RESET), REFCLK+/-, RX+/-, TX+/- and GND. There is a single PERSTn and REFCLK per PCIe connector. You could then cut up the second PCIe x1 cable and only solder the RX +/-, TX +/-, and GND pins to the 8th lane of the Riser board. The idea is to convert a PCIe x16 to two x1's. Unless your motherboard has seperate gated clocks and resets for each PCIe connector, this should work.

Note that the first and last PRSNT (Presence Detect) pins in PCIe are connected together to inform the host of the add-in card's lane width. You may need to short the presence pins (connect PRSNT1 and PRSNT2).

> I don't own the required adapter or programmer

From the Alveo Programming Cable User Guide (ug1377) it is just a pair of FTDI ICs and some voltage translation. Any Xilinx-Compatible Cable or clone should be usable.

Alveo_JTAG_is_Dual_FT2232H.png

PCIe has 1mm-spaced board-edge pads. From the photos provided it looks like the Alveo JTAG Programming connector is less than that, which would be 0.8mm. Xilinx uses Samtec connectors for all their high-speed interfaces so the Alveo U25 edge-board connector is likely a Samtec MEC connector. I believe it is a MEC8-113-02-L-D-RA1 (DigiKey).

Alveo_JTAG_has_0.8mm-Spaced_Pins.png

Figure 5 from ug1377:

Alveo_JTAG_Connector_Adapter.png

Anyone have a picture of the back of this Adapter Card?

Xilinx unfortunately does not intend for customers to use the U25 SmartNIC for FPGA-only designs.

Some References I believe may be useful:
Building U25 XRT from Scratch Solves some Issues
SmartNIC Drivers and Software
Alveo Debugging Steps
Vitis Getting Started
Zynq Programmable Logic (PL) Programming
Zynq Ulstrascale+ Technical Reference Manual
Vitis AI DPU for U25 is a complete example design but it depends on a working XRT Shell, [1], [2], [3].
Reverting Alveo Card to Factory Image does not include U25 notes.

The standard usage of a Zynq is to program the PS (Processing System) which then programs the PL (Programmable Logic). I found some conflicting information as to whether it is possible to program and use the PL portion of the FPGA without using the PS, [1], [2], [3], [4].
 

mwrnd

New Member
Jul 5, 2022
25
18
3
@astronomy8
> I noticed that lspci -vvv shows the buswidth as 8x

@victhor393
> I only have access to the NIC

Please post the output of sudo lspci -vvvvxxxx

Can you install XRT? The U25 Member Page should have an XRT Runtime or Deployment Target Platform. xbutil can then be used to program the U25. xclbinutil has --info. The following is from ug1456 via this question.


ug1456-u25.png


If your card does not work with XRT tools, it may be missing its shell. Try xrt_202020.2.8.743_18.04-amd64-xrt.deb from Flashing Shell to U25. Does the U25 Support Site include xilinx_u25_gen3x8_xdma_base_1 or similar?

Please post the output of sudo /opt/xilinx/xrt/bin/xbmgmt flash --scan --verbose

@astronomy8 mentioned U25N-SmartNIC-Solution which includes JTAG instructions for programming the shell. Once the Maintenance Connector Interface (Alveo Programming Cable) pinout is found, you will be able to connect a standard Xilinx-Compatible 1.8V JTAG or clone.

Once XRT and the Shell are working, Vitis-AI 1.3.2 includes dpu.xclbin for the Alveo U25; a working example design. Note an "xclbin contains sections describing user compiled acceleration engines/kernels, memory subsystems, clocking information etc. It also contains an FPGA bitstream for the user partition, UUIDs, platform name, etc."

Vitis 1.4.1 contains xRNN for "unified xRNN runtime for U25 & U50LV" but no explicit U25 notes. The U25 may not be supported in newer versions of Vitis-AI. Best to clone v1.3.2 and use an older docker image.

Code:
git clone --branch 1.3.2 --recursive --single-branch https://github.com/Xilinx/Vitis-AI.git
cd Vitis-AI
git checkout 1.3.2
git describe --tags

docker pull xilinx/vitis-ai-cpu:1.3.598
./docker_run.sh xilinx/vitis-ai-cpu:1.3.598
Check out Vitis Application Development Flow and Deep Learning Processor Unit (DPU).
 

astronomy8

New Member
Sep 2, 2017
12
3
3
123
@astronomy8
> I noticed that lspci -vvv shows the buswidth as 8x

@victhor393
> I only have access to the NIC

Please post the output of sudo lspci -vvvvxxxx
Code:
0000:17:00.0 Ethernet controller: Solarflare Communications Device 0b13 (rev 01)
    Subsystem: Solarflare Communications Device 0000
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 255
    NUMA node: 0
    Region 0: Memory at b5000000 (64-bit, non-prefetchable) [disabled] [size=8M]
    Region 2: Memory at b5804000 (64-bit, non-prefetchable) [disabled] [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s (ok), Width x8 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS+ TPHComp+ ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS+
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+ EqualizationPhase1+
             EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable- Count=64 Masked-
        Vector table: BAR=2 offset=00000000
        PBA: BAR=2 offset=00002000
    Capabilities: [d0] Vital Product Data
        Not readable
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [148 v1] Device Serial Number 00-0f-53-ff-ff-00-00-00
    Capabilities: [158 v1] Power Budgeting <?>
    Capabilities: [168 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 1
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [178 v1] Secondary PCI Express
        LnkCtl3: LnkEquIntrruptEn- PerformEqu-
        LaneErrStat: 0
    Capabilities: [1a8 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration-, Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy+
        IOVSta:    Migration-
        Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency Link: 00
        VF offset: 16, stride: 1, Device ID: 1913
        Supported Page Size: 00000553, System Page Size: 00000000
        Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
        Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Capabilities: [1e8 v1] Transaction Processing Hints
        Interrupt vector mode supported
        Device specific mode supported
        No steering table available
    Capabilities: [274 v1] Access Control Services
        ACSCap:    SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        ACSCtl:    SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
    Capabilities: [27c v1] Latency Tolerance Reporting
        Max snoop latency: 0ns
        Max no snoop latency: 0ns
    Capabilities: [284 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
              PortCommonModeRestoreTime=10us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
               T_CommonMode=0us LTR1.2_Threshold=0ns
        L1SubCtl2: T_PwrOn=10us
    Capabilities: [294 v1] Dynamic Power Allocation <?>
    Capabilities: [2c4 v1] Readiness Time Reporting <?>
    Capabilities: [2d0 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
    Capabilities: [308 v1] Precision Time Measurement
        PTMCap: Requester:+ Responder:- Root:-
        PTMClockGranularity: Unimplemented
        PTMControl: Enabled:- RootSelected:-
        PTMEffectiveGranularity: Unknown
    Capabilities: [314 v1] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
00: 24 19 13 0b 40 01 10 00 01 00 00 02 08 00 80 00
10: 04 00 00 b5 00 00 00 00 04 40 80 b5 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 24 19 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
40: 01 50 c3 5f 08 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 b0 02 00 23 87 00 10 36 29 09 00 03 3d 43 00
80: 40 00 83 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 1f 18 0c 80 00 00 00 00 0e 00 00 80
a0: 03 00 1f 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 d0 3f 00 02 00 00 00 02 20 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 01 00 82 14 00 00 00 00 00 00 10 00 30 20 06 00
110: 00 20 00 00 00 20 00 00 a0 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 03 00 81 15 00 00 00 ff
150: ff 53 0f 00 00 00 00 00 04 00 81 16 00 00 00 00
160: 00 00 00 00 01 00 00 00 0e 00 81 17 00 01 00 00
170: 00 00 00 00 00 00 00 00 19 00 81 1a 00 00 00 00
180: 00 00 00 00 00 27 00 27 00 27 00 27 00 27 00 27
190: 00 27 00 27 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f
1a0: 00 7f 00 7f 00 00 00 00 10 00 81 1e 02 00 00 00
1b0: 10 00 00 00 00 00 00 00 00 00 00 00 10 00 01 00
1c0: 00 00 13 19 53 05 00 00 00 00 00 00 04 00 00 00
1d0: 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 17 00 41 27 07 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
270: 00 00 00 00 0d 00 c1 27 00 10 00 00 18 00 41 28
280: 00 00 00 00 1e 00 41 29 1f 0a 28 00 00 00 00 00
290: 28 00 00 00 16 00 41 2c 1f 11 80 ff 00 00 00 00
2a0: 00 01 00 00 20 1f 1e 1d 1c 1b 1a 19 18 17 16 15
2b0: 14 13 12 11 10 0f 0e 0d 0c 0b 0a 09 08 07 06 05
2c0: 04 03 02 01 22 00 01 2d 00 00 00 80 00 00 00 00
2d0: 0b 00 81 30 01 00 81 03 00 00 00 00 10 00 00 00
2e0: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00
2f0: c0 00 c0 00 c0 00 c0 00 01 00 00 00 00 00 00 00
300: 00 00 00 00 00 00 00 00 1f 00 41 31 01 00 00 00
310: 00 00 00 00 0b 00 01 00 03 00 41 05 00 09 00 00
320: 02 00 00 00 6a cc 6d 97 5a 02 00 00 00 00 00 00
330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
360: 0b 00 00 00 0c 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 35 00 a0 00 ff ff ff ff 04 00 00 00 00 40 40 0a
710: 20 01 1f 00 00 00 00 78 01 c0 01 20 80 02 08 00
720: 00 00 00 00 00 00 00 00 91 14 4e 03 10 00 00 08
730: c0 f2 07 00 18 f0 07 00 ff ff 0f 00 00 00 00 00
740: 0f 00 00 00 00 00 00 00 00 01 24 c5 08 20 20 05
750: 00 00 20 05 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 80 10 01 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 ff 1f 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00
890: 01 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: a0 aa aa aa 00 00 00 00 00 00 00 00 40 00 00 00
8b0: 00 00 00 00 00 00 00 00 ff ff 00 00 03 00 00 00
8c0: 80 00 00 00 44 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 2a 30 39 34 2a 2a 61 67
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 0a 00 00 00 d2 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

0000:17:00.1 Ethernet controller: Solarflare Communications Device 0b13 (rev 01)
    Subsystem: Solarflare Communications Device 0000
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin B routed to IRQ 255
    NUMA node: 0
    Region 0: Memory at b4800000 (64-bit, non-prefetchable) [disabled] [size=8M]
    Region 2: Memory at b5800000 (64-bit, non-prefetchable) [disabled] [size=16K]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [70] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
        DevCtl:    CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
        LnkCap:    Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
            ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
        LnkCtl:    ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s (ok), Width x8 (downgraded)
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
             10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
             EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
             FRS+ TPHComp+ ExtTPHComp-
             AtomicOpsCap: 32bit- 64bit- 128bitCAS-
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
             AtomicOpsCtl: ReqEn-
        LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
             EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
             Retimer- 2Retimers- CrosslinkRes: unsupported
    Capabilities: [b0] MSI-X: Enable- Count=64 Masked-
        Vector table: BAR=2 offset=00000000
        PBA: BAR=2 offset=00002000
    Capabilities: [d0] Vital Product Data
        Not readable
    Capabilities: [100 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
        AERCap:    First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
            MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
        HeaderLog: 00000000 00000000 00000000 00000000
    Capabilities: [148 v1] Device Serial Number 00-0f-53-ff-ff-00-00-00
    Capabilities: [158 v1] Power Budgeting <?>
    Capabilities: [168 v1] Alternative Routing-ID Interpretation (ARI)
        ARICap:    MFVC- ACS-, Next Function: 0
        ARICtl:    MFVC- ACS-, Function Group: 0
    Capabilities: [1a8 v1] Single Root I/O Virtualization (SR-IOV)
        IOVCap:    Migration-, Interrupt Message Number: 000
        IOVCtl:    Enable- Migration- Interrupt- MSE- ARIHierarchy-
        IOVSta:    Migration-
        Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency Link: 01
        VF offset: 16, stride: 1, Device ID: 1913
        Supported Page Size: 00000553, System Page Size: 00000000
        Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
        Region 2: Memory at 0000000000000000 (64-bit, non-prefetchable)
        VF Migration: offset: 00000000, BIR: 0
    Capabilities: [1e8 v1] Transaction Processing Hints
        Interrupt vector mode supported
        Device specific mode supported
        No steering table available
    Capabilities: [274 v1] Access Control Services
        ACSCap:    SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        ACSCtl:    SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
    Capabilities: [294 v1] Dynamic Power Allocation <?>
    Capabilities: [2c4 v1] Readiness Time Reporting <?>
    Capabilities: [2d0 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
    Capabilities: [314 v1] Vendor Specific Information: ID=0003 Rev=1 Len=054 <?>
00: 24 19 13 0b 40 01 10 00 01 00 00 02 08 00 80 00
10: 04 00 80 b4 00 00 00 00 04 00 80 b5 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 24 19 00 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00
40: 01 50 c3 5f 08 00 00 00 00 00 00 00 00 00 00 00
50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 10 b0 02 00 23 87 00 10 36 29 09 00 03 3d 43 00
80: 40 00 83 10 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 1f 18 0c 80 00 00 00 00 0e 00 00 00
a0: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 11 d0 3f 00 02 00 00 00 02 20 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
100: 01 00 82 14 00 00 00 00 00 00 10 00 30 20 06 00
110: 00 20 00 00 00 20 00 00 a0 00 00 00 00 00 00 00
120: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
140: 00 00 00 00 00 00 00 00 03 00 81 15 00 00 00 ff
150: ff 53 0f 00 00 00 00 00 04 00 81 16 00 00 00 00
160: 00 00 00 00 01 00 00 00 0e 00 81 1a 00 00 00 00
170: 00 00 00 00 00 00 00 00 19 00 81 1a 00 00 00 00
180: 00 00 00 00 00 27 00 27 00 27 00 27 00 27 00 27
190: 00 27 00 27 00 7f 00 7f 00 7f 00 7f 00 7f 00 7f
1a0: 00 7f 00 7f 00 00 00 00 10 00 81 1e 00 00 00 00
1b0: 00 00 00 00 00 00 00 00 00 00 01 00 10 00 01 00
1c0: 00 00 13 19 53 05 00 00 00 00 00 00 04 00 00 00
1d0: 00 00 00 00 04 00 00 00 00 00 00 00 00 00 00 00
1e0: 00 00 00 00 00 00 00 00 17 00 41 27 07 00 00 00
1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
220: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
270: 00 00 00 00 0d 00 41 29 00 00 00 00 00 00 00 00
280: 00 00 00 00 1e 00 41 29 1f 0a 28 00 00 00 00 00
290: 28 00 00 00 16 00 41 2c 1f 11 80 ff 00 00 00 00
2a0: 00 01 00 00 20 1f 1e 1d 1c 1b 1a 19 18 17 16 15
2b0: 14 13 12 11 10 0f 0e 0d 0c 0b 0a 09 08 07 06 05
2c0: 04 03 02 01 22 00 01 2d 00 00 00 80 00 00 00 00
2d0: 0b 00 41 31 01 00 81 03 00 00 00 00 10 00 00 00
2e0: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00
2f0: c0 00 c0 00 c0 00 c0 00 01 00 00 00 00 00 00 00
300: 00 00 00 00 00 00 00 00 00 00 40 31 00 00 00 00
310: 00 00 00 00 0b 00 01 00 03 00 41 05 00 09 00 00
320: 02 00 00 00 ac de 7b 97 5a 02 00 00 00 00 00 00
330: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
340: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
360: 0b 00 00 00 0c 00 00 00 00 00 00 00 00 00 00 00
370: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
390: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
430: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
440: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
450: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
460: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
470: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
490: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
520: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
530: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
590: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
600: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
620: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
630: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
640: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
650: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
660: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
670: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
690: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
700: 35 00 a0 00 ff ff ff ff 04 00 00 00 00 40 40 0a
710: 20 01 1f 00 00 00 00 78 01 c0 01 20 80 02 08 00
720: 00 00 00 00 00 00 00 00 91 44 7d 03 10 00 00 08
730: c0 f2 07 00 18 f0 07 00 ff ff 0f 00 00 00 00 00
740: 0f 00 00 00 00 00 00 00 00 01 24 c5 08 20 20 05
750: 00 00 20 05 00 00 00 00 00 00 00 00 00 00 00 00
760: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
770: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
790: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
800: 00 00 00 00 00 00 00 00 00 00 00 00 80 10 01 00
810: 00 00 00 00 00 00 00 00 00 00 00 00 ff 1f 00 00
820: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
830: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
840: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
880: 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00
890: 01 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8a0: a0 aa aa aa 00 00 00 00 00 00 00 00 40 00 00 00
8b0: 00 00 00 00 00 00 00 00 ff ff 00 00 03 00 00 00
8c0: 80 00 00 00 44 00 00 00 00 00 00 00 00 00 00 00
8d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8f0: 00 00 00 00 00 00 00 00 2a 30 39 34 2a 2a 61 67
900: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
910: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
920: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
930: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
940: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
950: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
960: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
970: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
980: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
990: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ad0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b40: 0a 00 00 00 d2 00 00 00 00 00 00 00 00 00 00 00
b50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ba0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
be0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
bf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ca0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
cf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
da0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
db0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
dd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
de0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
df0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ea0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
eb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Note, the usual device ID for the SFC9250/X2 when working correctly is the 0b03, where this device reports itself as 0b13. See the comment in #2 suggesting this is the uninitialized device id.

Further, a kernel error was reported failing to read the VPD section 0xd0.
 
  • Like
Reactions: mwrnd

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
Looks like you can actually download some (X2 and X3) firmware and firmware uploaders freely at: NIC Software and Drivers

If you check the contents of the packages they contain a bunch of tools and some have embedded firmware payloads it seems (AES encrypted, but the key seems to be in the tools as well - I didn't have a box on hand to test it on yet and I don't have a card myself).

The URLs point to vendor-specific names which to me sounds like everyone is using the same SDK to build their packages from, which in turn means that unless they go out of their way to make their exported packages not include tools and firmware payloads, most vendors release packages similar to classic BIOS updates where a universal program, a config file and a payload file.

Since SmartNICs (essentially pre-DPU) were sold by a bunch of brands, it seems reasonable to assume a lot of them had similar hardware but tried to differentiate on integration with their platform or acceleration technologies. If you don't care for the acceleration, that means the NIC part should be usable no matter what you flash, considering that doesn't actually change and if you don't use acceleration features the data path doesn't bother going via IP blocks that do special things (like application-level protocol acceleration).

The SFUpdate package seems to contain VxWorks payloads, so that must be targeting the zynq (in part), sfboot seems to contain a blowfish encrypted payload, probably a boot loader from the smell of it.

Edit: because I couldn't help myself I did a bit of binwalk/strings on some of the binaries and this one pop up a few times:

Code:
%s: failed to get NVRAM information
(__extension__ (__builtin_constant_p (3) && ((__builtin_constant_p (dev_name) && strlen (dev_name) < ((size_t) (3))) || (__builtin_constant_p ("mtd") && strlen ("mtd") < ((size_t) (3)))) ? __extension__ ({ size_t __s1_len, __s2_len; (__builtin_constant_p (dev_name) && __builtin_constant_p ("mtd") && (__s1_len = strlen (dev_name), __s2_len = strlen ("mtd"), (!((size_t)(const void *)((dev_name) + 1) - (size_t)(const void *)(dev_name) == 1) || __s1_len >= 4) && (!((size_t)(const void *)(("mtd") + 1) - (size_t)(const void *)("mtd") == 1) || __s2_len >= 4)) ? __builtin_strcmp (dev_name, "mtd") : (__builtin_constant_p (dev_name) && ((size_t)(const void *)((dev_name) + 1) - (size_t)(const void *)(dev_name) == 1) && (__s1_len = strlen (dev_name), __s1_len < 4) ? (__builtin_constant_p ("mtd") && ((size_t)(const void *)(("mtd") + 1) - (size_t)(const void *)("mtd") == 1) ? __builtin_strcmp (dev_name, "mtd") : (__extension__ ({ __const unsigned char *__s2 = (__const unsigned char *) (__const char *) ("mtd"); register int __result = (((__const unsigned char *) (__const char *) (dev_name))[0] - __s2[0]); if (__s1_len > 0 && __result == 0) { __result = (((__const unsigned char *) (__const char *) (dev_name))[1] - __s2[1]); if (__s1_len > 1 && __result == 0) { __result = (((__const unsigned char *) (__const char *) (dev_name))[2] - __s2[2]); if (__s1_len > 2 && __result == 0) __result = (((__const unsigned char *) (__const char *) (dev_name))[3] - __s2[3]); } } __result; }))) : (__builtin_constant_p ("mtd") && ((size_t)(const void *)(("mtd") + 1) - (size_t)(const void *)("mtd") == 1) && (__s2_len = strlen ("mtd"), __s2_len < 4) ? (__builtin_constant_p (dev_name) && ((size_t)(const void *)((dev_name) + 1) - (size_t)(const void *)(dev_name) == 1) ? __builtin_strcmp (dev_name, "mtd") : (__extension__ ({ __const unsigned char *__s1 = (__const unsigned char *) (__const char *) (dev_name); register int __result = __s1[0] - ((__const unsigned char *) (__const char *) ("mtd"))[0]; if (__s2_len > 0 && __result == 0) { __result = (__s1[1] - ((__const unsigned char *) (__const char *) ("mtd"))[1]); if (__s2_len > 1 && __result == 0) { __result = (__s1[2] - ((__const unsigned char *) (__const char *) ("mtd"))[2]); if (__s2_len > 2 && __result == 0) __result = (__s1[3] - ((__const unsigned char *) (__const char *) ("mtd"))[3]); } } __result; }))) : __builtin_strcmp (dev_name, "mtd")))); }) : strncmp (dev_name, "mtd", 3))) == 0

modprobe mtdchar >/dev/null 2>&1
modprobe mtdblock >/dev/null 2>&1
This command requires either the 'mtdchar' or 'mtdblock' driver for
access to flash memory on Solarstorm adapters.  The current kernel does
not have the 'mtdchar' driver, but may have the 'mtdblock' driver.
You may try to use the 'mtdblock' driver by running 'modprobe mtdblock'
and then running this command again, but you MUST reboot when you have
finished.
Failed to read partition %s offset %d size %d
(buf_in != ((void *)0)) ^ (buf_out != ((void *)0))
error %1$d (%2$s) doing nv-read MCDI ioctl on %3$d
Failed to prepare for NVRAM update: %s
Error. Attempting to erase a partition of size %d,                  which is not a multiple of the erase size %d
Failed to erase NVRAM sector: %s
Failed to write NVRAM sector: %s
Failed to finish NVRAM update: %s
grep sfc_control /proc/devices
failed to open /proc/devices; are you root?
failed to find sfc_control in /proc/devices; is driver loaded?
failed to unlink sfc_control and it's on wrong device
ESX host detected but could not create /dev/sfc_control on major
This seems to indicate that the flash memory somehow gets mapped as a standard MTD when 'updating' the card. I don't know if that means that this is just the runtime OS for the Zynq and the NIC parts keep on running or if it is something else. If there are two flash chips it seems to indicate that the FPGA(s) can be hot-booted into flash mode which just turns them into expensive flash controllers.

Other interesting help pages from sfupdate:
Code:
1$s firmware update utility [v%2$d.%3$d.%4$d]
Copyright Solarflare Communications 2006-2017, Level 5 Networks 2002-2005
%1$s is not a valid MAC address or %2$s adapter name
Usage: sfupdate [%s adapter] [options]
Available options are:
  %s, %shelp
    This help.
  %sV, %sversion
    Display version and exit.
  %sM, %smanifest
    Show details of available firmware images and exit.
  %slist
    Show the adapter list.
  %s, %sadapter=(%sN|XX:XX:XX:XX:XX:XX|XX-XX-XX-XX-XX-XX)
    Operate only on the specified adapter.
  %swrite
    Write updated firmware to the adapter(s).
  %sforce
    Force update of all firmware, even if the installed firmware version
    is the same or more recent.
  %sbackup
    Backup existing image before updating. This option may be used
    with the %swrite and %sforce options.
  %sy, %syes
    Update without prompting for a final confirmation. This option
    may be used with the %swrite and %sforce options.
  %simage=(filename)
    Update a firmware image from the given file.
  %sipxe-image=(filename)
    Install the iPXE image from the given file, replacing the Solarflare bootrom image.
    sfupdate will not attempt to replace this image in future unless %srestore-bootrom is used.
  %srestore-bootrom
    Replace an iPXE image with the Solarflare bootrom image in this version of sfupdate
  %sfull-erase
    Force a full-erase of the partition before updating the given file.
  %sv, %sverbose
    Verbose mode.
  %ss, %ssilent
    Silent mode, output errors only.
%1$s firmware update utility: %2$s
Network adapter driver version
  %s %-2d  %s %-2d  %s %-15s  %s %-15s  %s %zu
    Firmware cannot be accessed via this interface.
    Firmware cannot be accessed via this interface,
    please use %s.
Dynamic configuration is corrupt. Run 'sfboot -i %s --repair' to revert back to firmware defaults.
Please contact support@solarflare.com to report this incident and for more information on recovery procedure
Failed to check firmware on %1$s: %2$s
The PHY firmware is up to date
   - run "sfupdate %sforce %swrite" to perform an update
This is more recent PHY firmware [%s]
This utility contains more recent PHY firmware [%s]
   - run "sfupdate %swrite" to perform an update
This utility does not contain PHY firmware for this adapter
The current Boot ROM is an iPXE image.
The Boot ROM firmware is up to date
The provided image is an iPXE image
This is more recent Boot ROM firmware [%s]
This utility contains more recent Boot ROM firmware [%s]
This utility does not contain Boot ROM firmware for this adapter
The UEFI ROM firmware is up to date
This is more recent UEFI ROM firmware [%s]
This utility contains more recent UEFI ROM firmware [%s]
This utility does not contain UEFI ROM firmware for this adapter
The controller firmware is up to date
This is more recent controller firmware [%s]
This utility contains more recent controller firmware [%s]
This utility does not contain controller firmware for this adapter
The FPGA bitfile is up to date
This is a more recent FPGA bitfile [%s]
This utility contains a more recent FPGA bitfile [%s]
This utility does not contain an FPGA bitfile for this adapter
The diagnostic FPGA bitfile is up to date
This is a more recent diagnostic FPGA bitfile [%s]
This utility contains a more recent diagnostic FPGA bitfile [%s]
This utility does not contain an diagnostic FPGA bitfile for this adapter
The AOE firmware is up to date
This is a more recent AOE firmware [%s]
This utility contains a more recent AOE firmware [%s]
This utility does not contain AOE firmware for this adapter
The MUM firmware is up to date
This is a more recent MUM firmware [%s]
This utility contains a more recent MUM firmware [%s]
This utility does not contain MUM firmware for this adapter
The CPLD bitfile is up to date
This is a more recent CPLD bitfile [%s]
This utility contains a more recent CPLD bitfile [%s]
This utility does not contain an CPLD bitfile for this adapter
No %1$s adapters found with MAC address %2$s
No adapters could be accessed. You may need to run this utility as root.
%1$s cannot be applied to %2$s
%s cannot be applied to any adapter
No firmware is available for %1$s
No firmware is available for any adapter
I didn't spot a list of PCI IDs yet, I suspect the firmware files might have those in their headers, but I haven't decoded those. Well, I couldn't resist, so I decompiled the binaries and it turns out they are not stripped and also contain pretty much all DWARF data which helps a lot in finding out what's what.

Edit3: looks like the PCI ID isn't included in this package and medford is only mentioned in the PXE payload.
Edit4: sfboot contains a ton of devices but no medford or 0x19240b13 but it does look like this is the type of un-init device they had in mind? If someone can find other sfboot or sfupdate packages those should be easy to check. Even with just decompressing and searching for hex sequence 13 0b 24 19 you could get easy hits.
 

Attachments

Last edited:
  • Like
Reactions: mwrnd

victhor393

New Member
Feb 13, 2022
26
11
3
I tried to trace the pins for the top JTAG port, but I was unable to. There's no continuity. It seems there's something between the Zynq and the port, but I don't know what it might be. I found these pins by overlaying the pin diagram available on one of the Zynq Ultrascale datasheets over a photo of the board.
Legend for the colors on the image is as follows:
Red - TCK
Blue - TDO
Green -TDI
Cyan - TMS
I also found that pins A1, A7, B1 and B7 on the JTAG port are connected to ground.
It would be an incredible coincidence if it used the same pinout as JTAG pins on a PCIe connector!
 

Attachments

  • Like
Reactions: mwrnd

oneplane

Well-Known Member
Jul 23, 2021
844
484
63
So this
Code:
#define    EFX_PCI_DEVID_MEDFORD2_PF_UNINIT    0x0B13
#define    EFX_PCI_DEVID_MEDFORD2            0x0B03    /* SFC9250 PF */
#define    EFX_PCI_DEVID_MEDFORD2_VF        0x1B03    /* SFC9250 VF */
is in dpdk source code: drivers/common/sfc_efx/base/efx.h (v21.11)
Dunno if that helps any. Seems like it's a family called a Medford 2?
Looks like it is indeed some sort of 'version 2' of what was medford: https://github.com/Xilinx/U25N-Smar...n/u25n_driver/drivers/net/ethernet/sfc/ef10.c it specifically mentions "SFC9250 (Medford2)".

it also appears that the docs (src at GitHub - Xilinx/U25N-SmartNIC-Solution ) are all centered around this specific card: https://xilinx.github.io/U25N-SmartNIC-Solution/docs/build/html/index.html which due to the topology needs to have at least a passthrough and MAC controlling bitstream loaded for the FPGA before you can even use it as a basic NIC. But from what the driver shows, you can both write persistent images to both NVRAM flash chips from 'dead' mode, and the default product ID seems to confirm that (no JTAG needed, but a bitstream is definitely required).

Easiest might be to find all OEMs that have this combo package and rip one of their firmwares.

It does appear that if the correct keys are loaded in the EEPROMs to allow encryption IP to load those cards go from 2k to 8k on eBay right now :oops:

Well well well, what do we have here: xilinx U25 FPGA Utility (For Unknown OS) - Lenovo Support US
and this one: xilinx U25 X2 NIC Firmware (For Unknown OS) - Lenovo Support US

That second one contains a bunch of firmwares inside sfutils-8.1.3.1011-1.x86_64.rpm including the SFC9250 bootrom and the runtime payload! When decompiled it seems to refer to the port 0 and port 1 a lot so this is probably something that needs to be loaded before the FPGA can be programmed over PCIe.
 
Last edited:
  • Like
Reactions: mwrnd