NIC firmware update on ROMED8-2T failed?!

[Dolpa]

Dear Folks,

Update: here is the solution!

I use an AsrockRack ROMED8-2T mobo with EPYC 7451 CPU and running Proxmox VE for virtualization.
This mobo has a dual Intel X550-T2 NIC onboard which has a very old firmware from 2018.

The shipped NIC firmware is 1.93 80000AEE 05/25/2018.
The latest on official Intel site is 3.15 80001373 11/10/2020.

Non-Volatile Memory (NVM) Update Utility for Intel® Ethernet Network Adapter X550 Series

This download record contains all files required to update the Non-Volatile Memory (NVM) update utility version 3.30 for Intel® Ethernet Converged Network Adapter X550 Series.

downloadcenter.intel.com

I mailed to AsrockRack support one month ago how can I update the firmware but the first question was why i want to update?
So I would like to use the SR-IOV function for KVM guest and bridge mode for LXC but this does not work at the same time with old firmware.
Intel fixed this with latest firmwares.

After a long exchange of letters with support stayed in to try the official update.
So I pull the trigger and follow the Intel official method I used the UEFI tools.

# nvmupdate64e -u -b -l -o update.xml -c nvmupdate.cfg
Intel(R) Ethernet NVM Update Tool
NVMUpdate version 1.35.42.7
Copyright (C) 2013 - 2020 Intel Corporation.

Config file read.
Inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Shadow RAM inventory started.
        Shadow RAM inventory finished.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Update
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Creating backup images in directory: D05099DB878F.
        Backup images created.
        Flash update started.
|======================[100%]======================|
        NVM verification started.
        Shadow RAM verification started.
|======================[100%]======================|
        Shadow RAM verification finished.
        Flash verification started.
|======================[100%]======================|
        Flash verification finished.
        NVM verification finished.
        Flash update successful.
Update security revisions
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Skipping update minimum security revisions.
Checking update availability for next tool run.
Post update inventory
[00:098:00:00]: Intel(R) Ethernet Controller X550-T2
        Flash inventory started.
        Flash inventory finished.
        OROM inventory started.
        OROM inventory finished.
[00:098:00:01]: Intel(R) Ethernet Controller X550-T2
        Device already inventoried.
Power Cycle is required to complete the update process.

The update.xml content is:

<?xml version="1.0" encoding="UTF-8"?>
<DeviceUpdate lang="en">
    <Instance vendor="8086" device="1563" subdevice="1563" subvendor="1849" bus="98" dev="0" func="0" PBA="000000-000" port_id="Port 1 of 2" display="Intel(R) Ethernet Controller X550-T2">
        <Module type="PXE" version="2.4.44" previous_version="2.4.32" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="EFI" version="7.8.13" previous_version="7.0.19" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="NVM" version="80001373" previous_version="80000AEE" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <VPD>
            <VPDField type="String">Intel (r) Ethernet Controller X550</VPDField>
        </VPD>
        <MACAddresses>
            <MAC address="D05099DB878F">
            </MAC>
            <AltMAC address="D05099DB878F">
            </AltMAC>
        </MACAddresses>
    </Instance>
    <Instance vendor="8086" device="1563" subdevice="1563" subvendor="1849" bus="98" dev="0" func="1" PBA="000000-000" port_id="Port 2 of 2" display="Intel(R) Ethernet Controller X550-T2">
        <Module type="PXE" version="2.4.44" previous_version="2.4.32" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="EFI" version="7.8.13" previous_version="7.0.19" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <Module type="NVM" version="80001373" previous_version="80000AEE" display="">
            <Status result="Success" id="0">All operations completed successfully.</Status>
        </Module>
        <VPD>
            <VPDField type="String">Intel (r) Ethernet Controller X550</VPDField>
        </VPD>
        <MACAddresses>
            <MAC address="D05099DB8790">
            </MAC>
            <AltMAC address="D05099DB8790">
            </AltMAC>
        </MACAddresses>
    </Instance>
    <NextUpdateAvailable> 0 </NextUpdateAvailable>
    <RebootRequired> 0 </RebootRequired>
    <PowerCycleRequired> 1 </PowerCycleRequired>
</DeviceUpdate>

I did the power cycle and started the Proxmox. The NIC recognized and the linux kernel module loaded successful.

# ethtool -i enp98s0f0
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001373, 1.2203.0
expansion-rom-version:
bus-info: 0000:62:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

# ethtool -i enp98s0f1
driver: ixgbe
version: 5.1.0-k
firmware-version: 0x80001373, 1.2203.0
expansion-rom-version:
bus-info: 0000:62:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

The port1 working fine BUT the port2 has no link.
Both worked before update.
Now link is only on port1.

# ethtool enp98s0f0
Settings for enp98s0f0:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: 10000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes

# ethtool enp98s0f1
Settings for enp98s0f1:
    Supported ports: [ TP ]
    Supported link modes:   100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Supported pause frame use: Symmetric
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  100baseT/Full
                            1000baseT/Full
                            10000baseT/Full
    Advertised pause frame use: Symmetric
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Speed: Unknown!
    Duplex: Unknown! (255)
    Port: Twisted Pair
    PHYAD: 0
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: Unknown
    Supports Wake-on: umbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: no

I checked the firmware:

# nvmcheck64e /DEVICES
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NIC B/D/F     Ven-Dev   MAC          ENA Branding string
=== ========= ========= ============ === ======================================
1) 098/00/00 8086-1563 D05099DB878F YES Intel(R) Ethernet Controller X550-T2
2) 098/00/01 8086-1563 D05099DB8790 YES Intel(R) Ethernet Controller X550-T2

# nvmcheck64e /NIC=1 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

# nvmcheck64e /NIC=2 /VERIFY
Intel(R) NVM Integrity Verification Tool
Nvmcheck version 1.35.57.00
QV SDK version 2.35.57.00
Copyright(C) 2012 - 2020 Intel Corporation.

NVM module: Option ROM.
NVM Integrity verification PASSED.

NVM module: PHY Firmware.
NVM Integrity verification PASSED.

NVM module: EMP image.
NVM Integrity verification PASSED.

So now there is no link on port2 (no LEDS light up on RJ45), I double checked my CAT-6E cables checked the NETGEAR GS110EMX switch changed another swith but don't working.

Intel diagnostic tool port1 and port2:




I made a backup from the old firmware, but the update tool said "Rollback blocked". So I can't rollback to the old version.



I tried on Windows10 too but no success, tried oldest firmwares same story.

ps: The AsrockRack support send me an old eeprom tool try to change the MAC address. WOT? LOL.
The tool is useless for the X550-T2, there has been silence ever since last mail.

Any idea? How I can revive the link?

Best Regards,
Dolpa

 

[RTM]

That is a tough one, I only have one idea and it is really just grasping at straws.

Assuming the Asrock board is not too different from your run-of-the-mill Supermicro board, you can configure the IPMI to share one of the 10G RJ45 ports (shared mode vs dedicated). Suppose the reason why one of the ports is not working is related to this, you might be able to resolve it by configuring the IPMI to shared mode and then back to dedicated afterwards.

I doubt it will work, and there's probably an equal chance of failing spectacularly, so exercise caution (if it fails you may be able to configure the "mode" from BIOS).

To be honest I would start planning the RMA, given Asrock suggested this I think you are in the clear.

 

[Dolpa]

@RTM Thank You for the answer.
Yes I thought about the share but the IPMI is shared with the port1 which is work. BTW I dont's use the bounding mode, the IPMI port is separated from the X550-T2 port1.

I would like to avoid the RMA, because my system is a massive WC build and it would be difficult and lot of work to disassemble.
And I think this is not a hardware issue but the software/firmware.

 

[Rand__]

I just flashed the X772 on my SM X11SPH today which also had an ages old FW.
Contrary to your process i was not able to use the Intel tools but got a SM package.

The point is, that from the very old firmware I had on the manual instructed to run the flash tool *twice* or it would not work properly.
Have you tried that? (i.e. rereun the flash, same fw)?

 

[Dolpa]

@Rand__ Thank You for the answer.
Yes I tried many times, tried older versions too and tried only just the port2 by MAC address selecting.
Interesting that all checking is successful, driver loaded just there is no link.

 

[Rand__]

Hm.
You could try to contact Intel...

 

[RTM]

@RTM
I would like to avoid the RMA, because my system is a massive WC build and it would be difficult and lot of work to disassemble.
And I think this is not a hardware issue but the software/firmware.

Fair enough, I don't think anyone enjoys doing RMAs.

There are not many other (if any?) options left, when their support staff starts ignoring you.
Have you tried calling them? (if that is even an option), or even trying a different office (if you are in the EU try US or vice versa).

 

[Dolpa]

Thanks for the ideas.
I am trying to get some higher level company support and contact Intel.

 

[Dolpa]

I have got a feedback from AsrockRack today.
They said the rollback belongs to Intel which has NDA. I can't rollback myself, mobo may need to return RMA center to flash it back.

It's unacceptable for me. If I understand correctly this motherboard will be stuck a very old NIC firmware forever because there is no tool for update.
I think I need to look for a new mobo and company.

 

[Rand__]

Well there is an option for running the update, isnt there?

The problem is its not working properly/consistently apparently.
But that does not seem to be Asrocks fault at this point but possibly Intel's as provider of the firmware.

Its not bad that Asrock is offering an RMA despite them not really doing anything wrong (although o/c they should provide either a tool for updating the fw or validating that is working as expected).

 

[Dolpa]

I think that when I buy a product from a company, it is up to that company to support it, even if it includes a product from another company. I didn't buy the product from Intel but from AsrockRack.

I bought this motherboard because the X550-T2 NIC can support SR-IOV and also works in traditional bridge mode as well to work with KVM and LXC. Unfortunately there was no any information on the product site that the NIC contained an outdated firmware. If I had known this, then I would have chosen another product.

I think it will be RMA. Although I don't want to believe that this couldn't be solved by software myself.

 

[Rand__]

Valid point of view

 

[RTM]

I guess we have been accustomed to Intel NICs not needing firmware updates (ignoring option ROM) (at least as far as I am aware, a card like the X520 was not software upgradeable).

I suppose it is to be expected, vendors like asrock (and probably asus and gigabyte, perhaps even Supermicro and Tyan) have never been great at providing firmware updates for additional controllers (such as SAS) on motherboards, compared to addon stock from the actual vendor.

The big deal here is that certain features do not work with the default firmware, and who knows if Intel intends to ever support the controller with their firmware redistributable? The implication of this is that you can have two boards from different vendors that seemingly have the same specs, but with wildly different levels of actual support. And what happens when we are not talking missing features or bugs but software vulnerabilities?

Of course we as hardware people have a tendency to look single-mindedly at hardware specs and ignore elements like support (because it is difficult to measure).

It is the same problem as what Google have with Android, it is up to the individual vendors to validate and push the updates to the devices, and some are definitely better than others.

@Patrick here's a thought that you might be able to push onto the vendors (and/or turn into a discussion piece on the main site) (or just ignore as a rant ):

• Why not implement something akin to Android One for (server) boards?
  • Ensure a minimum amount of time the device (including its controllers) is supported
  • Implement more generalization on hardware and software levels (to simplify validation and shorten time to deployment)
  • Perhaps controllers could support both customized firmwares and stock
    • This would give users more choice and possibly assist with troubleshooting issues like in this thread.

Sure this is all probably just a pointless rant, that fails because vendors want to sell more hardware, not supporting it means users are forced to buy new hardware because the old fails to work as intended.
But I imagine that vendors like Asrock, Asus and Gigabyte especially want to show server buyers that they serious vendors (that can compare favorably with bigger vendors like Dell, HP and Lenovo), it seems to me that something like this might help them achieve this or even give them market differentiation.

/end rant

 

[Dolpa]

Finally I contacted to Intel Support and they promised some help.

 

[blublub]

Finally I contacted to Intel Support and they promised some help.

Hi,

What was the solution? I have a X470DU with X550-T2 and I wnated to update the NVM on the NIC as I do ahve some network issues.
But when I runt he update tool it says "no update available" - yea right NVM 1.48 and current on intel page 3.15...

Is there an guide how to update the NICs NVM? - otherwise I will just RMA the board and let Asrock do it, this is serious BS in their part

 

[Dolpa]

Hi,

What was the solution? I have a X470DU with X550-T2 and I wnated to update the NVM on the NIC as I do ahve some network issues.
But when I runt he update tool it says "no update available" - yea right NVM 1.48 and current on intel page 3.15...

Is there an guide how to update the NICs NVM? - otherwise I will just RMA the board and let Asrock do it, this is serious BS in their part

Hi,

There is no solution yet, intel engineers have been working on it for 3 weeks. I'll write here what the result was.
But I don't recommend to update the nic firmware, try to ask some help from Asrock. Unfortunately I had no luck with them.
Now I wait the answer from Intel support.

 

[blublub]

There is no solution yet, intel engineers have been working on it for 3 weeks. I'll write here what the result was.
But I don't recommend to update the nic firmware, try to ask some help from Asrock. Unfortunately I had no luck with them.
Now I wait the answer from Intel support.

Ok, thx.

This is some serious BS in Asrocj and/or Intel side. They sell sever HW with 3 year old NIC FW but it is not updateable!?? I havent come across this in 20 years...

 

[Dolpa]

Not intel, rather just Asrock.
It is the responsibility of Asrock to keep the firmware up to date on the goods it sells. The Intel NIC firmware is available.
But yes big BS and they couldn't tell me why the firmware is so old, why they not use the last one?

 

[blublub]

Not intel, rather just Asrock.
It is the responsibility of Asrock to keep the firmware up to date on the goods it sells. The Intel NIC firmware is available.
But yes big BS and they couldn't tell me why the firmware is so old, why they not use the last one?

I do not understand why the Intel flash tool fails - from ym perspective it is just an Intel NIC chip soldere to the mainboard, so theoretically it should just be able to flash it just like any other PCIe card/NIC - this is really annoying as hell and is likely keeping me from buying Asrock again

 

[Dolpa]

I do not understand why the Intel flash tool fails - from ym perspective it is just an Intel NIC chip soldere to the mainboard, so theoretically it should just be able to flash it just like any other PCIe card/NIC - this is really annoying as hell and is likely keeping me from buying Asrock again

Theoretically yes, its Intel NIC, but onboard chip is a little different because usually its handle the BMC port access too with NCSI (Network Communications Services Interface) function. So the onboard NIC update is not equivalent to a standalone PCIe NIC card update.

My biggest problem is why they not send me an update firmware kit to restore the onboard X550 NIC to factory default state? Its only a software.
Why it would be necessary to return the whole motherboard to Asrock and wait several weeks for a factory firmware update?
And yes next time I'm will avoid Asrock products.