NIC firmware update on ROMED8-2T failed?!

[philwo]

I have exactly the same problem, but with an ASRockRack E3C246D4I-2T board. I updated the firmware using the Intel NVM update tool and now only one of the two ports gets a link. I sent a mail to ASRock to request a firmware update that will restore the functionality, but they haven't replied yet (it's been 2 weeks now).

If you figure this out, please share your solution.

 

[Dolpa]

Hi @philwo ,

Now I'm waiting a response from Intel support. They have been working on this for 3 weeks.
I'll post here the solution.

 

[i386]

from ym perspective it is just an Intel NIC chip soldere to the mainboard, so theoretically it should just be able to flash it just like any other PCIe card/NIC

From my understanding a "nic chip" is not one, but multiple devices* with their own firmwares. Maybe asrack is using intel controller and some custom phy (that are different from retail intel add on cards) and that casues the problems?

*See this post on stackexchange: The SERDES/transceiver design inside the Ethernet MAC controller

Spoiler: pictures

From the i210 datasheet:


Implementation on a fpga:

 

[Dolpa]

Yes, the big mistake was that Asrock did not rewrite the firmware revision number to another different one, but left the original intel revision number and thus the official firmware update recognized it as an updatable NIC. Fail!
For example I have a Cisco branded Intel NIC too and that firmware revision number not matched to any Intel NIC firmware revision number.

 

[Daniel Baird]

Thankyou all for this thread. i have a X570D4U-2L2T and have gone through the exact same experience. watching this thread with interest. i haven't had a reply back from ASRock yet. I'd rather not RMA, but i can if i have to. my X570D4U-2L2T based system is still being tested, and hasn't "gone into Prod" yet. The custom cooling setup will be a PITA to remove and re-add later.

 

[Dolpa]

Hi there,

I just informed you that Intel engineers has been working hard for 2 months to solve the problem. There is no solution yet.

Have a nice day.

 

[philwo]

Thank you for the update, Dolpa.

For what it’s worth, I never heard back from ASRockRack.

Hope that Intel can eventually figure something out and help us fix the broken port.

 

[Dolpa]

I have got an answer from Intel.

We believe the issue is that Debug Mode is enabled which results in the single port failure following the update.

Debug Mode is entered when the signal to X550 between the Flash's SCLK pin is pulled up when the X550 powers on (like a boot strapping configuration).

This mode is for engineering and debug work only and is not supported in normal operation or basically in manufacturing.

We surmise that ASROCK is shipping the motherboards in this configuration.

The Debug Mode tells the Firmware onboard the X550 it is ok to make certain writes to the NVM it might normally. This would explain the behavior we see evidence of, and it also correlates to the isolation to this one manufacturer.

It is highly recommended to request RMA to your boards and share the aforementioned information with the OEM, ASROCK.

ASRock > Support

Well I'm not happy with this mobo, so it seems will be RMA.

 

[i386]

+1 on my asrock strikes list

 

[Dolpa]

There is a good news!

AsrockRack finally released a firmware update for the X550 onboard NIC. I eventually kept my own MB.
I found it on the bottom of the FAQ page.
> How to Update X550-AT2 on ROMED8-2T and X570D4u-2L2T

Direct link to FW: X550T2 FW v3.15

There is only EFI update package but the point is the ASRR_X550_0_0_002_0.bin file.
If You already updated to the official fw v3.15 (and made a halfbrick) must be edit the nvmupdate.cfg file, search the 80000AEE in the REPLACES line, and insert at end of the line this EEPID: 80001373 like this:

REPLACES: 80000526 8000059F 80000603 8000069C 800007F6 8000049A 800005D0 800005DE 800005EE 800005F8 80000634 80000651 80000688 8000069D 800007B5 8000084F 80000948 800009BA 800009E7 800009F9 80000A29 80000A73 80000AEE 80000D03 80000E3E 80000D37 80000D78 80000F49 80001112 80000F98 80000F9B 80000F9C 80001050 80001138 80001180 800011FA 8000134A 80001373

This is the v3.15 EEPID, without this You can't update the FW.

USE AT YOUR OWN RISK! DON'T USE THIS FW FOR ANOTHER NIC, IS PREPARED ONLY FOR THESE MB!

 

[RTM]

That is good to hear, I assume this means it fixed your NIC?

 

[Dolpa]

That is good to hear, I assume this means it fixed your NIC?

Yes I confirm, my NIC is working again with this 3.15 FW.

 

[aholmes5]

Thanks so much. You got my port 2 working again. I made the mistake of updating via the Intel software as well and bricked one port.

There is a good news!

AsrockRack finally released a firmware update for the X550 onboard NIC. I eventually kept my own MB.
I found it on the bottom of the FAQ page.
> How to Update X550-AT2 on ROMED8-2T and X570D4u-2L2T

Direct link to FW: X550T2 FW v3.15

There is only EFI update package but the point is the ASRR_X550_0_0_002_0.bin file.
If You already updated to the official fw v3.15 (and made a halfbrick) must be edit the nvmupdate.cfg file, search the 80000AEE in the REPLACES line, and insert at end of the line this EEPID: 80001373 like this:

REPLACES: 80000526 8000059F 80000603 8000069C 800007F6 8000049A 800005D0 800005DE 800005EE 800005F8 80000634 80000651 80000688 8000069D 800007B5 8000084F 80000948 800009BA 800009E7 800009F9 80000A29 80000A73 80000AEE 80000D03 80000E3E 80000D37 80000D78 80000F49 80001112 80000F98 80000F9B 80000F9C 80001050 80001138 80001180 800011FA 8000134A 80001373

This is the v3.15 EEPID, without this You can't update the FW.

USE AT YOUR OWN RISK! DON'T USE THIS FW FOR ANOTHER NIC, IS PREPARED ONLY FOR THESE MB!

 

[philwo]

Thank you, Dolpa! I couldn't have fixed this without your help.

I finally managed to try this firmware with my ASRockRack E3C246D4I-2T. Even though the firmware is only meant for AMD mainboards, it works fine on my Intel Xeon mainboard, too, and fixed the issue.

Immediately after flashing the firmware and rebooting the computer the second port still didn't work in Windows, but something I did afterwards must have helped. In case someone else stumbles upon this - here's what I did. Can't exactly say which step did it, but maybe just try until it works:
- Unplugged and replugged power
- Flashed the same BIOS version that was already installed again via the BMC website to force it to restore the default settings, as for some reason my BIOS setup just crashed reproducibly when I pressed the "Load default settings" option..
- Reset the BMC to default settings via the BIOS
- Did "Reset network" in Windows 10 network settings

Now both ports work again.

 

[shog63]

Thank you, Dolpa! I couldn't have fixed this without your help.

I finally managed to try this firmware with my ASRockRack E3C246D4I-2T. Even though the firmware is only meant for AMD mainboards, it works fine on my Intel Xeon mainboard, too, and fixed the issue.

Immediately after flashing the firmware and rebooting the computer the second port still didn't work in Windows, but something I did afterwards must have helped. In case someone else stumbles upon this - here's what I did. Can't exactly say which step did it, but maybe just try until it works:
- Unplugged and replugged power
- Flashed the same BIOS version that was already installed again via the BMC website to force it to restore the default settings, as for some reason my BIOS setup just crashed reproducibly when I pressed the "Load default settings" option..
- Reset the BMC to default settings via the BIOS
- Did "Reset network" in Windows 10 network settings

Now both ports work again.

I'm a little late to the party, but I'm interested in your issues with the ASRockRack E3C246D4I-2T.

I have 2 of these, one with on-metal TrueNAS and one with Vsphere 6.7 with a TrueNAS server along with a few Windows & Linux guests. You would probably term it a "onesie". I thought this was a perfect board for my needs.

The Vsphere system runs an I9-9900T CPU (ES though) and the TrueNAS board an i3-8100T. The I9-9900T provides Intel GFX on passthrough in vSphere to Emby & Plex running in containers on an FC client, which can utilise this for hardware encoding/decoding (quicksync). Storage is provided by a combination of NFS4 for everything apart from the Plex configuration volume, which is provided as an iSCSI zvol. From my initial setting up, I can tell you that Plex can do thoroughly indecent amounts of IO to this volume and the iSCSI vol was the answer to severe lag. I do get the odd unexpected reboot of the TrueNAS box, not sure if it is related to the ES CPU. Must do some more investigation. The TrueNAS box was using one x550-t2 port in passthrough originally.

The initial board has been pretty good from day one. The second initially seemed ok, but when I tried to test network throughput I would get messages mentioning a degraded PCIe bus. I also was only getting around 1/2 10G throughput. IPMI errors logged:
332 | 05/11/2021, 06:37:05 | BIOS | Critical Interrupt | Bus Degraded - Asserted
| | | | Bus number: 0x02
| | | | Link width degraded

ASRockRack provided a new BIOS, the error messages went away but more testing revealed the same throughput issue. I delved deeper into the BIOS and found an option to disable certain error reporting turned on. I disabled it, and at that point the bus error messages returned. I got an RMA, and the replacement board had an issue where the BMC powered up, but the mainboard itself would not power up by either power button or BMC. A further RMA later, and I have a board which appears sound. This has taken a year.

I now have the issue (and I believe I always have) where the x550-t2 ports randomly go down for periods of 5 seconds, sometimes at times of high IO, sometimes not. To avoid outage on the Vsphere based system, I have removed the passthrough and provided a virtual port instead with both x550-T2 ports provided as uplinks in Vsphere. I only really noticed this when some software I was running on a W10 client blew out with messages saying it couldn't access its DB. Errors from Vsphere:

2022-01-17T11:18:15.789Z cpu8:2097642)ixgben: ixgben_CheckLink:2745: Link is down for device vmnic0 (0x430b22f7c0c0)
2022-01-17T11:18:20.330Z cpu8:2097642)ixgben: ixgben_CheckLink:2745: Link is up for device vmnic0 (0x430b22f7c0c0)
2022-01-17T20:39:53.045Z cpu11:2097652)ixgben: ixgben_CheckLink:2745: Link is down for device vmnic1 (0x43052503b0c0)
2022-01-17T20:40:13.691Z cpu11:2097652)ixgben: ixgben_CheckLink:2745: Link is up for device vmnic1 (0x43052503b0c0)


IXsystems have suggested it is a hardware or firmware issue, and I do not doubt them considering I see this on both Vsphere & TrueNAS. I'm just wondering whether anyone else with this board is seeing this, either with original version of FW or the new?

Thanks all

 

[shog63]

So I applied the firmware. Didn't fix the down/up issue for me:

Jan 19 11:36:50 thug kernel: ix0: link state changed to DOWN
Jan 19 11:36:55 thug kernel: ix0: link state changed to UP

The box was doing nothing of any signficance at the time

If anything, it seems more frequent now. Completely idle box:

Jan 19 11:36:50 thug kernel: ix0: link state changed to DOWN
Jan 19 11:36:55 thug kernel: ix0: link state changed to UP
Jan 19 12:35:47 thug kernel: ix0: link state changed to DOWN
Jan 19 12:35:52 thug kernel: ix0: link state changed to UP
Jan 19 13:48:12 thug kernel: ix0: link state changed to DOWN
Jan 19 13:48:17 thug kernel: ix0: link state changed to UP
Jan 19 14:52:33 thug kernel: ix0: link state changed to DOWN
Jan 19 14:52:38 thug kernel: ix0: link state changed to UP

 

[Dolpa]

Hi shog63,

As I wrote this firmware is related only for the mentioned motherboards and fixed the dead port issue. I think Your issue is another story and another type of motherboard, maybe a switch, a cable or another parts in your setup.
I hope you will find a solution soon. Unfortunately Asrock support is terrible.

Best Redards,
Dolpa

 

[octave]

@Dolpa I have a brand new ROMED8-2T, none of the Intel ports were working (only IPMI). Before reading this thread I updated the Intel firmware to the latest version directly from Intel which is 3.60. But now it still doesn't work at all. And I can't go back to the version you posted, since it's a previous version... I contacted Asus but if you have any pointers in the meantime that would be great.

Thank you,
Octave

 

[Dolpa]

Hi @octave

Scroll a little up and read my post #30.
Cheers.

 

[octave]

You are going to laugh but the IO shield tabs on the RJ45 ports were pushing my RJ45 connectors down and preventing them to make proper contact. I bent those tabs and it works as expected.