Dell Broadcom 57810S "MDC/MDIO access timeout"

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
D

Damo

Active Member
Sep 7, 2022
113
30
28
I can't figure out why but these cards do not seem to be working with my motherboard (X570D4U) attached is the log file.


Code: 
auto lo
iface lo inet loopback

auto enp38s0
iface enp38s0 inet manual

auto enp45s0f0
iface enp45s0f0 inet manual

auto enp45s0f1
iface enp45s0f1 inet manual

auto bond1
iface bond1 inet manual
        bond-slaves enp45s0f0 enp45s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.220/24
        gateway 192.168.0.1
        bridge-ports enp38s0
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 10.0.10.220/24
        gateway 10.0.10.1
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
lspci
Code: 
2d:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
2d:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 10)
Log attached or Nov 12 15:17:04 ENT1 pve-guests[24907]: <root@pam> starting task UPID:ENT1:00006 - Pastebin.com
 

Attachments

P

prdtabim

Active Member
Jan 29, 2022
175
67
28
I have some problems with bonding too,

Try this to the bonding config
Code: 
iface bond1 inet manual
        bond-slaves enp45s0f0 enp45s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        up ip link set enp45s0f0 master bond1
        up ip link set enp45s0f1 master bond1
The boot is delayed a little but the interface will come up with the slaves ok.

In my case the interfaces doesn't join the bond device. Putting that lines add the interfaces when the system try to config the bonding device .
 
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
I have some problems with bonding too,

Try this to the bonding config
Code: 
iface bond1 inet manual
        bond-slaves enp45s0f0 enp45s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3
        up ip link set enp45s0f0 master bond1
        up ip link set enp45s0f1 master bond1
The boot is delayed a little but the interface will come up with the slaves ok.

In my case the interfaces doesn't join the bond device. Putting that lines add the interfaces when the system try to config the bonding device .
Click to expand...
This is not a bonding issue if I remove the bond config and change to the following the errors still show

Code: 
auto vmbr1
iface vmbr1 inet static
        address 10.0.10.220/24
        gateway 10.0.10.1
        bridge-ports enp45s0f0
        bridge-stp off
        bridge-fd 0
 
P

prdtabim

Active Member
Jan 29, 2022
175
67
28
Damo said:
This is not a bonding issue if I remove the bond config and change to the following the errors still show

Code: 
auto vmbr1
iface vmbr1 inet static
        address 10.0.10.220/24
        gateway 10.0.10.1
        bridge-ports enp45s0f0
        bridge-stp off
        bridge-fd 0
Click to expand...
Well ...
Looking more closely in the logs that you posted.
Looks like a firmware bug in that the card became unresponsive after many errors or time outs.
https://www.reddit.com/r/homelab/comments/vx8byq As explained in the last post of the reddit a firmware update in the card solve the problem. Use ethtool -i to see the actual firmware of the card.
You can grab generic firmware/drivers for 578xx line on Marvell website.
www.marvell.com

Public Driver Downloads | QLogic Fibre Channel HBAs and FastLinQ Ethernet adapters and controllers - Marvell

Download Marvell drivers by Platform or Part Number for Marvell QLogic Fibre Channel HBA and Marvell FastLinQ Ethernet adapters and controllers.
www.marvell.com www.marvell.com
-> Converged network adapters
-> QLE84xx-CNA
Look for " Flash Image for 34xx/84xx/578xx Adapters" .
 
  • Like
Reactions: Damo
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
Well ...
Looking more closely in the logs that you posted.
Looks like a firmware bug in that the card became unresponsive after many errors or time outs.
https://www.reddit.com/r/homelab/comments/vx8byq As explained in the last post of the reddit a firmware update in the card solve the problem. Use ethtool -i to see the actual firmware of the card.
You can grab generic firmware/drivers for 578xx line on Marvell website.
www.marvell.com

Public Driver Downloads | QLogic Fibre Channel HBAs and FastLinQ Ethernet adapters and controllers - Marvell

Download Marvell drivers by Platform or Part Number for Marvell QLogic Fibre Channel HBA and Marvell FastLinQ Ethernet adapters and controllers.
www.marvell.com www.marvell.com
-> Converged network adapters
-> QLE84xx-CNA
Look for " Flash Image for 34xx/84xx/578xx Adapters" .
Click to expand...
Thanks for the reply. I could not get ethtool -i working

Code: 
root@ENT1:~# ethtool -i bond1
driver: bonding
version: 5.15.64-1-pve
firmware-version: 2
expansion-rom-version:
bus-info:
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
root@ENT1:~# ethtool -i vmbr1
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

I downloaded ql_mbi_7.16.01.zip which contains a bin file how do I install it?
 
P

prdtabim

Active Member
Jan 29, 2022
175
67
28
Damo said:
Thanks for the reply. I could not get ethtool -i working

Code: 
root@ENT1:~# ethtool -i bond1
driver: bonding
version: 5.15.64-1-pve
firmware-version: 2
expansion-rom-version:
bus-info:
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
root@ENT1:~# ethtool -i vmbr1
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

I downloaded ql_mbi_7.16.01.zip which contains a bin file how do I install it?
Click to expand...
Sorry ...
Same list look for " 34xx/84xx/578xx Firmware Upgrade Utility for Linux " or " 34xx/84xx/578xx Firmware Upgrade Utility for Windows " depending of the OS used. I suppose that you are using Proxmox that is Debian based. I used this flasher in Proxmox 7.

You must inquire the actual device not the bond. ethtool -i enp45s0f0 ou enp45s0f1
 
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
Sorry ...
Same list look for " 34xx/84xx/578xx Firmware Upgrade Utility for Linux " or " 34xx/84xx/578xx Firmware Upgrade Utility for Windows " depending of the OS used. I suppose that you are using Proxmox that is Debian based. I used this flasher in Proxmox 7.

You must inquire the actual device not the bond. ethtool -i enp45s0f0 ou enp45s0f1
Click to expand...
Code: 
root@ENT1:~# ethtool -i enp45s0f0
driver: bnx2x
version: 5.15.64-1-pve
firmware-version: mbi 255.255.255 FFV14.04.11 bc
expansion-rom-version:
bus-info: 0000:2d:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@ENT1:~# ethtool -i enp45s0f1
driver: bnx2x
version: 5.15.64-1-pve
firmware-version: mbi 255.255.255 FFV14.04.11 bc
expansion-rom-version:
bus-info: 0000:2d:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
As for the downloaded files I also need MBI file?

Code: 
         1)  lnxfwnx2-{arch}.sdk.tgz - SDK Package
         2)  LnxQlgcUpg.sh           - Script file
         3)  Marvell provided MBI file
Full paste of requirements are ========== RE - Pastebin.com
 
P

prdtabim

Active Member
Jan 29, 2022
175
67
28
Damo said:
Code: 
root@ENT1:~# ethtool -i enp45s0f0
driver: bnx2x
version: 5.15.64-1-pve
firmware-version: mbi 255.255.255 FFV14.04.11 bc
expansion-rom-version:
bus-info: 0000:2d:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@ENT1:~# ethtool -i enp45s0f1
driver: bnx2x
version: 5.15.64-1-pve
firmware-version: mbi 255.255.255 FFV14.04.11 bc
expansion-rom-version:
bus-info: 0000:2d:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
As for the downloaded files I also need MBI file?

Code: 
         1)  lnxfwnx2-{arch}.sdk.tgz - SDK Package
         2)  LnxQlgcUpg.sh           - Script file
         3)  Marvell provided MBI file
Full paste of requirements are ========== RE - Pastebin.com
Click to expand...
Unpack the .zip file.
Go to x86_64 folder.
Verify if the interfaces are up. If not up do ifconfig enp45s0f0 up and ifconfig enp45s0f1 up .
Execute "bash ./LnxQlgcUpg.sh" <--- this script will autodetect the cards and do the update. Wait until it ends.
 
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
Unpack the .zip file.
Go to x86_64 folder.
Verify if the interfaces are up. If not up do ifconfig enp45s0f0 up and ifconfig enp45s0f1 up .
Execute "bash ./LnxQlgcUpg.sh" <--- this script will autodetect the cards and do the update. Wait until it ends.
Click to expand...
I ran it but it showed (I have Linux_FWupg_34xx_84xx_2.11.14.zip)

Code: 
./LnxQlgcUpg.sh
Extracting package contents...

QLogic Firmware Upgrade Utility for Linux: v2.11.14

NIC is not supported.
Quitting program ...
Program Exit Code: (16)
Failed to upgraded MBI
however it looks like it has changed (this may have been from reinstalling the OS?)

Code: 
root@ent3:~/x86_64# ethtool -i enp45s0f0
driver: bnx2x
version: 5.15.64-1-pve
firmware-version: FFV08.07.26 bc 7.13.54
expansion-rom-version:
bus-info: 0000:2d:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
But the issue still persists.

pastebin.com

Nov 13 13:58:04 ENT2 pmxcfs[2434]: [dcdb] notice: starting data syncronisation - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com
 
P

prdtabim

Active Member
Jan 29, 2022
175
67
28
Damo said:
I ran it but it showed (I have Linux_FWupg_34xx_84xx_2.11.14.zip)

Code: 
./LnxQlgcUpg.sh
Extracting package contents...

QLogic Firmware Upgrade Utility for Linux: v2.11.14

NIC is not supported.
Quitting program ...
Program Exit Code: (16)
Failed to upgraded MBI
however it looks like it has changed (this may have been from reinstalling the OS?)

Code: 
root@ent3:~/x86_64# ethtool -i enp45s0f0
driver: bnx2x
version: 5.15.64-1-pve
firmware-version: FFV08.07.26 bc 7.13.54
expansion-rom-version:
bus-info: 0000:2d:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
But the issue still persists.

pastebin.com

Nov 13 13:58:04 ENT2 pmxcfs[2434]: [dcdb] notice: starting data syncronisation - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
pastebin.com pastebin.com
Click to expand...
This line looks different from before
Code: 
firmware-version: FFV08.07.26 bc 7.13.54
Did you reboot after the version is changed ? the new firmware need the reboot.

Did you execute "bash ./LnxQlgcUpg.sh" or tuns the file .sh executable ( chmod +x ) ?
In my case the 1st option works but not the second.

Another think ... if you execute "./lnxfwnx2" this bring up a console with many info including temperature . This console only opens if a compatible card is detected.
 
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
This line looks different from before
Code: 
firmware-version: FFV08.07.26 bc 7.13.54
Did you reboot after the version is changed ? the new firmware need the reboot.

Did you execute "bash ./LnxQlgcUpg.sh" or tuns the file .sh executable ( chmod +x ) ?
In my case the 1st option works but not the second.

Another think ... if you execute "./lnxfwnx2" this bring up a console with many info including temperature . This console only opens if a compatible card is detected.
Click to expand...
I did chmod it but I just tried again fresh install same issue.

./lnxfwnx2 does work but running "bash ./LnxQlgcUpg.sh" inside x86_64 folder instantly returns

Code: 
bash ./LnxQlgcUpg.sh
Extracting package contents...

QLogic Firmware Upgrade Utility for Linux: v2.11.14

NIC is not supported.
Quitting program ...
Program Exit Code: (16)
Failed to upgraded MBI
./lnxfwnx2 brings up

Code: 
***********************************************************************************************
QLogic Firmware Upgrade Utility for Linux v2.11.14
***********************************************************************************************

C  Brd      MAC      Drv                     Name                                        MFW
-  ---- ------------ --- ----------------------------------------------------------- ----------
0* 168E 000E1EAC6710 Yes BCM57810 10 Gigabit Ethernet rev 10 (enp45s0f0)             7.13.54
1  168E 000E1EAC6712 Yes BCM57810 10 Gigabit Ethernet rev 10 (enp45s0f1)             7.13.54
#1006500 - Missing bnx2x firmware 7.13.21.0 renders NIC unusable with Linux 5.16 - Debian Bug report logs talks about this too is Proxmox causing issues? running "update-initramfs -u" before installing Proxmox showed similar errors to the ones below but it doesn't after installing Proxmox

Code: 
W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e2-7.13.21.0.fw
for module bnx2x
W: Possible missing firmware
/lib/firmware/bnx2x/bnx2x-e1h-7.13.21.0.fw for module bnx2x
W: Possible missing firmware /lib/firmware/bnx2x/bnx2x-e1-7.13.21.0.fw
for module bnx2x
/lib/firmware/bnx2x also contains the files listed below (not sure if it does before proxmox installation)

Code: 
bnx2x-e1-7.13.11.0.fw  bnx2x-e1h-7.13.11.0.fw  bnx2x-e2-7.13.11.0.fw
bnx2x-e1-7.13.15.0.fw  bnx2x-e1h-7.13.15.0.fw  bnx2x-e2-7.13.15.0.fw
bnx2x-e1-7.13.21.0.fw  bnx2x-e1h-7.13.21.0.fw  bnx2x-e2-7.13.21.0.fw

I also have 3 of these cards all are showing different results for "ethtool -i enp45s0f0" (firmware-version is only different)

firmware-version: FFV14.04.11 bc 7.14.10
firmware-version: FFV08.07.26 bc 7.13.54
firmware-version: mbi 255.255.255 FFV08.07.26 bc
 
Last edited:
D

Damo

Active Member
Sep 7, 2022
113
30
28
P

prdtabim

Active Member
Jan 29, 2022
175
67
28
Damo said:
@prdtabim Could it just be this motherboard? ASRock X570D4U as it's working fine in my SuperMicro H12DSi-N6.

To get them working on the SuperMicro machine I had to disable OPROM I have done that on the ASRock too for PCIE5 (https://download.asrock.com/Manual/X570D4U.pdf) but made no difference. Is there anything else I could try in BIOS?

These are dirt cheap I think I may just get a few see if they work Solarflare SF432-1012 PCIe Dual port 10Gbps 2xSFP Network Card low profile | eBay
Click to expand...
I don't know. I use an ASROCK X570 Creator and an ASROCK X370 Profesional Gaming without problems with 57810 cards ( HP branded 10GBase-T ). The only parameter in bios was ARI/ACS to permit diferent groups for virtualization.
One last test ... do you try to enable flow control in both sides of the link ? this can detect overflow and pause the packet tansmission at ethernet level avoiding many types of errors with low impact in performance.
Code: 
ethtool -a enp45s0f0        to show 
ethtool -A enp45s0f0  rx on tx on     to set
I used Solarflare SFN7122F dual port too. This card have low power consumption ( 5.9w ).
 
  • Like
Reactions: Damo
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
I don't know. I use an ASROCK X570 Creator and an ASROCK X370 Profesional Gaming without problems with 57810 cards ( HP branded 10GBase-T ). The only parameter in bios was ARI/ACS to permit diferent groups for virtualization.
One last test ... do you try to enable flow control in both sides of the link ? this can detect overflow and pause the packet tansmission at ethernet level avoiding many types of errors with low impact in performance.
Code: 
ethtool -a enp45s0f0        to show
ethtool -A enp45s0f0  rx on tx on     to set
I used Solarflare SFN7122F dual port too. This card have low power consumption ( 5.9w ).
Click to expand...
I will give that a go need to look into how to do that on the Brocade ICX switch
 
D

Damo

Active Member
Sep 7, 2022
113
30
28
prdtabim said:
I don't know. I use an ASROCK X570 Creator and an ASROCK X370 Profesional Gaming without problems with 57810 cards ( HP branded 10GBase-T ). The only parameter in bios was ARI/ACS to permit diferent groups for virtualization.
One last test ... do you try to enable flow control in both sides of the link ? this can detect overflow and pause the packet tansmission at ethernet level avoiding many types of errors with low impact in performance.
Code: 
ethtool -a enp45s0f0        to show
ethtool -A enp45s0f0  rx on tx on     to set
I used Solarflare SFN7122F dual port too. This card have low power consumption ( 5.9w ).
Click to expand...
They where both already set to on
 
You must log in or register to reply here.
Top Bottom