Recovering bad Broadcom cards

Discussion in 'Networking' started by Stanza, Jun 21, 2014.

  1. Stanza

    Stanza Active Member

    Joined:
    Jan 11, 2014
    Messages:
    205
    Likes Received:
    40
    OK, I am in the process of trying to recover an IBM X3400 M3 server

    I have two of these, both identical ....

    Tho the problem child seems to have flakey onboard Broadcom NetXtreme II (dual port on motherboard) BCM5716C (similar to a BCM5709)

    From cold boot it's random if the nics work or not

    eg
    Plug power in
    boot
    go into bios
    Both cards have MAC address 00:00:00:00:00:00 and iSCSI mac is the same 00:00:00:00:00:00

    Try to upgrade firmwares gives no help as the devices are not seen

    unplug power, wait a minute or so, plug power back in and they seem to come to life

    both cards have proper Mac Addresses and can firmware update them no problems.

    Pull power again and they go back to the blank 00:00:00:00:00:00 Mac Addresses:mad:

    Google searching I came across this page
    HP Proliant DL380 G5 NIC's not found after firmwar... - HP Enterprise Business Community

    Which it seems HP had a problem of a firmware update killing broadcom BCM5708 cards a while back.

    So I thought I would give the fix a try, but naturally using they correct firmware / type etc

    so far so good

    Managed to overwrite the firmware with a slightly later one ..... from a package found on Supermicro's site believe it or not........as I couldn't find anything on the broadcom site.

    So now I am stuck trying to get the iSCSI Mac Addresses back onto the cards.

    in original x3400 Bios the cards were listed as (when they worked)

    =================
    Chip Type BCM5716 C0
    Perm Mac Address E4:1F:13:C2:E4:1C
    Virt Mac Address
    iSCSI MAC Address 00:1A:64:B3:49:31
    Virt iSCSI Mac Address same
    Bus : Dev:Func 0B:00:00

    Firmware image menu
    Bootcode 7.4.0
    MBA N/A
    iSCSI Boot N/A
    IPMI N/A
    NCSI 2.0.11
    UMP N/A

    Chip Type BCM5716 C0
    Perm Mac Address E4:1F:13:C2:E4:1D
    Virt Mac Address
    iSCSI MAC Address 00:1A:64:B3:49:33
    Virt iSCSI Mac Address same
    Bus : Dev:Func 0B:00:00

    Firmware image menu
    Bootcode 7.4.0
    MBA N/A
    iSCSI Boot N/A
    IPMI N/A
    NCSI 2.0.11
    UMP N/A

    ==========

    I have managed to put bootcode on, iSCSI bootcode on and also the NCSI code using the xdiag tool

    eg

    made a dos boot floppy
    on that floppy added the files
    xdiag.exe << firmware update program
    bc09x740.bin <<firmware
    ibootv76.50 <<iSCSI firmware
    ncsifw_x.2011 << NCSI firmware

    booted to floppy and issued command

    Code:
    xdiag -b06eng
    (which puts you in engineering mode)

    then issued command

    Code:
    nvm fill 0 0x600 0
    (which makes sure the first 1.5kb of flash content are zero's)

    then wrote the firmware to the card

    Code:
    nvm upgrade -bc bc09x740.bin
    (Firmware went on fine)

    then reset everything back to defaults

    Code:
    nvm cfg
    (scrolled to bottom and typed in "default")

    setup the mac address

    Code:
    1=E4:1F:13:C2:E4:1C
    (saved and exited)

    did the same for the second card......with it's mac address

    saved and rebooted / powered down server / repowered and checked all is good.

    but to add the iSCSI mac address back it seems I need to use the command

    Code:
    xdiag -c 1 -pwd -iscssimac 00:1A:64:B3:49:31
    manual says the option -iscsimac needs two other options -c and -pwd

    -c option is the card number
    -pwd option is the password

    What the hell is the password??:confused:

    anyone done this before and can give me a hint?
    or is it a secret password.......as in the iSCSI offload / boot option on NORMAL broadcom cards is a PAID addition? ala 10g cards which have FCOE options...

    anyone?
     
    #1
    Last edited: Jun 22, 2014
  2. Stanza

    Stanza Active Member

    Joined:
    Jan 11, 2014
    Messages:
    205
    Likes Received:
    40
    OK next I tried to perform a dump of the settings from the current GOOD working Broadcom controllers of the good server

    and overwrite the settings of the BAD server.

    Didn't work as there was a hardware mismatch.... Which I thought might happen ala cross flashing SAS controllers...... As Device ID numbers are different

    But looking through the dump output, gave me some hints as to what settings are on the GOOD server, and applied them to the BAD servers controller settings

    tho I am a bit suspicious that the firmware I have loaded is maybe not quite the right one. as the part number on the BAD server shows as a 5709 instead of a 5716 controller

    anyways onwards and upwards

    performing a nvm cfg dump showed me where the differences lay between the two

    sections of the nvm cfg file that are now different between the 2 Servers are

    items 1, 6, 7, 9, 14, 16, 21, 25, 39, 48, 61 below

    Code:
    #  1: [P] MAC Address                                     : 5c:f3:fc:55:ed:3a
    #  2: [P] NIC Power Dissipated (D3:D2:D1:D0)              : 25:0:0:25
    #  3: [P] NIC Power Consumed (D3:D2:D1:D0)                : 25:0:0:25
    #  4: [P] Vendor ID                                       : 14E4
    #  5: [P] Vendor Device ID                                : 163B
    #  6: [P] Subsystem Vendor ID                             : 1014
    #  7: [P] Subsystem Device ID                             : 03E0
    #  8: [S] Product Name                                    : Broadcom NetXtreme II Ethernet Controller
    #  9: [S] Part Number                                     : BCM95716A0
    # 10: [S] Engineering Change                              : 220197-2
    # 11: [S] Serial Number                                   : 0123456789
    # 12: [S] Manufacturing ID                                : 14e4
    # 13: [P] Led Mode { MacMode(0), PhyMode1(1),
    #         PhyMode2(2),..., PhyMode7(7) }                  : PhyMode1
    # 14: [S] Design Type: {NIC(0), LOM(1)}                   : LOM
    # 16: [P] BAR size: {Disabled(0), 64K(1), 128K(2),
    #         ...1M(5),...16M(9), ...64M(11), ...1G(15)}      : 32M
    # 17: [P] Magic Packet WoL { Disable(0), Enable(1) }      : Enabled
    # 18: [P] WoL link speed { Auto(0), ...1000FD(6) }        : Auto
    # 21: [P] MBA { Disable(0), Enable(1) }                   : Disabled
    # 22: [P] MBA Link Speed { Auto(0), ...1000FD(6),
    #         2500FD(8)}                                      : Auto
    # 23: [P] MBA Boot Protocol { PXE(0), RPL(1), BOOTP(2),
    #         iSCSI boot(3), reserved(4-6), None(7) }         : PXE
    # 24: [P] MBA Boot Type { Auto(0), BBS(1), Int18(2),
    #     Int19(3) }                                          : Auto
    # 25: [P] MBA Delay Time (0-15) (0 defaults to 4 seconds,  
    #     15 is no delay, 1-14 is delay value in second)      : 2
    # 26: [P] MBA Setup Hot Key {Ctrl-S(0), Ctrl-B(1)}        : Ctrl-S
    # 27: [P] MBA hide setup prompt {Disable(0), Enable(1)}   : Enabled
    # 28: [S] Expansion ROM size { Disabled(0),
    #         ...64k(7),...16M(15) }                          : Disabled
    # 29: [S] Mgmt Firmware { Disable(0), Enable(1) }         : Enabled
    # 34: [P] MBA VLAN { Disable(0), Enable(1) }              : Disabled
    # 35: [P] MBA VLAN value                                  : 0
    # 36: [S] Vaux current overdraw { Disable(0), Enable(1) } : Disabled
    # 37: [S] UMP/NCSI uses RMII { Disable/MII only(0),
    #         Enable(1) }                                     : Enabled
    # 39: [S] Spare Number                                    : BCM95716A0
    # 43: [S] UMP/NCSI PHY Timing (5708 B0 and after)
    #         {Disable(0),Enable(1)}                          : Disabled
    # 44: [P] SMBus Address {one byte even hex value
    #         (i.e.bit0=0)}                                   : 0x00
    # 47: [S] UMP Echo mode { Disabled(0), Enable(1) }        : Disabled
    # 48: [S] Allow gigabit link on Vaux
    #         {Disabled(0), Enable(1)}                        : Enabled
    # 49: [P] Default link setting (serdes only) {Autoneg(0),
    #         forced 1G(3), forced 2.5G(4), AN w/1G(19),
    #         AN w/2.5G(20)}                                  : Autoneg
    # 58: [S] Read only VPD Vendor Specific Data (V0)         :
    # 59: [S] Port swap { Disable(0), Enable(1) }
    #         (5709/16 only)                                  : Disabled
    # 61: [S] PCIE Gen2 (5709/16 only) { Disable(0),
    #         Enable(1) }                                     : Enabled
    # 62: [S] Function hide {None(0), Reserved(1), Func1(2)}
    #         (5709/16 only)                                  : None
    # 63: [S] Mgmt FW load choice { Any(0), NCSI(1), UMP(2),
    #         IPMI(3), NCSI/IPMI(4), UMP/IPMI(5), NCSI/UMP(6) }
    #         (4, 5, 6 are determined by SPIO4 (0/1), 5709
    #         & 5716 only)                                    : Any
    # 64: [S] NCSI Package ID assignment method { SPIO(0),
    #         NVRAM(1) } (5709/16 only)                       : NVRAM
    # 65: [S] NCSI Package ID assigned value { 0-3 } (5709/16): 0
    # 66: [S] SMBus Timing { 100kHz(0), 400kHz(1) } (5709/16) : 100kHz
    # 67: [S] PCIE power budget data 0 (Bits 20-0) (5709/16)  : 0x000000
    # 68: [S] PCIE power budget data 1 (Bits 20-0) (5709/16)  : 0x000000
    # 69: [S] PCIE power budget data 2 (Bits 20-0) (5709/16)  : 0x000000
    # 70: [S] PCIE power budget data 3 (Bits 20-0) (5709/16)  : 0x000000
    # 71: [S] PCIE power budget data 4 (Bits 20-0) (5709/16)  : 0x000000
    # 72: [S] PCIE power budget data 5 (Bits 20-0) (5709/16)  : 0x000000
    # 73: [S] PCIE power budget data 6 (Bits 20-0) (5709/16)  : 0x000000
    # 74: [S] PCIE power budget data 7 (Bits 20-0) (5709/16)  : 0x000000
    # 75: [S] Prevent PCIE L1-entry {Disable(0), Enable(1)}
    #         (5708, 5709/5716 only)                          : Disabled
    # 76: [P] Prevent PCIE relax ordering {Disable(0),
    #     Enable(1)} (5709/16)                                : Disabled
    # 77: [P] Backup primary MAC address                      : 00:00:00:00:00:00
    # 78: [P] Force Expansion ROM advertisement { Disable(0),
    #         Enable(1) }                                     : Disabled
    # 79: [S] PCIE Gen2 Preemphasis: { HW(0), 0dB(1),      
    #         3_5dB(2), 6_0dB(3) } (5709/16)                  : HW
    # 80: [P] MBA Boot Retry Count: {0, 1,...6, 7(Indefinite)}: 0
    These I have matched back up correctly, using the nvm cfg commands

    going to check the license keys, but it doesn't seem to work on a 5716 controller??.......so I am not sure what the iSCSI boot parameters in the BIOS are all about ??

    Now to see if I can perform a dump of the the actual NVRAM of the good Servers Broadcom cards and can I load them on the BAD Server?

    if so can just a change of the MAC Addresses fix it up?

    Code:
    1.NONE:> nvm dump TESTDMP.BIN
    which should dump the entire NVRAM into the file TESTDMP.BIN

    And then, can we force load this back on the BAD Server? using the option -no_chip_chk

    lets try to load it

    Code:
    nvm prg 0 TESTDMP.BIN
    
    Programming from: 0x00000000 to 0x00040000
    Status: 0x00040000
    only errors shown are about ISB_CFG not being in the Image file......and I can't find any info as to what ISB_CFG is ?? can only find one google reference which I believe is for the LED BLINK to identify a Port through the BIOS options

    reboot and what do we get?

    the NVRAM overwrite worked, and the original MAC Addresses remained.....tho I still have iSCSI Mac's of 00:00:00 etc.....

    so not sure what gives there...... but I doubt I will ever use iSCSI boot on these.... so the rest can wait for another day.

    .
     
    #2
    Last edited: Jun 22, 2014
  3. parawizard

    parawizard New Member

    Joined:
    Jan 28, 2014
    Messages:
    21
    Likes Received:
    0
    What I saw in the manual was most functions for broadcom utilities had -pwd for field technician use only. Did you try sending them an email?

    ftp://ftp.supermicro.nl/driver/Broadcom/Previous Releases/B57BCMCD_T7.2.4.2/dos/onchip_firmware/PXE/NX2/release.txt

    (Forum kills link no matter what I do. Does not handle ftp links properly it seems. Have to copy and paste)

    I saw in here as well it mentions added support for BCM5716C and that setting isci mac would with this would be

    To add to this here is the PDF manual for the CLP utility as well that use that command above...

    ftp://ftp.supermicro.nl/driver/Broa..._T7.2.4.2/dos/onchip_firmware/PXE/NX2/clp.pdf

    (Forum kills link no matter what I do. Does not handle ftp links properly it seems. Have to copy and paste)

     
    #3
    Last edited: Jul 5, 2014
Similar Threads: Recovering Broadcom
Forum Title Date
Networking Recovering a Packeteer Packetshaper 6500 Jan 17, 2017
Networking Dell W1GCR HN10N Broadcom 57810S unRAID & Windows 10 drivers? Mar 6, 2020
Networking Broadcom BCM57416 10Gbe vs Intel X550 10Gbe adapters? Sep 29, 2019
Networking anyone know how to recover a broadcom NIC from bad firmware? Sep 6, 2019
Networking New broadcom programmable switch chip Jun 23, 2019

Share This Page