Recovering bad Broadcom cards

Stanza

Active Member
Jan 11, 2014
205
41
28
OK, I am in the process of trying to recover an IBM X3400 M3 server

I have two of these, both identical ....

Tho the problem child seems to have flakey onboard Broadcom NetXtreme II (dual port on motherboard) BCM5716C (similar to a BCM5709)

From cold boot it's random if the nics work or not

eg
Plug power in
boot
go into bios
Both cards have MAC address 00:00:00:00:00:00 and iSCSI mac is the same 00:00:00:00:00:00

Try to upgrade firmwares gives no help as the devices are not seen

unplug power, wait a minute or so, plug power back in and they seem to come to life

both cards have proper Mac Addresses and can firmware update them no problems.

Pull power again and they go back to the blank 00:00:00:00:00:00 Mac Addresses:mad:

Google searching I came across this page
HP Proliant DL380 G5 NIC's not found after firmwar... - HP Enterprise Business Community

Which it seems HP had a problem of a firmware update killing broadcom BCM5708 cards a while back.

So I thought I would give the fix a try, but naturally using they correct firmware / type etc

so far so good

Managed to overwrite the firmware with a slightly later one ..... from a package found on Supermicro's site believe it or not........as I couldn't find anything on the broadcom site.

So now I am stuck trying to get the iSCSI Mac Addresses back onto the cards.

in original x3400 Bios the cards were listed as (when they worked)

=================
Chip Type BCM5716 C0
Perm Mac Address E4:1F:13:C2:E4:1C
Virt Mac Address
iSCSI MAC Address 00:1A:64:B3:49:31
Virt iSCSI Mac Address same
Bus : Dev:Func 0B:00:00

Firmware image menu
Bootcode 7.4.0
MBA N/A
iSCSI Boot N/A
IPMI N/A
NCSI 2.0.11
UMP N/A

Chip Type BCM5716 C0
Perm Mac Address E4:1F:13:C2:E4:1D
Virt Mac Address
iSCSI MAC Address 00:1A:64:B3:49:33
Virt iSCSI Mac Address same
Bus : Dev:Func 0B:00:00

Firmware image menu
Bootcode 7.4.0
MBA N/A
iSCSI Boot N/A
IPMI N/A
NCSI 2.0.11
UMP N/A

==========

I have managed to put bootcode on, iSCSI bootcode on and also the NCSI code using the xdiag tool

eg

made a dos boot floppy
on that floppy added the files
xdiag.exe << firmware update program
bc09x740.bin <<firmware
ibootv76.50 <<iSCSI firmware
ncsifw_x.2011 << NCSI firmware

booted to floppy and issued command

Code:
xdiag -b06eng
(which puts you in engineering mode)

then issued command

Code:
nvm fill 0 0x600 0
(which makes sure the first 1.5kb of flash content are zero's)

then wrote the firmware to the card

Code:
nvm upgrade -bc bc09x740.bin
(Firmware went on fine)

then reset everything back to defaults

Code:
nvm cfg
(scrolled to bottom and typed in "default")

setup the mac address

Code:
1=E4:1F:13:C2:E4:1C
(saved and exited)

did the same for the second card......with it's mac address

saved and rebooted / powered down server / repowered and checked all is good.

but to add the iSCSI mac address back it seems I need to use the command

Code:
xdiag -c 1 -pwd -iscssimac 00:1A:64:B3:49:31
manual says the option -iscsimac needs two other options -c and -pwd

-c option is the card number
-pwd option is the password

What the hell is the password??:confused:

anyone done this before and can give me a hint?
or is it a secret password.......as in the iSCSI offload / boot option on NORMAL broadcom cards is a PAID addition? ala 10g cards which have FCOE options...

anyone?
 
Last edited:

Stanza

Active Member
Jan 11, 2014
205
41
28
OK next I tried to perform a dump of the settings from the current GOOD working Broadcom controllers of the good server

and overwrite the settings of the BAD server.

Didn't work as there was a hardware mismatch.... Which I thought might happen ala cross flashing SAS controllers...... As Device ID numbers are different

But looking through the dump output, gave me some hints as to what settings are on the GOOD server, and applied them to the BAD servers controller settings

tho I am a bit suspicious that the firmware I have loaded is maybe not quite the right one. as the part number on the BAD server shows as a 5709 instead of a 5716 controller

anyways onwards and upwards

performing a nvm cfg dump showed me where the differences lay between the two

sections of the nvm cfg file that are now different between the 2 Servers are

items 1, 6, 7, 9, 14, 16, 21, 25, 39, 48, 61 below

Code:
#  1: [P] MAC Address                                     : 5c:f3:fc:55:ed:3a
#  2: [P] NIC Power Dissipated (D3:D2:D1:D0)              : 25:0:0:25
#  3: [P] NIC Power Consumed (D3:D2:D1:D0)                : 25:0:0:25
#  4: [P] Vendor ID                                       : 14E4
#  5: [P] Vendor Device ID                                : 163B
#  6: [P] Subsystem Vendor ID                             : 1014
#  7: [P] Subsystem Device ID                             : 03E0
#  8: [S] Product Name                                    : Broadcom NetXtreme II Ethernet Controller
#  9: [S] Part Number                                     : BCM95716A0
# 10: [S] Engineering Change                              : 220197-2
# 11: [S] Serial Number                                   : 0123456789
# 12: [S] Manufacturing ID                                : 14e4
# 13: [P] Led Mode { MacMode(0), PhyMode1(1),
#         PhyMode2(2),..., PhyMode7(7) }                  : PhyMode1
# 14: [S] Design Type: {NIC(0), LOM(1)}                   : LOM
# 16: [P] BAR size: {Disabled(0), 64K(1), 128K(2),
#         ...1M(5),...16M(9), ...64M(11), ...1G(15)}      : 32M
# 17: [P] Magic Packet WoL { Disable(0), Enable(1) }      : Enabled
# 18: [P] WoL link speed { Auto(0), ...1000FD(6) }        : Auto
# 21: [P] MBA { Disable(0), Enable(1) }                   : Disabled
# 22: [P] MBA Link Speed { Auto(0), ...1000FD(6),
#         2500FD(8)}                                      : Auto
# 23: [P] MBA Boot Protocol { PXE(0), RPL(1), BOOTP(2),
#         iSCSI boot(3), reserved(4-6), None(7) }         : PXE
# 24: [P] MBA Boot Type { Auto(0), BBS(1), Int18(2),
#     Int19(3) }                                          : Auto
# 25: [P] MBA Delay Time (0-15) (0 defaults to 4 seconds,  
#     15 is no delay, 1-14 is delay value in second)      : 2
# 26: [P] MBA Setup Hot Key {Ctrl-S(0), Ctrl-B(1)}        : Ctrl-S
# 27: [P] MBA hide setup prompt {Disable(0), Enable(1)}   : Enabled
# 28: [S] Expansion ROM size { Disabled(0),
#         ...64k(7),...16M(15) }                          : Disabled
# 29: [S] Mgmt Firmware { Disable(0), Enable(1) }         : Enabled
# 34: [P] MBA VLAN { Disable(0), Enable(1) }              : Disabled
# 35: [P] MBA VLAN value                                  : 0
# 36: [S] Vaux current overdraw { Disable(0), Enable(1) } : Disabled
# 37: [S] UMP/NCSI uses RMII { Disable/MII only(0),
#         Enable(1) }                                     : Enabled
# 39: [S] Spare Number                                    : BCM95716A0
# 43: [S] UMP/NCSI PHY Timing (5708 B0 and after)
#         {Disable(0),Enable(1)}                          : Disabled
# 44: [P] SMBus Address {one byte even hex value
#         (i.e.bit0=0)}                                   : 0x00
# 47: [S] UMP Echo mode { Disabled(0), Enable(1) }        : Disabled
# 48: [S] Allow gigabit link on Vaux
#         {Disabled(0), Enable(1)}                        : Enabled
# 49: [P] Default link setting (serdes only) {Autoneg(0),
#         forced 1G(3), forced 2.5G(4), AN w/1G(19),
#         AN w/2.5G(20)}                                  : Autoneg
# 58: [S] Read only VPD Vendor Specific Data (V0)         :
# 59: [S] Port swap { Disable(0), Enable(1) }
#         (5709/16 only)                                  : Disabled
# 61: [S] PCIE Gen2 (5709/16 only) { Disable(0),
#         Enable(1) }                                     : Enabled
# 62: [S] Function hide {None(0), Reserved(1), Func1(2)}
#         (5709/16 only)                                  : None
# 63: [S] Mgmt FW load choice { Any(0), NCSI(1), UMP(2),
#         IPMI(3), NCSI/IPMI(4), UMP/IPMI(5), NCSI/UMP(6) }
#         (4, 5, 6 are determined by SPIO4 (0/1), 5709
#         & 5716 only)                                    : Any
# 64: [S] NCSI Package ID assignment method { SPIO(0),
#         NVRAM(1) } (5709/16 only)                       : NVRAM
# 65: [S] NCSI Package ID assigned value { 0-3 } (5709/16): 0
# 66: [S] SMBus Timing { 100kHz(0), 400kHz(1) } (5709/16) : 100kHz
# 67: [S] PCIE power budget data 0 (Bits 20-0) (5709/16)  : 0x000000
# 68: [S] PCIE power budget data 1 (Bits 20-0) (5709/16)  : 0x000000
# 69: [S] PCIE power budget data 2 (Bits 20-0) (5709/16)  : 0x000000
# 70: [S] PCIE power budget data 3 (Bits 20-0) (5709/16)  : 0x000000
# 71: [S] PCIE power budget data 4 (Bits 20-0) (5709/16)  : 0x000000
# 72: [S] PCIE power budget data 5 (Bits 20-0) (5709/16)  : 0x000000
# 73: [S] PCIE power budget data 6 (Bits 20-0) (5709/16)  : 0x000000
# 74: [S] PCIE power budget data 7 (Bits 20-0) (5709/16)  : 0x000000
# 75: [S] Prevent PCIE L1-entry {Disable(0), Enable(1)}
#         (5708, 5709/5716 only)                          : Disabled
# 76: [P] Prevent PCIE relax ordering {Disable(0),
#     Enable(1)} (5709/16)                                : Disabled
# 77: [P] Backup primary MAC address                      : 00:00:00:00:00:00
# 78: [P] Force Expansion ROM advertisement { Disable(0),
#         Enable(1) }                                     : Disabled
# 79: [S] PCIE Gen2 Preemphasis: { HW(0), 0dB(1),      
#         3_5dB(2), 6_0dB(3) } (5709/16)                  : HW
# 80: [P] MBA Boot Retry Count: {0, 1,...6, 7(Indefinite)}: 0
These I have matched back up correctly, using the nvm cfg commands

going to check the license keys, but it doesn't seem to work on a 5716 controller??.......so I am not sure what the iSCSI boot parameters in the BIOS are all about ??

Now to see if I can perform a dump of the the actual NVRAM of the good Servers Broadcom cards and can I load them on the BAD Server?

if so can just a change of the MAC Addresses fix it up?

Code:
1.NONE:> nvm dump TESTDMP.BIN
which should dump the entire NVRAM into the file TESTDMP.BIN

And then, can we force load this back on the BAD Server? using the option -no_chip_chk

lets try to load it

Code:
nvm prg 0 TESTDMP.BIN

Programming from: 0x00000000 to 0x00040000
Status: 0x00040000
only errors shown are about ISB_CFG not being in the Image file......and I can't find any info as to what ISB_CFG is ?? can only find one google reference which I believe is for the LED BLINK to identify a Port through the BIOS options

reboot and what do we get?

the NVRAM overwrite worked, and the original MAC Addresses remained.....tho I still have iSCSI Mac's of 00:00:00 etc.....

so not sure what gives there...... but I doubt I will ever use iSCSI boot on these.... so the rest can wait for another day.

.
 
Last edited:

parawizard

New Member
Jan 28, 2014
21
0
1
What I saw in the manual was most functions for broadcom utilities had -pwd for field technician use only. Did you try sending them an email?

ftp://ftp.supermicro.nl/driver/Broadcom/Previous Releases/B57BCMCD_T7.2.4.2/dos/onchip_firmware/PXE/NX2/release.txt

(Forum kills link no matter what I do. Does not handle ftp links properly it seems. Have to copy and paste)

I saw in here as well it mentions added support for BCM5716C and that setting isci mac would with this would be

4. Problem: iSCSI MAC address cannot not configured with the following
CLP string:
"set netport2 PermanentAddress=xxxxxxxxxxxx"
(CQ#36962)

Change : Made changes to support above CLP string. User can still
configure iSCSI MAC address with "set netport1 iSCSIAddresss=xxxxxxxxxxxx"
as before.
To add to this here is the PDF manual for the CLP utility as well that use that command above...

ftp://ftp.supermicro.nl/driver/Broa..._T7.2.4.2/dos/onchip_firmware/PXE/NX2/clp.pdf

(Forum kills link no matter what I do. Does not handle ftp links properly it seems. Have to copy and paste)

7.6 iSCSI MAC Address

set <target> OEM<vendor>_iScsiAddress=<12 hex digits>

Description – iSCSI MAC Address
Values – 12 hexadecimal digits
Example – set netport1 OEMHP_iScsiAddress=001018123457

Note : User can also configure iSCSI MAC address with the following command:
set netport2 OEMHP_PermanentAddress=0010181234567

Note: Specifying zero MAC address will set MAC address to factory default
[/quote

I don't know if this will be at all helpful but it was what I was able to google up.
 
Last edited: