DavidRa

How-to Guide C6100 - Fixing broken BMC / IPMI

How to fix a C6100 node with a broken IPMI controller (e.g. can't set IP address, wrong MAC)

Tags:
  1. DavidRa
    WARNING: Running the following procedure with the wrong BIOS or BMC ROM files can, and PROBABLY WILL permanently brick your node. Running the commands below can permanently brick your node ANYWAY if you look at it the wrong way.

    To continue, open Notepad or another text editor. Create a new document and type in this text:

    I understand that what I am about to do can permanently destroy, mutilate, damage and render into a useless pile of boards, components and metal, this node, the other nodes in the chassis, all the nodes in my rack or racks and the powered off computers on the floor. It may also shave my cat, give my dog diabetes, cause my car to rust and a meteorite to crash through the roof of my house, turning me into a wheelchair-bound quadriplegic.

    Done that? OK save it to your desktop as "I DIDNT LISTEN TO THE INSTRUCTIONS.TXT". Let's begin.

    You will need:
    BMC and BIOS ROMs
    Product Support | Dell US

    FreeDOS Images
    Index of /pub/FreeDOS/bootable-usb/
    I used: http://ftp.chtaube.eu/pub/FreeDOS/bootable-usb/FreeDOS-1.1-memstick-3-30M.img.bz2 - because the USB key I used is only 128MB (I keep it specifically for this sort of thing)

    Win32 Disk Imager
    http://sourceforge.net/projects/win32diskimager/files/latest/download

    PLD Rescue CD
    http://rescuecd.pld-linux.org/beta/RCDx86_13_03_10.iso

    USB Key
    At least 32MB in size, and it's going to be completely wiped, so back up NOW. 256MB is better, in which case you can use one of the larger FreeDOS images.

    1. Setup
    • Download FreeDOS boot image, Image Writer, ROMs, PLD Linux image
    • Install Win32DiskImager and write the FreeDOS image to USB key
    2. Use SOCFLASH to flash the BMC
    • Copy extracted files from Dell BMC download to USB key "BMC" directory - use the "Floppy" version, File Name: PEC6100BMC133.exe
    • Boot FreeDOS USB key
    • Specify A:\FDOS\COMMAND.COM /E:1024 /P for the location of the command interpreter, if prompted by FDOS boot code
    • Run these commands:
      • cd BMC\SOCFLASH\DOS
      • SOCFLASH -sf ..\5442M133.ROM
    • Remove power from chassis for 30 seconds, then plug back in and continue
    • This process will reset all BMC configuration (e.g. Shared/Dedicated NIC) and may ZERO the MAC (see step 4 to fix that).
    3. Use AFUDOS to flash BIOS
    • Copy extracted files from Dell BIOS download to USB "BIOS" directory - use the "Floppy" version, File Name: PEC6100BIOS017100.exe
    • Boot FreeDOS USB key
    • Specify A:\FDOS\COMMAND.COM /E:1024 /P for the location of the command interpreter, if prompted by FDOS boot code
    • Run these commands:
      • cd BIOS
      • AFUDOS 6100v171.ROM -x -reboot
    4. Use PLD to force IPMI MAC
    • Boot PLD CD image (I used my IP KVM, you could also try UNetBootIn, or real media ...)
    • Run these three commands to load IPMI drivers
      • modprobe ipmi_devintf
      • modprobe ipmi_si
      • modprobe ipmi_msghandler
    • Decide your new MAC address (in the form: NN:NN:NN:NN:NN:NN) and convert each hex digit and colon to its ASCII equivalent. I matched the first 8 digits to other BMC MAC addresses I have, then numbered from 02:00 to 02:03 based on the chassis and blade number. It's "Good Enough" (if you have hundreds of nodes and possible conflicts, you can easily choose your own scheme). Here's the mapping table you need:
      • 0x30 == "0"
      • 0x31 == "1"
      • 0x32 == "2"
      • 0x33 == "3"
      • 0x34 == "4"
      • 0x35 == "5"
      • 0x36 == "6"
      • 0x37 == "7"
      • 0x38 == "8"
      • 0x39 == "9"
      • 0x61 == "A"
      • 0x62 == "B"
      • 0x63 == "C"
      • 0x64 == "D"
      • 0x65 == "E"
      • 0x66 == "F"
      • 0x3a == ":"
    • Convert your MAC address into a series of hex values separated by spaces, and end it with 0x00. For example:
      • 00:14:25:8d:30:7f
    • Becomes this string which I'll call $MYDATA (yes you could set environment variables etc)
      • 0x30 0x30 0x3a 0x31 0x34 0x3a 0x32 0x35 0x3a 0x38 0x64 0x3a 0x33 0x30 0x3a 0x37 0x66 0x00
    • Run the following command:
      • ipmitool lan print 1 | grep "MAC Address"
        This should show the current address is 00:00:00:00:00:00
    • Run the following command, substitute your own numbers for $MYDATA:
      • ipmitool raw 0x2e 0x21 $MYDATA
      • ipmitool raw 0x2e 0x23 $MYDATA
      • ipmitool mc reset cold
    • Wait 30s
    • Run the following command:
      • ipmitool lan print 1 | grep "MAC Address"
        This should show your shiny new MAC address
    • Reboot, do your thing (install your OS, switch NIC to dedicated NIC etc)
    References:
    PLD LINUX Rescue CD
    Dell C6100 BMC / BIOS HELP - Cloud Services - Forum - Cloud Services - Dell Community
    Dell C6100 BMC / IPMI Mac address all zeros after update? 00:00:00:00:00:00 | Web Hosting Talk
    ipmi_on_linux [Adam Sweet's Wiki]
    IPMI HOWTO for Debian GNU/Linux on the Intel SR2300 (Server Board SE7501WV2)
    Radian, Patrick, legopc and 1 other person like this.

Recent Reviews

  1. Patrick
    Patrick
    5/5,
    Great how-to guide David!