BNT/IBM G8124 firmware upgrade

Discussion in 'Networking' started by hanshans, Apr 7, 2017.

  1. hanshans

    hanshans New Member

    Joined:
    Dec 14, 2013
    Messages:
    5
    Likes Received:
    1
    Dear colleagues,

    I set up a ceph cluster for the lab and therefore was looking for payable 10G switches. So I slipped over the G8124 as you can get those for a good price. The switches basically do what they should. Nevertheless the firmware installed does not support vLAG, which I want to use to achieve bonding over two switches.
    I used the latest IBM firmware to update one of the devices but bricked ist. System does not boot any more.

    Code:
    Software Version 1.1.1.5, Boot Version 1.1.1.5, active config block
    
    Manufacturing Date:     1210
    Hardware Revision:       48
    Board Revision:          2
    PLD Firmware Version:    0x0
    Hardware ASIC Info:      Family FM4000, Model FM4224, Version FM4224_A3
    
    My question is: is there a way of update them to a newer firmware or has there been a change in hardware, that will limit upgrade possibilities? Someone has those switches in use and already did the update?

    Thanks for your help.
     
    #1
    legopc likes this.
  2. Sanjay Kumar

    Sanjay Kumar New Member

    Joined:
    Jan 24, 2017
    Messages:
    1
    Likes Received:
    0
    Hi,

    Did you manage to revive your bricked 8124 switch?

    I have just bricked mine. It had firmware version 1.1.13 and I upgraded to 7.7.5. After the upgrade it didn't boot. All lights stays green and no console.

    I found some info but this procedure didn't work as I didn't get anything on console.

    "Symptom: All the switch LEDs stay on, and the command prompt is not displayed
    on the console.
    Solution: The operating system might be damaged. Use the console port to
    perform a serial upgrade of the switch firmware. See the Command Reference"

    Thanks
     
    #2
  3. abq

    abq Active Member

    Joined:
    May 23, 2015
    Messages:
    374
    Likes Received:
    53
    Thank You for the heads-up. I was not aware of the firmware upgrade risk. My understanding was that the Lenovo/IBM switch upgrades were straight forward, and I have not seen any special notice of hardware revision issues. ...This sounds like a definite bummer. I will post back if I find anything.
     
    #3
  4. Elijah Neo

    Elijah Neo New Member

    Joined:
    Dec 21, 2014
    Messages:
    2
    Likes Received:
    0
    Anyone managed to bring the switch back to life?
    I did the upgrade and have the same "All the switch LEDs stay on, and the command prompt is not displayed on the console." problem.
    I opened the switch and found a JTAG connector, maybe it can be reprogrammed using the JTAG connector.
     
    #4
  5. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    Well it looks like I joined the club here. Picked up a G8124 on eBay. It was working fine and I figured I'd update the firmware before I started doing any real configuration. Pulled down the latest firmware bundle from IBM's web site and used the web interface (and TFTP) to flash both the boot and OS images... and after reboot it was dead. Nothing on the console. Wish I had found this thread before I started :(

    All of the instructions online say to hold "SHIFT-B" to get into the boot bios but it seems that the problem is that the boot bios is hosed. Oh well. Was really looking forward to using this switch :(
     
    #5
  6. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    are you in the US? and if so, where at
     
    #6
  7. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    I'm in Northern Virginia (DC area). Any prospect of resurrecting this switch? Trying to come to grips with just recycling it and getting a different one on eBay (like a Quanta LB6M or something). I paid $250 for this one so sucks flushing that down the drain :( I went with this one because I have a G8000 that has been great.. and the first thing I did when I got that one was to upgrade the firmware..
     
    #7
  8. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    I'm pretty sure the G8124 is powerPC based, powerPC JTAG units are rare but I have one. If you cover shipping (probably 12 dollars) I can mail you the jtag unit and Ill give you instructions similar to this JTAG Recovery - Fohdeesha Docs

    then once recovered you'd mail me the unit back. Can you take the lid off and take a high resolution of the internal board?
     
    #8
  9. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    Awesome, thanks! I'm trying to get the case off -- a couple of the screws are seized or have tamper glue on them. Looks like I'm going to need to drill them out. I'll add some photos after I get the case lid off..
     
    #9
  10. PnoT

    PnoT Active Member

    Joined:
    Mar 1, 2015
    Messages:
    560
    Likes Received:
    113
    I've had my G8000 and G8124 for about 3 years and have always manually applied the firmware from IBM via CLI / TFTP without issues. I wonder why a rash of people are having problems now? Sorry to hear about your issues but hopefully fohdeesha can get you sorted out.
     
    #10
    fohdeesha likes this.
  11. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    Attached are some photos of the logic board. There's definitely a JTAG header. I'm not sure if it's a PPC since I wasn't ready to remove the CPU heatsink.. and I didn't see anything that obviously indicated chip type.. but I'm guessing you are correct. Let me know if these photos help:

    Imgur: The magic of the Internet

    (BTW, I tried to upload the photos via the forum system but got a generic error message.. maybe they were too big?)
     
    #11
  12. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    definitely powerPC:

    Code:
    root@testing:~# binwalk G8124-7.11.12.0_Boot.img
    
    DECIMAL       HEXADECIMAL     DESCRIPTION
    --------------------------------------------------------------------------------
    1487081       0x16B0E9        uImage header, header size: 64 bytes, header CRC: 0x3076DAC2, created: Mon Mar  5 09:54:33 2018, image size: 6827043 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0xE7106728, OS: Linux, CPU: PowerPC, image type: RAMDisk Image, compression type: gzip, image name: "scooter ramdisk"
    
    so the powerPC JTAG header is the 16 pin one labeled ICE, the 10 pin header labeled JTAG is the JTAG interface for the intel switching ASIC

    should be fixable pretty easily
     
    #12
  13. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    I'll need the model number of the onboard flash to be sure I can write to it over jtag but I don't see it anywhere obvious in the pictures, can you look/take a high res picture of this area? I think it may be one of the circled chips covered up by the cabling

    [​IMG]
     
    #13
  14. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    Sure thing, I'll try to get some better pictures. And thanks for your assistance with this!
     
    #14
  15. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    nevermind found it hidden in your first picture, Spansion GL512N, can write to that no problem
     
    #15
  16. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    Cool, let me know what the next steps are. If you are willing to loan me your JTAG unit I'd love to give it a whirl. Obviously happy to cover any shipping costs, etc.

    I haven't done any HW/firmware work at this level, but you've piqued my interest! I'm confident I can follow the instructions from the document you linked earlier in the thread, perhaps with any other guidance needed for this specific box.

    After I saw your info about 'binwalk' I got that installed and poked around the G8124 image files a bit. Before I flashed the boot firmware, I backed up the one that was on the switch when it was delivered. Here's what binwalk says about the original bootrom and here's what it says with the one I tried to flash:

    1.1.1.5 Boot:
    Code:
    $ binwalk -e ../G8124-1.1.1.5_Boot.img
    
    
    DECIMAL       HEXADECIMAL     DESCRIPTION
    --------------------------------------------------------------------------------
    16            0x10            uImage header, header size: 64 bytes, header CRC: 0xF05DA519, created: 2009-10-28 19:04:08, image size: 1249723 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0x8DFAAB05, OS: Linux, CPU: PowerPC, image type: OS Kernel Image, compression type: gzip, image name: "Linux-2.6.22"
    80            0x50            gzip compressed data, maximum compression, from Unix, last modified: 2009-10-28 19:04:03
    1249819       0x13121B        uImage header, header size: 64 bytes, header CRC: 0x6757EA8E, created: 2009-06-20 11:55:17, image size: 5700677 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0xF079E9FB, OS: Linux, CPU: PowerPC, image type: RAMDisk Image, compression type: gzip, image name: "RootFileSystem"
    1249883       0x13125B        gzip compressed data, maximum compression, has original file name: "root", from Unix, last modified: 2009-06-20 11:48:23
    6950576       0x6A0EB0        uImage header, header size: 64 bytes, header CRC: 0x552D426F, created: 2029-04-03 03:29:53, image size: 775040560 bytes, Data Address: 0x20285365, Entry Point: 0x70203234, data CRC: 0x20323030, image name: "08:44:34) *** Blade Network Technologies - RS8124 Platform ***"
    6950580       0x6A0EB4        U-Boot version string, "U-Boot 1.2.0 (Sep 24 2009 - 08:44:34) *** Blade Network Technologies - RS8124 Platform ***"
    7106224       0x6C6EB0        CRC32 polynomial table, big endian
    
    7.11.13_0 Boot:
    Code:
    $ binwalk -e ../G8124-7.11.13.0_Boot.img
    
    DECIMAL       HEXADECIMAL     DESCRIPTION
    --------------------------------------------------------------------------------
    220           0xDC            gzip compressed data, maximum compression, has original file name: "vmlinux.bin.2849", from Unix, last modified: 2017-08-29 21:51:30
    1487081       0x16B0E9        uImage header, header size: 64 bytes, header CRC: 0x3076DAC2, created: 2018-03-05 14:54:33, image size: 6827043 bytes, Data Address: 0x0, Entry Point: 0x0, data CRC: 0xE7106728, OS: Linux, CPU: PowerPC, image type: RAMDisk Image, compression type: gzip, image name: "scooter ramdisk"
    1487145       0x16B129        gzip compressed data, has original file name: "scooter_new_rfs", from Unix, last modified: 2018-03-05 14:54:31
    8314204       0x7EDD5C        uImage header, header size: 64 bytes, header CRC: 0x552D426F, created: 2029-04-03 03:29:54, image size: 808466734 bytes, Data Address: 0x30362028, Entry Point: 0x41756720, data CRC: 0x32302032, image name: "- 02:00:10)"
    8501464       0x81B8D8        CRC32 polynomial table, big endian
    8541368       0x8254B8        device tree image (dtb)
    
    I extracted the bits from each image and poked around a little more. It seems like each use a different boot image structure. Wonder if there is an issue with hardware rev compatibility at some point that wasn't documented :(

    1.1.1.5 Boot:
    Code:
    $ ls -al
    total 64012
    drwxrwxr-x 2 arion arion     4096 Nov 18 20:45 .
    drwxrwxr-x 4 arion arion     4096 Nov 18 20:45 ..
    -rw-rw-r-- 1 arion arion  2621573 Nov 18 20:45 50
    -rw-rw-r-- 1 arion arion 62914560 Nov 18 20:45 root
    $ file 50 && file root
    50: data
    root: Linux rev 1.0 ext2 filesystem data, UUID=6288004b-e3d7-455f-9e73-4a572f344df7
    $ hexdump -C ../../G8124-1.1.1.5_Boot.img | head -10
    00000000  22 b7 e4 4f 00 13 11 fb  03 53 43 4f 01 01 01 05  |"..O.....SCO....|
    00000010  27 05 19 56 f0 5d a5 19  4a e8 95 a8 00 13 11 bb  |'..V.]..J.......|
    00000020  00 00 00 00 00 00 00 00  8d fa ab 05 05 07 02 01  |................|
    00000030  4c 69 6e 75 78 2d 32 2e  36 2e 32 32 00 00 00 00  |Linux-2.6.22....|
    00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    00000050  1f 8b 08 00 a3 95 e8 4a  02 03 d4 5a 0f 70 14 65  |.......J...Z.p.e|
    00000060  96 7f 3d 33 81 49 32 2c  dd 49 c0 28 a8 13 0d 98  |..=3.I2,.I.(....|
    00000070  7f 4a 44 bc ed 21 13 ec  84 78 db 74 a2 37 c8 e4  |.JD..!...x.t.7..|
    00000080  8c 88 da 63 82 02 89 6b  a2 b1 ae 75 bb 9d 9e 49  |...c...k...u...I|
    00000090  e3 41 c8 5e cd 24 a9 2b  2e 20 0e 18 fa 30 dc ee  |.A.^.$.+. ...0..|
    
    7.11.13_0 Boot:
    Code:
    $ ls -al
    total 19112
    drwxrwxr-x 2 arion arion     4096 Nov 18 20:44 .
    drwxrwxr-x 4 arion arion     4096 Nov 18 20:45 ..
    -rw-rw-r-- 1 arion arion 16332288 Nov 18 20:44 scooter_new_rfs
    -rw-rw-r-- 1 arion arion  3226288 Nov 18 20:44 vmlinux.bin.2849
    $ file vmlinux.bin.2849 && file scooter_new_rfs
    vmlinux.bin.2849: data
    scooter_new_rfs: ASCII cpio archive (SVR4 with no CRC)
    $ hexdump -C ../../G8124-7.11.13.0_Boot.img | head -10
    00000000  53 bc 11 33 00 16 b0 3d  03 53 43 4f 07 0b 0d 00  |S..3...=.SCO....|
    00000010  48 44 52 33 00 03 00 9c  00 02 00 84 2b 42 6f 72  |HDR3........+Bor|
    00000020  6e 3a 20 30 39 2f 30 36  2f 31 38 00 00 00 00 00  |n: 09/06/18.....|
    00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    *
    00000090  00 00 00 00 00 00 00 00  00 00 00 00 27 05 19 56  |............'..V|
    000000a0  cb 32 6d 9a 59 a5 e1 e2  00 16 af 71 00 00 00 00  |.2m.Y......q....|
    000000b0  00 00 00 00 7d fd bc 9c  05 07 02 01 4c 69 6e 75  |....}.......Linu|
    000000c0  78 2d 32 2e 36 2e 33 32  2e 31 32 2d 34 36 39 2d  |x-2.6.32.12-469-|
    000000d0  69 62 6d 62 61 73 65 32  38 37 5f 73 1f 8b 08 08  |ibmbase287_s....|
    
     
    #16
    fohdeesha likes this.
  17. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    that's a massive version jump - 1.x to 7.x - are you sure this was the bootloader download for g8124 and not the g8124E?

    PowerPC switches generally bottom-boot, so you'll need to look at the end of the image. they're configured to start executing from fffffffc out of reset, and with the way external flash is generally aligned that's the last few bytes of flash. The instruction is almost always 4b ff f0 04, which says "jump to 0xfffff000" - where that actually is depends on where the beginning of flash starts

    Anyway, I have a feeling these newer bootloaders are a little more complicated than "just copy to the end of flash so the end instruction lines up". The "bootloader" they distribute is actually a full copy of u-boot (an actual bootloader), then a full ramdisk linux install, and a linux kernel. my guess is the "bootloader" is actually the entire bootloader and linux OS the switch boots, then the "OS" file you flash is just a binary application linux then runs. Still fixable, but i'll probably have to poke around your flash (and make a dump of it first) over teamviewer.

    You can see that flashing the bootloader file does a lot more than just copy a bootloader directly to flash on these IBM's:

    Code:
    **** VMLINUX ****
    Un-Protected 10 sectors
    Erasing Flash............. done
    Writing to Flash.............done
    Protected 10 sectors
    
    **** RAMDISK ****
    Un-Protected 44 sectors
    Erasing Flash............................................... done
    Writing to Flash...............................................done
    Protected 44 sectors
    
    **** BOOT CODE ****
    Un-Protected 8 sectors
    Erasing Flash........... done
    Writing to Flash...........done
    Protected 8 sectors
     
    #17
  18. Arion

    Arion New Member

    Joined:
    May 11, 2017
    Messages:
    10
    Likes Received:
    2
    Yeah I was kind of surprised that the switch came with a 1.x.x.x and the latest from IBM was 7.x.x.x. Everything from IBM's web site seems to refer to all G8124 switches without differentiation about make/model/etc.:

    IBM RackSwitch G8124/G8124E firmware

    The firmware change history for the most recent release doesn't mention anything about separate releases for different hardware (E vs. non-E types or different revs):

    G8124_7.11.13.0_change_history.txt

    Yeah, I think you are right about this: the "boot" firmware really is the embedded Linux install - it contains the kernel, initial ram disk, and the user space. It looks like there is a binary at /bin/bladeOSLaunch that front-ends the 'OS' executable. The OS executable in the other firmware image is just a great big ELF binary that has all of the switch UI code.

    1.1.1.5 Boot:
    Code:
    # ls -l
    total 82
    drwxr-xr-x. 2 root root  3072 Dec  3  2008 bin
    drwxr-xr-x. 2 root root  3072 Nov 28  2007 dev
    drwxr-xr-x. 5 root root  1024 Oct 15  2008 etc
    drwxr-xr-x. 2 root root  1024 Nov 28  2007 ftp
    drwxrwxrwx. 2 root root  1024 Dec 14  2007 fulcrum
    drwxr-xr-x. 2 root root  1024 Nov 28  2007 home
    drwxr-xr-x. 4 root root  1024 Jun 20  2009 lib
    -rwxr-xr-x. 1 root root 52173 Jun 20  2009 librt-2.3.5.so
    lrwxrwxrwx. 1 root root    14 Nov 28  2007 linuxrc -> ../bin/busybox
    drwxr-xr-x. 2 root root  1024 Nov 28  2007 proc
    drwxr-xr-x. 2 root root  1024 Nov 28  2007 sbin
    drwxr-xr-x. 2 root root  1024 Nov 28  2007 tmp
    drwxr-xr-x. 4 root root  1024 Nov 28  2007 usr
    drwxr-xr-x. 4 root root  1024 Nov 28  2007 var
    # ls -al bin/sh
    lrwxrwxrwx. 1 root root 14 Nov 28  2007 bin/sh -> ../bin/busybox
    # cat etc/inittab  | grep -i launch
    ::once:/bin/agetty -n -l /bin/bladeOSLaunch 9600 ttyS0
    #::once:/bin/bladeOSLaunch
    
    7.11.13.0 Boot:
    Code:
    $ ls -l
    total 60
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 bin
    drwxr-xr-x 4 arion arion 4096 Nov 18 21:53 dev
    drwxr-xr-x 4 arion arion 4096 Nov 18 21:53 etc
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 fulcrum
    drwxr-xr-x 3 arion arion 4096 Nov 18 21:53 home
    lrwxrwxrwx 1 arion arion   11 Nov 18 21:53 init -> bin/busybox
    drwxr-xr-x 3 arion arion 4096 Nov 18 21:53 lib
    lrwxrwxrwx 1 arion arion   11 Nov 18 21:53 linuxrc -> bin/busybox
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 mnt
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 proc
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 root
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 sbin
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 sys
    drwxr-xr-x 3 arion arion 4096 Nov 18 21:53 tmp
    drwxr-xr-x 2 arion arion 4096 Nov 18 21:53 user
    drwxr-xr-x 6 arion arion 4096 Nov 18 21:53 usr
    drwxr-xr-x 9 arion arion 4096 Nov 18 21:53 var
    $ ls -al bin/sh
    lrwxrwxrwx 1 arion arion 7 Nov 18 21:53 bin/sh -> busybox
    $ cat etc/inittab | grep -i launch
    ::once:/sbin/getty -n -L -l /bin/launcher ttyS0 9600
    
    The oldest firmware available from IBM is 6.8.12.0. I poked at it and it looks similar to the layout of the 1.1.1.5 image above (it has an ext2 root file system rather than the CPIO one in the latest release). I didn't poke any further to see when they went to the newer boot/Linux image layout (perhaps the 7.x.x.x series?).

    Anyway, I suppose at the least I could try to get the ancient firmware (1.1.1.5) back on this beast and that would be better than nothing.. and it looks likely I could get to the 6.x.x.x series too.. Perhaps if I could get 6.x.x.x on, an upgrade to 7.x.x.x would then work properly..
     
    #18
  19. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    the nice thing about JTAG is we'll be able to try a bunch of different versions in rapid succession without worrying about bricking it

    I think the most telling thing will be doing a dump of your flash as-is and seeing how it laid down that new boot image, if it did it correctly or if something happened and it wrote incorrectly (or maybe even the older software image doesn't know how to write these newer bootloaders, and wrote the ramdisk etc to the wrong area)

    Ill send you a pm re: shipping
     
    #19
  20. fohdeesha

    fohdeesha Kaini Industries

    Joined:
    Nov 20, 2016
    Messages:
    940
    Likes Received:
    683
    We managed to succesfully revive @Arion 's switch over the weekend via JTAG. Took some figuring out to get the IBM bootloader file split up into the correct 3 pieces (ramdisk, kernel, and uboot). There also seems to be some fpga, cpld, or other component on the board that does something odd with the CPU out of reset that made it very difficult to halt the processor correctly but after some JTAG config trial and error we eventually got it to the point we could read and write flash

    here's a zip with the working bdi2000 jtag config, the files needing to be flashed, and instructions. You'll need a BDI2000 running PowerPC firmware. I have one I am willing to lend to existing STH members if they'd like to recover these bricked units

    http://fohdeesha.com/data/other/IBM-Recovery.zip

    I did a dump of the bricked flash beforehand (the bootloader section anyway), and it was definitely corrupted. The firmware itself is not an issue, it runs just fine on the older G8124's if you write it to flash correctly. However something in the older firmware's flashing process screws up while writing out u-boot. You can see in these screenshots the end gets mangled with garbage data (and the data does not exist anywhere in the firmware file, so it's not like it's just writing the correct information to the wrong place): IBM G8124 brick
     
    #20
Similar Threads: BNT/IBM G8124
Forum Title Date
Networking IBM Blade G8000 G8124 G8124-E Serial-USB cable pinout Oct 8, 2018
Networking G8124 issues... Jul 5, 2016
Networking Making a IBM G8124 less noisy Nov 16, 2015
Networking IBM Blade RackSwitch G8124 / G8124R Nov 10, 2014
Networking IBM G8124E - 24port 10GbE switches Jul 10, 2014

Share This Page