Flashing Delta 7024 to Dell 8024

[fohdeesha]

Finally found someone willing to test the method I came up with, and it works great (nearly the same process as the LB6M flash, with some differences due to a larger bootloader from dell etc)

Unlike the LB6M > Brocade flash where there were some slight hardware differences that caused things like the SFP+ lights to stop working, the Dell and Delta are absolutely identical. This means there's no changed behavior, all the lights work, etc.

Unlike brocade, Dell does not ship the bootloader in a separate file, so the most difficult part was hunting it down in the 30MB stk image and ensuring the trimmed bootloader code had the correct geometry

The guide is available here: (7024) Delta To Dell - Brokeaids

For people who previously bought already-converted Delta 7024's off ebay (eg they came preloaded with dell firmware) - I personally confirmed you can run the cpld update command to flash the CPLD with the proper dell code. This will fix temp/fan status reporting, and should theoretically allow the fans to spin down a little quieter. Just update to the latest dell firmware, then update the bootcode from the boot menu. then when it's all booted and updated, just run "dev cpldUpdate" at the enable level. Check the CPLD section in the guide for the output you should expect. You'll need to unplug the switch fully for a few seconds and repower it to reset the cpld and get it running the newly flashed code

 

[i6feetunder]

#wow, very impressive and nice documentation.

 

[javooooo]

Thanks for the tutorial @fohdeesha!

Everything went well for me, up until the CPLD update. I get this error:

console#dev cpldUpdate

Unrecognized device
CPLD update exited with a return code of 0
error_address 180595718, exit_code 6, format_version 2
value = -1 = 0xffffffff

Has anyone had this issue before? I've tried cold booting the switch, reinstalling the update again via TFTP from the Dell firmware. The rest of the process completed without any issues and it seems to be running the correct Dell bootloader and firmware.

 

[fohdeesha]

Hello! This actually seems to be a dell issue - if you search around, you'll find many legitimate Dell 8024 owners with this issue, and no fix from dell. (I'm guessing you know this, as I see now you've bumped the dell thread). First time I've personally seen this issue through, after 4 or 5 Delta crossflashes.

If I had to take a wild guess, their CPLD flashing code is lazy, and when it sees a very old Altera CPLD revision it just gives up. It doesn't seem to be a huge issue, and all your CPLD related functions seems to be fine (it's reporting all the fans and psu's etc correctly)

if you really want to mess with it, you can try flashing a really old FW image, which probably has a different updater that is maybe more forgiving of an altera device that out of date. That might be able to get it bumped up enough for the latest FW to bump it to the latest. (now that you have the proper Dell OS flashed, you can easily load in new firmware images using TFTP)

like this one from 2014 for example: PowerConnect 8024/8024F, Firmware Version 5.1.3.7,A19 | Dell Canada

(for all the old versions to try, go here - Support for PowerConnect 8024F | Drivers & downloads | Dell US then click the dropdown arrow by the download name, then click "older versions")

Something else I've seen that might help is unplug it, plug it in to do a cold boot, wait at least 10 minutes for it to finish initializing vxworks, THEN try the update command. If you had any links or cables connected when you tried to update, UNPLUG them. Also only run the command over a serial connection

 

[fohdeesha]

In the secret boot menu mentioned in the guide, there's also a CPLD specific option that might have some low level commands or options to force a flash to the CPLD, but I don't have one to look at or test. If you go into the secret boot menu, hit the option for CPLD diagnostics, take a screenshot of all the options that come up and post it here

there's also a raw VXworks shell in that menu that could also be promising, but once again I'd have to look at what all is available

 

[javooooo]

Thanks for the quick response! Yeah, it's an odd issue.

Option 9 in the hidden menu just spits similar output to "dev cpldTest" in the OS:

Select option: 9
Board Type                : Campbell 24F
Board Version             : 2
CPLD Revision             : 3 (0x3)
RPS2 Present
RPS2 Status               : Failed
RPS2 Temperature          : Normal
RPS2 Fan Status           : Working
RPS1 Present
RPS1 Status               : Working
RPS1 Temperature          : Normal
RPS1 Fan Status           : Working
Port   SFP Status  Link Status  Tx Power
----   ----------  -----------  --------
   1  Not Present         Down  Disabled
   2  Not Present         Down  Disabled
   3  Not Present         Down  Disabled
   4  Not Present         Down  Disabled
   5  Not Present         Down  Disabled
   6  Not Present         Down  Disabled
   7  Not Present         Down  Disabled
   8  Not Present         Down  Disabled
   9  Not Present         Down  Disabled
  10  Not Present         Down  Disabled
  11  Not Present         Down  Disabled
  12  Not Present         Down  Disabled
  13  Not Present         Down  Disabled
  14  Not Present         Down  Disabled
  15  Not Present         Down  Disabled
  16  Not Present         Down  Disabled
  17  Not Present         Down  Disabled
  18  Not Present         Down  Disabled
  19  Not Present         Down  Disabled
  20  Not Present         Down  Disabled
  21  Not Present         Down  Disabled
  22  Not Present         Down  Disabled
  23  Not Present         Down  Disabled
  24  Not Present         Down  Disabled
Fan1 Status               : Working
Fan2 Status               : Working
Fan3 Status               : Working

As for the VXWorks option, I've attached a screenshot.

I also tried the devshell option to execute the cpldUpdate command directly but that got a weird error too.

Boot code utilities menu
Options are:
1  - Read/Write memory
2  - Display PCI bus
3  - Display PCI bus details
4  - Display core info and bus speeds
5  - Display file system details
6  - RAM test
7  - File system directory listing
8  - Generate and store VPD info
9  - CPLD diagnostics
10 - Switch diagnostics
11 - Format file system
12 - File system test
13 - Comprehensive test (RAM, PCI, FLASH)
14 - Start vxWorks shell
15 - Devshell command interface
16 - Upload a file through XMODEM
17 - Download a file through XMODEM
18 - Run flash diagnostics
19 - Display Bootcode Build Information
20 - List Main Menu Options
0  - Return to main menu
Select option: 15

devshell> cpldUpdate

executing 'cpldUpdate'

No CPLD available!

value = 19 = 0x13

I tried an old FW version earlier and it didn't work either, but I'll go back to an even earlier version and try again.

 

[fohdeesha]

hmm. The only other thing I can think of is the scorched earth method: you can use a USB Blaster connected to the CPLD programming/JTAG header and just use Alteras devtool to dump the proper CPLD image right to the damn thing directly, but you'd need the CPLD image to dump.

The chances of me succesfully digging that out of the Dell STK image are basically zero (I'm not a CPLD/FPGA guy so I have no idea what I'd even be looking for). So you'd need to obtain the proper image by connecting the USB Blaster tool to a dell 8024 that has had it's CPLD succesfully updated, and dumping it's contents with Alteras tool.

All that said, I probably wouldn't worry about it unless you start getting odd behavior from your switch (which doesn't seem to be happening).

 

[javooooo]

Hmm... I've just tried versions from the 5.1 and 5.0 releases again and neither worked. I'm tempted to try 4.2 although I'm concerned it may install an older version than what's already on there.

I do have another Dell 8024 that updated successfully so I will research the USB Blaster you mentioned further - although all of this is new to me so it sounds a bit too complex. I am just a bit concerned about the comments in the thread on Dell's site about the switches being unstable with the old CPLD version.

 

[fohdeesha]

Well since you already have a working cpld on hand, I don't think the USB blaster would be terribly difficult. I can help walk you through it, the altera program is a nice gui with dump and write type buttons, and a USB blaster clone is only about $15.

As for the reports of odd behavior, I wouldn't give them any credence. I saw reports of links no longer working at 1gbe and odd multicast behavior, but the cpld has nothing to do with any of that. It's just a glorified low speed I/o multiplexer that runs the status lights, monitors fans and psus, etc. Strange cpld behavior would be psus not showing up, temps not being reported, link lights not working (ran into this personally with the quanta to brocade flash) etc

First step to know if it's possible is take a high res (do not resize) picture of the inside of the delta (just take the top cover off). That'll tell me if it does indeed have a 10pin cpld header.

If it does, it's theoretically as simple as connect the header on the working switch, run alteras devtool, click dump program and save it as a bin file somewhere.

Then same process on broken switch, but click write program instead. If it works, I can host that bin file on the site/guide and anyone else with an un-updateable cpld and an extra $15 can fix theirs as well

 

[javooooo]

Ah, gotcha. I may wait a bit for my optics to arrive so I can check the link lights - if they work, there's probably no point in updating the CPLD. The fans are spinning down nicely and the ethernet lights work, so fingers crossed.

The other unit I have is also a Delta that I converted following your guide, but the CPLD update did work on that one. I'll crack it open and take some photos next week once the optics arrive. I'll be converting a few more in the next few weeks so hopefully those have the newer revision chip.

Cheers.

 

[fohdeesha]

So I ended up going down a CPLD hole and reading a lot of Altera whitepapers last night, and I have a rough idea of what's going on.

The Altera CPLD's have two small flash areas:

UFM = User Flash Memory - a small 8096 bit area for storing whatever you like, stuff like serials, metadata, vendor tags, CPLD software revision tags, etc. The CPLD itself doesn't actually use anything in here to run, but the host system can access it.

CFM = Configuration Flash Memory - this is the flash that actually stores the compiled CPLD code that runs the thing.

What seems to be going on, is the Dell CPLD flash routine first reads the UFM and looks for a known vendor string and version string, so it knows it has the correct device address and isn't about to flash some other part or unrelated CPLD. Some of the older Deltas and Dell chassis seem to have a different vendor and CPLD ident string written in here, and when the Dell update procedure sees them and doesn't recognize them, it quits the update process out of caution.

Alteras free programming-only software (Quartus Prime Programmer) has nice simple "Read" and "Write" buttons that will read and write both flash areas to and from a file on your PC. So it should only be a 5 minute or less process to connect the USB Blaster to the board, click dump to a file on your desktop. That file will have a UFM with the correct vendor strings and tags that won't freak the dell out. Then just connect it to the "broken" switch and write.

I ordered one to play with, I didn't realize how cheap the clones are - altera Mini Usb Blaster Cable For CPLD FPGA NIOS JTAG Altera Programmer | eBay

I don't have any 8024's but I have a lot of switches with the same Altera MAX II CPLD (quanta lb6m, brocade icx6610, etc), so I'll play around with reading and writing.

If you pick up a dongle and read the flash off the known-correct CPLD and send me the file it'd be a huge help. Can then write up a short little guide that anyone in that broken dell thread can use, just need to ebay or amazon a $7 dongle. I'll paypal you $7 if you need it for your troubles. More than anything I'm just curious what strings they actually have in the UFM

 

[fohdeesha]

So I ended up buying one of the $5 ebay USB Blaster clones off ebay, and downloading the free Quartus Prime programming software. When you plug in the clone dongle, it shows up as as a USB Blaster device with no found driver - so I manually installed the driver, choosing the driver inf files that ship with the Quartus programming software. This ended up causing BSOD's - which turned out to be a known issue with Altera and this dongle, even the genuine ones. They recommend an alternative driver which took me a while to find, but tracked down: http://fohdeesha.com/data/other/blaster-driver.zip

Once I used that driver, everything else turned out to be incredibly easy - just connect the dongle to the 10 pin header, launch the software, and hit auto detect. It'll scan the JTAG chain and automatically figure out what CPLD model you have and apply the correct settings. Then you can just click "Examine" and dump the contents to a file. if you want to write that file back, you just click Program

 

[N1h1l1ty]

Welp, I bid on this guy Delta ET-DT7024 Running DELL 8024F Firmware | eBay

1 prior transaction on fleabay. Either the beginning of a great (or terrible) journey

 

[kkolasa]

Is there any chance that you can export the full programming file from a working switch? I am trying to recover a bricked switch for a friend using a USB blaster and Quantas software (commercial version).

All I have for firmwares are the Dell .stk format for the flash and the raw bootloader in binary, neither of which I can upload directly using Altera. I am not so familiar with making a .pof programming file from scratch, so I was wondering if you could read your working switch's firmware and do a direct dump in .pof programmer format. That way I can load the correct code and simply move over my user data. I just can't seem to get the firmware to inject the hex in the right format and area of memory (it keeps injecting HEX or binary in an incompatible format, maybe 16bit or 8bit??).

Any help is appreciated.

Cheers,
Kevin

 

[fohdeesha]

bad news: the usb blaster and software you're using are for programming the Altera CPLD, all this part does is control the LED lights and chassis fans etc, it has nothing to do with the actual switch software or booting the switch. To program the bootloader and unbrick it, you need a JTAG unit that can talk to the PowerPC CPU and write to its flash, like a BDI2000 (not super easy to find). If you do find one, you'd just write the bootloader to the bootloader section of flash:

erase 0xfff00000 0x20000 8
prog 0xfff00000 8024boot.bin bin

 

[kkolasa]

Thanks for the quick reply. I do have a JTAG system here at work, as we have a full board design and manufacturing unit that I am working with now. I will see if I can JTAG into the main PowerPC chip. Does the IC have internal memory or does it address externally ? ie: is the bootloader stored inside the IC itself, or is it loaded on external flash (I will check whitepapers later, but if you know off the top of your head that would speed things up)? The JTAG chain can see other devices, but without knowing how to configure them, I will need to build a lot of information up to work with it. But I am down for the challenge, I will see what can work. I have half a dozen different JTAG compliant write/snooping/reading devices, but I may need to make some special cables to interface with the hardware I have here. There is another pin header which I believe is the PPC link. I will let you know what progress I make.

Thanks again,
Kevin

bad news: the usb blaster and software you're using are for programming the Altera CPLD, all this part does is control the LED lights and chassis fans etc, it has nothing to do with the actual switch software or booting the switch. To program the bootloader and unbrick it, you need a JTAG unit that can talk to the PowerPC CPU and write to its flash, like a BDI2000 (not super easy to find). If you do find one, you'd just write the bootloader to the bootloader section of flash:

erase 0xfff00000 0x20000 8
prog 0xfff00000 8024boot.bin bin

 

[kkolasa]

I forgot to add, the PPC chip does have both internal and external memory, I just am wondering if the bootloader and OS are on the external flash, or if the bootloader is internal and the OS external? I am thinking that it is the case.

Cheers,
Kevin

Thanks for the quick reply. I do have a JTAG system here at work, as we have a full board design and manufacturing unit that I am working with now. I will see if I can JTAG into the main PowerPC chip. Does the IC have internal memory or does it address externally ? ie: is the bootloader stored inside the IC itself, or is it loaded on external flash (I will check whitepapers later, but if you know off the top of your head that would speed things up)? The JTAG chain can see other devices, but without knowing how to configure them, I will need to build a lot of information up to work with it. But I am down for the challenge, I will see what can work. I have half a dozen different JTAG compliant write/snooping/reading devices, but I may need to make some special cables to interface with the hardware I have here. There is another pin header which I believe is the PPC link. I will let you know what progress I make.

Thanks again,
Kevin

 

[kkolasa]

I found a nice guide for using an open source JTAG programmer and a hardware/cable hack to make use of my USB blaster. The link for the guide is: Using JTAG on MPC824x boards . I have already made the cable.

"URJTAG" might have the features needed to do the flash. You can find it at UrJTAG - Universal JTAG library, server and tools. I will keep you posted.

 

[kkolasa]

Update: I was able to get urjtag to interface with my USB blaster. Now I am on the hunt for the BSDL file for the processor.

 

[fohdeesha]

Urjtag will work with the much older mpc82xx series, however the Dell is an mpc85xx (e500 core)