Turbocharge your Quanta LB6M - Flash To Brocade TurboIron - Big Update!

[Phinitris]

Hello,
one more successful flash
Unfortunately I encountered the same issue as @TheBloke. After trying to configure a LACP between the Turboiron (LB6M) and a stock FASTPATH LB6M it crashed and now idling for 2 days in the datacenter Seems like it doesnt have a watchdog timer @AT S37=0 .
I have seen a lot of log messages that the port was removed and added to the port channel every 2 seconds. As far as I know the Turboiron rotates the log every 100 messages so why should it trigger a crash? I can not recommend using this in a datacenter yet if it crashes and needs to be hard rebooted. Luckily I only use it for testing right now

@fohdeesha is this caused by a log overrun (which would happen always at some point) or only if many messages are logged in a short time?

 

[fohdeesha]

It's the rapid lacp flapping that causes the crash, not logs. I have fastiron devices with more than 500 days of uptime and TONS of port enables/disables

This is why I recommend using the port flap dampening command in lacp groups, regardless of what OS you're running. It will prevent this. You probably also need to configure the lacp timeouts to match to stop the flapping in the first place

 

[Phinitris]

Thanks. So a normal port flap will not cause this, only if the port in an lacp group flaps? Port dampening should prevent this right?

 

[fohdeesha]

Yes to both

See Configuring port flap dampening on a trunk EDIT - this is for trunks, not LACP. for LACP, just configure link-err-disable on every member port

Was port 1 in your lacp by chance? I'm going through the brocade release notes to see if this was ever fixed, there's the v7 codetrain that has less features but seemingly more bugfixes (last v7 release for the turboiron was 7.4 in April of 2016, versus 2015 for the v8 image in this guide)

I'll post a link to the v7 image if you want to try it and make it flap, to see if it also locks up

 

[Phinitris]

Thanks. The trunked ports were 5 and 6.

 

[TheBloke]

As usual if you find any typos or have suggestions let me know. Beyond that I think I'm officially done with this project and I'm glad so many people got use out of it, it's been fun working with so many people.

Thanks so much for all your hard work on this project. It's greatly appreciated.

I did figure out how to do the same procedure on a Delta 7024 to turn it into a Dell 8024, but I still haven't found someone willing to try it

Maybe there just aren't that many users out there - this thread is now the first hit on Google for "delta 7024" And most of the rest are about Delta airlines. I do see a single Delta 7024 on UK eBay, but just one, and at more than twice the price I paid for the LB6M. Though still significantly cheaper than the 8024, so definitely would worth doing for someone who had one.

I wonder if someone could make some money buying a 7024 then flashing to 8024 and re-selling Though you'd have to find a quite understanding buyer, willing to buy a switch with Delta branding but promised to operate as a Dell.

The Web UI on the 8024 looks quite nice. I usually prefer CLI wherever possible, but sometimes it's nice to also have a GUI that can result in CLI commands to learn from.

 

[Corsaire]

Yes to both

See Configuring port flap dampening on a trunk

Was port 1 in your lacp by chance? I'm going through the brocade release notes to see if this was ever fixed, there's the v7 codetrain that has less features but seemingly more bugfixes (last v7 release for the turboiron was 7.4 in April of 2016, versus 2015 for the v8 image in this guide)

I'll post a link to the v7 image if you want to try it and make it flap, to see if it also locks up

Oh wait, here is something interesting.
When I made a LACP link to trunk my Netgear switch with the 4 copper RJ45 ports I had an episode of bad flapping too.

I workaround it with the following, and set "short" on the Netgear too.

interface ethernet 25 ethernet 26 ethernet 27 ethernet 28
link-aggregate conf key 10000
link-aggregate conf timeout short
link-aggregate active

show link-aggregate

Could this dampening commands mitigate the flapping with out-of-the-box settings ?

 

[WeekendWarrior]

Hope that works out with the Dell firmware! That would make these right up up there with current enterprise class switch offerings! Would also make it pair up even better with the H3C S5800 that I just got and converted to the latest HP A5800 software. If you are looking for a top shelf 48 port copper gigabit switch with 4 10Gbe SFP+ ports, find one. ... Edit to add, if anyone is interested I'll document all the steps and make an easy guide like we now have for the LB6M. Will make things a lot faster and easier if the steps were actually documented with some quick config tips and whatnot. Only gotcha is if some prior owner actually configured a bootloader password. No need to really but if that is set and not known then its pretty much over before you start. Any other passwords for the CLI or anything else can be removed easily as long as you can get into the bootloader on one.

I would really appreciate such a write-up for the S5800.

 

[fohdeesha]

Would also make it pair up even better with the H3C S5800 that I just got and converted to the latest HP A5800 software.

Not sure how I missed this. If you can roughly match the format of the LB6M guide website I can throw your guide up on there as well (with credits to you of course). Check the github repo for the markdown source if you want a template to work off of. StackEdit is nice site if you want a simple way to write up markdown and see a preview as you work

Do you happen to know what management CPU the HP/H3C uses? It looks like there's a slim chance it could match the LB4M, and new firmware for that would make a lot of people happy. Did you need to extract the bootloader out of the HP firmware BIN? I see a u-boot header, but no ending load instruction at the end of the u-boot section, at least not one that matches the jump at the end of our u-boot. I guess it could just be jumping to a different memory address

 

[fohdeesha]

Also, it looks like that same HP firmware BIN supports the 5820X-24xg - a 24 port 10gbe SFP+ switch with 4 copper ports. If they're using the same management CPU and ASIC, that would give us new firmware (with ipv6 routing) for the LB6M as well. Judging by the newer age though I have a feeling the chances are very slim. It also seems HP is likely using their own asic. Looks like I have a new project to try tonight, I'm probably just going to straight dump that entire BIN file to flash via JTAG

 

[fohdeesha]

Could this dampening commands mitigate the flapping with out-of-the-box settings ?

the dampening feature doesn't solve the core problem that's making it flap, it just disables a port altogether (or for a specified amount of time) if it sees it go up and down a certain number of times. This prevents the flapping from carrying on forever and filling up your logs (and crashing in some cases too apparently). Still a very useful feature to enable. check this short guide, for a LACP group, just run these commands on every port member -

After some digging it seems I may have realized why some devices flap in the default configuration. During the LACP negotiation the devices are "supposed" to share timeout values and configure each other to use the correct values (long or short) automatically. When this doesn't happen they use an assumed default, and if it doesn't match the other ends assumed default, it'll flap.

Configuring LACP to some xenserver machines and some other brocade devices with no flapping on default configuration, the output of show link-aggregate indicates the switch has received preferred timing values and is using them instead of default timeouts. The "default" column: https://i.imgur.com/ZrCR62q.png

according to the brocade manual, "No" in the Def column means: "No – The port has received link aggregation information from the port at the other end of the link and is using the settings negotiated with that port."

If you LACP to another device and get flapping, see if your show link-agg output has a "Def" in that column instead of No. This indicates it hasn't received timing values, and is using it's defaults. In the Brocades case that's "long", and the device on the other end has probably defaulted to "short". That's why adding the timeout short command has fixed this for most, because then the timeout values match on both ends. I would think most switches would negotiate timeout values properly, I haven't found one here to link with that didn't - but it seems some devices do not bother sending these negotiation packets. I will dig around and try to find a device that flaps, as I could be totally wrong here

 

[mixmansc]

I had almost forgot I did do a write up in a Word doc. I put it up on Google docs. This link exists here only.
H3C to HP Switch Conversion

WARNING - the above instructions are NOT for the LB6M. Just want to be clear in case someone new to this thread pops in, sees this, and attempts to follow this for their LB6M. LOL

I have not had time since I wrote it to put the switch into service or anything and have not gone back through to double check that everything I did was correct in the document but it should be. I did afterwards configure an IP for the switch so I can access the GUI in a browser but apparently I did not document those steps...

I did have the top off - I'll check and see if I took any pics with the top off. If not I easily can pull the top and ID the the components and get a couple of good pics.

Jon - if you want to put this into a doc on the Brokeaid pages please feel free to do so.

Edit to add - Looked under the hood of the H3C. The main controllers and individual controllers - HP put the heatsinks on with adhesive instead of clamped so looking at those is out. The management processor is a Lattice MachXO LCMXO1200C. It does have two ten pin headers beside it. I'd assume at least one of those is a serial JTAG connection. Looks like eight 6 port processors, plus the 3 big processors. My guess on the three large processors is one is the fabric and the other 2 are two port 10g processors. I'll connect a console cable up in a few and query some things to see what other info it gives me concerning the hardware but it's probably a moot point.

Not sure how I missed this. If you can roughly match the format of the LB6M guide website I can throw your guide up on there as well (with credits to you of course). Check the github repo for the markdown source if you want a template to work off of. StackEdit is nice site if you want a simple way to write up markdown and see a preview as you work

Do you happen to know what management CPU the HP/H3C uses? It looks like there's a slim chance it could match the LB4M, and new firmware for that would make a lot of people happy. Did you need to extract the bootloader out of the HP firmware BIN? I see a u-boot header, but no ending load instruction at the end of the u-boot section, at least not one that matches the jump at the end of our u-boot. I guess it could just be jumping to a different memory address

 

[WeekendWarrior]

I had almost forgot I did do a write up in a Word doc. I put it up on Google docs. This link exists here only.
H3C to HP Switch ConversionView attachment 7971

Thank you very much for the write up!

 

[fohdeesha]

Nice! The Lattice is an FPGA so that wouldn't be the CPU, but just the presence of that alone and looking at the PHY layout I can tell there's no way the firmware would be applicable to our platform. well damn, maybe one day we'll find something else

also once I find some free time I'll try to get that doc layed out in guide form on the site

 

[mixmansc]

Happy to help Weekend Warrior.

Ah - so the processor must be the rearmost large chip then. The other two large ones must be the fabric and/or 10gbe chips. Did not bother looking up the spec on the Lattice chip - that is apparently just an I/O interface. Regardless not much point in digging deeper. Easy enough to cross-flash the H3C to HP firmware and HP still regularly updates them and makes the firmware publicly available. Other have cross-flashed them fine but the specific commands, order of things, what to check, and where to download things was just scattered all around and not easy to figure out. So to an extent I documented the steps. Somewhat nice that they even include a "brand" command specifically for converting one to a different brand firmware. As I mentioned in the doc I made though, my instructions are specific for the model I bought but the same process is applicable to a good number of model H3C switches.

 

[Phinitris]

@fohdeesha I have tried port dampening now (link-error-disable), however it seems that it doesn't work for LACP flaps.
The port goes from UP to DOWN all the time, however the link is not disabled. If I log into the other switch and shut the port manually down and re-enable it, port dampening does work and the port is disabled.

Testing V7 Firmware right now.

 

[Phinitris]

Well, I have flashed the V7 image now and the LACP channel doesn't flap anymore. I could not get it to flap. So now I have a V7 image in my primary flash and a V8 image in the secondary flash. Booting to the secondary (V8) flash with the same configuration makes the ports flap again and crash. Booting the primary (V7) image just works fine.

I have the following versions:

  SW: Version 07.3.00sT203 Copyright (c) 1996-2013 Brocade Communications Systems, Inc.
      Compiled on Sep 29 2016 at 23:42:25 labeled as TIR07300s
      (6643818 bytes) from Primary TIR07300s.bin
      Compressed Boot-Monitor Image size = 369366, Version:07.3.00T205 (trz07300)
  HW: Stackable TurboIron-X24
==========================================================================
      Serial  #: 00000000000
      P-ASIC  0: type B820, rev 11  subrev 00
==========================================================================
  825 MHz Power PC processor MPC8541E (version 8020/0020) 330 MHz bus
  512 KB boot flash memory
31744 KB code flash memory
  512 MB DRAM
The system uptime is 1 minutes 36 seconds
The system started at 19:15:19 GMT+00 Wed Feb 28 2018

The system : started=warm start   reloaded=by "reload"

Compressed Pri Code size = 6643818, Version:07.3.00T203 (TIR07300s.bin)
Compressed Sec Code size = 6845067, Version:08.0.01T203 (TIR08001e.bin)
Compressed Boot-Monitor Image size = 369366, Version:07.3.00T205 (trz07300)
Code Flash Free Space = 18219008

So I believe the V7 image is better for datacenter use and the V8 image only has some L3 features which I don't need anyway

 

[fohdeesha]

Well, I have flashed the V7 image now and the LACP channel doesn't flap anymore. I could not get it to flap. So now I have a V7 image in my primary flash and a V8 image in the secondary flash. Booting to the secondary (V8) flash with the same configuration makes the ports flap again and crash. Booting the primary (V7) image just works fine.

I have a feeling they just changed the fallback defaults for LACP from "short" in v7 to "long" in v8. Since your other end gear defaulted to "short", the v8 image default of "long" didn't match and caused flapping, but the "short" default in v7 matched. I don't think there's a right or wrong choice, the "short" default in v7 that works for your switches will break the other 50% of switches that default to long - this is why it's nice when gear auto negotiates timings like it should so it doesn't have to use default guesses in the first place

Can you do me a big favor and pastebin the output of "show link-aggregate" on v7? that would confirm this behavior and allow me to put a note in the guide.

So I don't think that's a bug, what *is* a bug is the complete OS crash with LACP flapping, and I'm not sure if that's fixed in v7 or not. To make your v7 image flap like v8 did, you should only need to set the timing for the LACP to "long" like it defaults to in v8, and see if it crashes

There's also a newer v7 image 07.4.00p, I'll add it to the zip (with release notes) when I have a couple spare min. The L2/L3 features should be the same between all of them, v8 just added some easier to configure command structures and things like that I believe

 

[fohdeesha]

added the latest v7 images to the zip and added an image selection blurb to the guide: Brokeaids

remember when I said I was finished with this? good times

 

[Phinitris]

Long  timeout: 120, default: 120
Short timeout: 3, default: 3
Port  [Sys P] [Port P] [  Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
5          1        1    10000   Yes   S   Agg  Syn  Col  Dis  No   No   Ope
6          1        1    10000   Yes   S   Agg  Syn  Col  Dis  No   No   Ope

It's using short but only after I configured it to do so. The V7 image was first using the long timeout (L) but then I set it manually to short with link-aggregate configure timeout short. So the LACP does work with BOTH timeouts.

I also noticed that I can get the V8 LACP to work if I only enable one interface. Once the LACP connection is established I can enable the second interface and it links up too and works fine. However when I enable both links at the same time it flaps. V7 image works fine even if I enable both links at the same time.