Turbocharge your Quanta LB6M - Flash To Brocade TurboIron - Big Update!

[fohdeesha]

The ram is worth a try, I've certainly seen weirder things. The wind River ice 2 does support PPC, but sadly these jtag boxes are useless without the accompanying software and board definition/target files, and these are impossible to find unless you pay thousands of dollars (in the case of the ice 2, it seems even the MANUAL isn't public). It took me more than 6 months to find someone on the internet to send me the powerpc firmware files for my bdi2000

(I also just realized you removed this part of your post, so maybe you know this already)

 

[Phinitris]

Well I just replaced the bricked switch with another that I pulled out of a test environment. Guess what happened after pulling power ...
So it's definitely not a RAM issue anymore. Weird that no-one else has this issue. Do you have any idea what's causing this. I flashed exactly as your guide said and I rebooted the switches multiple times with the reload command so the boot loader was alright.

 

[fohdeesha]

It seems the flash is damaged on your switches - Where did you buy them? You might have gotten a bad batch (the flash chip on these is pushing 10 years old, although it's rare for this many to fail at once). You could have also had a power surge or lightning strike nearby that induced enough voltage to damage the flash IC. This definitely isn't caused by the flash guide, which is why nobody else has had the issue. There's no way for a bootloader and software designed for this flash chip and CPU to slowly damage it somehow

Where did you buy them? If you bought them on ebay, which seller? I know one seller was purposefully trying to break the flash with a drill - it's possible the damage wasn't bad enough to notice at first but is now finally failing due to the damage. If you open them up look for the flash chip and see if it has any marks like this - https://i.imgur.com/6VCxXep.jpg

 

[Phinitris]

Hi,
yes I bought them on eBay. The switches are installed in a datacenter rack with UPS so I don't believe it's caused by lightning strike.
Even if the switch chip is damaged why did the quanta boot loader work without issues?

Do you think It can be fixed via JTAG? If you still want to lend me your JTAG debugger, I would highly appreciate it. If you don't need it anymore I would even buy it, but that's totally up to you. Shipping costs will be paid by me plus some little extra. Unfortunately I can't find a BDI to buy online

Edit: Chip seems alright. No visible damage. Imgur

 

[fohdeesha]

because you only had the quanta bootloader on there for how long? If the flash is slowly failing, it would have done the same if you left the quanta bootloader there.

I have received emails of more than 200 confirmed flashed switches (and who knows how many more by people who didn't bother to email me), and not a single one has had boot or flash issues. I have disassembled and sorted through most of the bootloader code, I have looked at the CPU clock and flash select/data lines with an oscilloscope, etc - the PowerPC assembly code that brings up the flash, processor and MMU is literally identical between the brocade bootloader and the Quanta BL, as in bit for bit identical. I can factually tell you 100% this is in no way caused by the flashing procedure. If it was, it would have happened the second you flashed the brocade bootloader and rebooted


Which ebay seller sold them? was it fifakingdom?

 

[fohdeesha]

If the flash is still physically OK then yes it can be fixed by JTAG - I just have to wonder if the flash is in a degraded state, if it would just happen again a few months down the road. Do you have any scheduled jobs or commands on the switch that write to flash? (like a scheduled saving of a config or similar)

I can't sell the JTAG since it took me almost a year to acquire, but if you can PROMISE to return it I can lend you it, but I have no idea how bad shipping would be. It weighs 440 grams, and measures 190 x 110 x 35 mm - you would need to ship it from, then back to, Indianapolis, Indiana, 46280

the power adapter says it will take 120 - 240 volts

 

[Phinitris]

The switches were running before for months/years without an issue and event after flashing to Brocade they worked fine for 1-2 months now. The only think I'm doing is reading the current configuration via SSH (with a cron).

Commands I use:

show running-config
show version
show flash

show running-config shouldn't be the issue as it only reads from RAM I believe. show version/flash may read from flash and wear it out. However I can no see any failures of the command itself in our cron so it did read the version fine while it was still intact. I may disable the version/flash read now. This may lead into the right direction.

I bought the switches from unixpluscom and r2disassemblyllc.

I will definitely return it to you but I'm curious what happened to the switches. I'll contact you via private message regarding shipping.

 

[fohdeesha]

I know unixpluscom is active on this forum and generally very helpful to STH members, If you contact them and explain the situation they may replace them. email jack - jack@unixplus.com

How often is "show flash" ran ? I think this might be a good clue

 

[Phinitris]

I don't even know from which seller I bought it so I don't think it's an option.

It's run once every hour. Guess I need to change it as a precaution.

 

[fohdeesha]

After doing some digging, it looks like both the show version and show flash command both have multiple flash operations they perform, with the important one being accessing the boot sector to grep the version string. In a perfect world, flash reads are infinite, but in reality (especially with 15 year old early-gen flash chips such as these) there can be issues such as corruption or read disturb - Flash memory - Wikipedia - newer flash controllers have logic to prevent this by counting reads per sector and if it approaches 100k without a refresh erase/program, they copy it to a new sector to prevent read disturb. However since this flash IC originally hit the market in 2001, it's missing a whole lot of these features

Since the output of "show version" and "show flash" will never change unless you flash a new firmware image, I think you can safely take them out of all your cron jobs without missing anything. To be safe I would also remove "show run", since nothing will change there either unless you manually make a config change

if this same cronjob was being performed on any other switches that are still alive, I would reflash their bootloaders while they are still alive to overwrite any corruption that may have occurred and is waiting to ruin your day when you next reboot. Just run "copy tftp flash 192.168.1.8 brocadeboot.bin bootrom" from the enable level

 

[Phinitris]

Yes that's true. I have removed it from the script and reflashed the boot loader on the working LB6M's.

 

[melsom]

Hello everyone,

We are experiencing an issue with static routes on the v7 software. Was wondering if anyone has experienced the same or have a workaround.

When issuing the following, the route works just fine and 1.2.3.4 is reachable.
ip route 1.2.3.4 255.255.255.255 3.3.3.3

After a while though, 1.2.3.4 stops responding. Only when issuing a "no ip route" and re-adding it again will it work as expected. Was hoping the v8 software might be better, but when reading these forums I think maybe not. Maybe our only option is to run stock FASTPATH :|

EDIT: I can see now that we actually have flashed this switch with v8, and that may be the root cause of this. Will try to revert to v7 first.

 

[fohdeesha]

EDIT: I can see now that we actually have flashed this switch with v8, and that may be the root cause of this. Will try to revert to v7 first.

I was going to say - the v7 firmware image was in use by thousands of people (including myself) with static routes, so the likelyhood of their being an issue with the switch/software is almost zero.

However if it's that beta v8 image, that makes sense. It was never publicly available, so who knows what issues it could have undiscovered (which is why the guide recommends using v7).

You can flash v7 from inside the booted switch at the enable level with just:

copy tftp flash 192.168.1.8 v7-layer3.bin primary
reload

EDIT: if this does indeed turn out to be another issue with the v8 firmware, I'm just going to remove it from the guide and site completely. with the LACP bug and now even a bug with simple routing, I don't see why anyone would want/need to use it

 

[melsom]

EDIT: if this does indeed turn out to be another issue with the v8 firmware, I'm just going to remove it from the guide and site completely. with the LACP bug and now even a bug with simple routing, I don't see why anyone would want/need to use it

I can confirm that the bug is now gone with the v7 firmware and that v8 does indeed have issues with simple static routing.

 

[fohdeesha]

thanks for testing! I'm going to completely remove the v8 image and info from the site in that case

 

[Corsaire]

Oh wow, a buggy firmware.
I did the procedure to go back to v7 too.

I am very interested in the kind of troubles with LACP.
I had a hard time to make it run at first as I explained in this thread, plus I have always had the sensation that my 4-way LACP on the ethernet ports did not gave the performance expected.
This would explain a lot.

 

[Corsaire]

I have another headache matter to report, not sure who to blame here, so an expert overview would be appreciated.

I made a trunk link with the 4 copper ports of my brocade modded quanta linked to 4 ports of a netgear gs724tv4.
the ports are configured LACP active, the link goes up and everyone is happy... except for one thing.

If I made a 4 ports LACP trunk between the 2 switchs, it is to use the bandwith aggregation, as intended in 802.3ad.
But, I only get at max 1GB/s, the max speed of one of the 4 links.
Interestingly however I can run in parallel two 1GB/s transfers to different machines through the trunked network.

This goes in par with the following piece I found on the web:

while the total bandwidth of a 2 port LACP link is 2Gbps, an individual flow will max out at 1Gps.  
It's a highway with a 100MPH speed limit.  
We can add another lane and 2 cars can go 100MPH but a single car can not go 200MPH.  
We're increasing the amount of traffic that can be carried, but not the max speed.

I also found out that it's doesn't have to be always the case.
the LAG can base its actions on a MAC-hash or a IP-hash, IP is L3 and can help on this field (at least that's what I could understand).
Is there a way to force the hash mode on a LAG on the quanta/brocade ?

SMB protocol in Windows and synology products seems to try and make multiple sessions to use all the channels on a LAG when it is detected too.

 

[Phinitris]

That is expected behavior. A single connection will always use the same physical switch port. Normally the hash function is set to include L2 and L3/L4 information (MAC, IP, Port). You should be able to start multiple transfers in parallel between the same servers but will never be able to get more than 1 Gbit/s in a single thread/process.

Edit: Not sure if the LB6M can do L4 hashing.

 

[epicurean]

Hi, I need help resetting the static ip address I assign to the LB6M because it currently clashes with an esxi server.
Can I set it such that it takes it from the DHCP server?
No familiar with brocade cli commands

 

[fohdeesha]

the normal ports in vlan 1 with VE 1 or the dedicated management port? I believe by default if you don't assign an IP (remove all IP configurations from any VE or management interfaces) it will default to taking DHCP leases