Turbocharge your Quanta LB6M - Flash To Brocade TurboIron - Big Update!

[fohdeesha]

the "normal" copper ports should absolutely come up at 1gbe full duplex on a cold boot. They're treated the same from brocade to quanta, same PHY and everything.

But the fact your management 2 port is coming up is blowing my mind. And the fact the speed/duplex is completely different person to person

after thorough research, I've come to the conclusion it's ghosts, and I give up. Adding these guys to the github repo, maybe they can make some headway

 

[fohdeesha]

Real talk though, I have enough reports now from here and elsewhere that this isn't just me, so I'm adding a disclaimer to the official site, the OP here, and the reddit thread. Something like: your management port may work fine, it may come up at 100mbit, or it may not come up ever again unless you revert to quanta. I guess the fact probably 50+ people have done this procedure and I've heard not a single person mention it, I guess not many use it often anyway.....

 

[mixmansc]

In all likelihood, once I put this into use, it will rarely every be touched so I'm not terribly worried about it. Only thing that would bother me is if totally lost the out of band management ports.

Next couple of tests....
Warm boot, exited the OS prompt and did a reload. Everything stayed the same, in fact the mgmt 1 port did not disconnect or anything during the reload. After a cold boot comes back up the copper ports are all back at half duplex. I think the mgmt 2 port coming up is just the physical port itself being powered is all. Even though it is not accessible in the Brocade firmware it is still powered up. While you can connect a cable to it and even get a link light (at half duplex no matter what since even the ethernet test does not seem to touch that port with the Brocade firmware) it is not in any way usable.

 

[fohdeesha]

That's the thing though, I have seen a couple chassis where the management ports DO completely stop linking, so I wish I could figure this out.

Also, I tried on multiple chassis to get MGMT 2 to link up at any speed or duplex, and it's completely dead on all of them. On all the chassis I've tested as well, the normal copper ports work perfectly at full duplex.

All of these quirks would make a semblance of sense if they happened uniformly, but what's breaking my brain is a bunch of people with identical chassis with identical software are seeing totally different results. I'm eager to bit-for-bit copy the flash and eeprom from a proper turboiron to a quanta and see if any of this changes, maybe with a lot of luck it'll end up being something wrong or out of place with the flash or perhaps even some bits left over from fastpath.

At least we found a procedure to revert to fastpath, so worst case I'm not responsible for permanantly breaking anyone's switches

 

[TheBloke]

OK I was wrong, I do have Half Duplex. 100mbit HD on both mgmt #1 and mgmt #2! At least according to a destination smart switch - the LB6M still reports it as FD!

So here's the score:

1. I set up my LB4M
2. From the LB6M, I connected mgmt#1, mgmt#2 and copper ports 25, 26, 27, 28 to ports 1 - 6 of the LB4M
3. This is the resulting show port all from the LB4M:

               Admin   Physical   Physical   Link   Link    LACP   Actor
 Intf   Type    Mode    Mode       Status   Status  Trap    Mode   Timeout
------ ------ ------- ---------- ---------- ------ ------- ------ --------
0/1           Enable   Auto       100 Half   Up     Enable Enable long
0/2           Enable   Auto       100 Half   Up     Enable Enable long
0/3           Enable   Auto       1000 Full  Up     Enable Enable long
0/4           Enable   Auto       1000 Full  Up     Enable Enable long
0/5           Enable   Auto       1000 Full  Up     Enable Enable long
0/6           Enable   Auto       1000 Full  Up     Enable Enable long

Meanwhile, this is what show int man 1 says on the LB6M:

switch10g>show int man 1
GigEthernetmgmt1 is up, line protocol is up
  Hardware is GigEthernet, address is 089e.0193.0832 (bia 089e.0193.084e)
  Configured speed auto, actual 100Mbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none

So it's lying to me? The nerve!

After I do the Ethernet Test, mgmt#1 changes to 1000 FD, as seen by the LB4M:

               Admin   Physical   Physical   Link   Link    LACP   Actor
 Intf   Type    Mode    Mode       Status   Status  Trap    Mode   Timeout
------ ------ ------- ---------- ---------- ------ ------- ------ --------
0/1           Enable   Auto       1000 Full  Up     Enable Enable long
0/2           Enable   Auto       100 Half   Up     Enable Enable long

mgmt#2 doesn't change, but not surprising as if the general OS can't see it, clearly t2 can't either.

Good news, though I know not a surprise, is that the copper ports are fine.

 

[fohdeesha]

At least according to a destination smart switch - the LB6M still reports it as FD!

Oh Lord - it just doesn't end does it?

 

[TheBloke]

Oh Lord - it just doesn't end does it?

No, it doesn't..

I just power cycled the LB6M again, and this time mgmt #1 and mgmt #2 show as TEN mbit, HD. Could this be the start of the slide to not working at all?

(Switching) #show port all
               Admin   Physical   Physical   Link   Link    LACP   Actor
 Intf   Type    Mode    Mode       Status   Status  Trap    Mode   Timeout
------ ------ ------- ---------- ---------- ------ ------- ------ --------
0/1           Enable   Auto       10 Half    Up     Enable Enable long
0/2           Enable   Auto       10 Half    Up     Enable Enable long

.. yes, possibly. A moment later it started flapping between DOWN and 10 Half.

Another power cycle, now it's up at 100mbit and so far hasn't dropped to 10 or dropped completely. So far it's stable again, including after another LB6M power cycle.

It definitely never went down completely any time before today. Though it is possible it could have dropped to 10mbit before today - I wouldn't have noticed because I'm sure my dumb Netgear switch would show the same yellow light for both 100 and 10.

I'll keep the two switches up for a bit longer, and try a few more power cycles, see what happens.

One more weirdness - it appears the LEDs for mgmt #1 are the other way around compared to the four copper ports. On the copper ports, the left LED is solid for connectivity, and the right LED flashes for activity. On mgmt#1, it's the right LED that's solid, and I guess the other will be for activity although I haven't see it flash when I ping from the other switch.

However, when I cold boot the LB6M, initially it's mgmt#1's LEFT LED that lights, solid. Ie during the bootloader stage I guess. After it gets to OS, it goes to solid right LED as above.

Hum!

 

[BeardedKiltGuy]

Hey sorry for the late reply. On first finishing the Flashing mine came up at 10M Half Duplex. after running the Ethernet tests it would jump to 1G Full. after cold booting the switch it dropped to 100M Half.

 

[mixmansc]

Yup..... a nice bit of bizarre behavior for sure. Another thing I noted. I was just doing the stuff to set the correct mac address, serial, number, etc. I pulled the top cover since I was curios. Based on the dates on the labels on the power supply 1544 (delta power supply made on April 4 2015) and the 512mb sodimm 2015/09 I can make the assumption this one was at the earliest made in late 2015. Mine is one of the new ones from Unix Depot and does have the Amazon label on the board inside as well. Mainboard also shows that it is Rev C.

On the board info page, it will accept just the letter C as the version number.

I think, if people want to keep these as original as possible it might be a good first step before flashing to Brocade to suggest first going into the diag_mode and document the board info data so it can be rentered exactly as it was if desired.

Since these do run HOT I'm going to pull all the heatsinks and put high quality thermal paste (I have both some AS5 and GC Extreme) on them and put them back on. That should help temps a small bit. Also for sh!ts and giggles I'm going to see if I have a 1gb SODIMM laying around to try in it instead of the 512MB one. Might not make any difference whatsoever but if I have one laying around it probably cannot hurt as long as its the same spec ram.

 

[TheBloke]

@mixmansc I was wondering about the RAM as well - given it's just laptop-style SO-DIMM, DDR 333 in a socket.

Unfortunately I don't have any lying around so I'd have to buy one, and I figured it probably wasn't worth it given it's unlikely to make much or any noticeable difference, and might not even have worked. One thing that did worry me is that the chip I have in mine is single rank (1Rx8), but 1GB chips are all dual-rank. Good chance it would still work, but it gave me enough pause to not bother spending the £5-£10 a chip would cost from eBay.

Of course if you try it first and it works fine, I might do it for the hell of it

PS. Mine, bought in the UK last April, is also Rev C with an Amazon label. My SO-DIMM appears to be dated 2012, and one PSU is 2013-05, so mine could be a couple of years older than yours, but still Rev C.

I haven't actually heard of an LB6M that wasn't Amazon - I had figured that maybe it was never even sold on the open market, but was a special unit for Amazon only? Google searches only seem to show up either used units, or discussions on forums like these. No archived sales pages or the like. So maybe a reference Brocade design that Quanta sold only to Amazon, and Brocade sold to the public as the TurboIron?

 

[mixmansc]

If it helps...... pulled the heatsinks. Some of these have some interesting specs and it would appear some of the limitation are simply the firmware.
The main CPU on the right under the square aluminum heatsink is a Freescale MPC8541E Powerquicc III.
The main fabric processor (under the large square heatsink with the copper plate) is a Broadcom BCM56820.
There are 12 2 port Broadcom BCM8726B PHY which run the 24 10G ports, 3 each under each of the four longer heatsinks.
The 4 copper ports are run by a Broadcom BCM5464.
The 2 management ports by a Broadcom BCM5482.
Up on the top front board which has the console port and front LED's there is an Altera Max II CPLD.
The main flash is a Cypress Spansion 256MB part.

Also of note. There is a 2 pin jumper on the board with a jumper and just beside the 256mb flash chip there is a 16 pin header which I'd assume is for flactory programming and debugging. Could likely spend a little bit of time and find transmit, receive, and a ground and then use serial to fiddle around even deeper in things or with the right tools you could directly read and write to the flash.

 

[TheBloke]

Good info, thanks. Regarding the headers - @fohdeesha has talked about a PowerPC JTAG cable which can be used to recover a bricked device, or apply a flash without needing a bootloader with the right commands, so I imagine that's what you're seeing?

 

[TheBloke]

Oh and let us know if you see any worthwhile drop in temperature after applying your better paste. Are you running with stock fans, or have you/are you going to do a fan mod?

At some point I think I'll put a 12cm fan in the top panel of the case, replacing the stock three fans, so as to reduce noise and get lots of airflow in just the right place. I also have some 16cm fans, but they only run at 1200rpm so I'll probably need to go with a faster 12cm.

 

[mixmansc]

Well.... out of the dozen or so SODIMM's I have laying around only two are PC2 and both of those are 512MB. So no clue if a 1GB stick of ram would work yet. I do have a couple of older printers out in the shop though and I think one of those uses PC2 SODIMM's and might have a 1GB one in it.

On the temps. Thats will be tough to tell. The better thermal pastes take a week ro two to really settle in good and I did not check temps beforehand since I've never had this switch in production. My thinking is, more often than not factory applied paste is noever as good as some of the higher end ones that computer enthusiasts get. I have a shop and a dedicated server rack so noise is not an issue for me so I'll be leaving the fans as they are in it.

 

[TheBloke]

However, when I cold boot the LB6M, initially it's mgmt#1's LEFT LED that lights, solid. Ie during the bootloader stage I guess. After it gets to OS, it goes to solid right LED as above.

To expand on this:

1. From cold boot:
   1. When power is applied, Mgmt#1 left LED comes on solid
   2. About 15 seconds later, around the time the '...' starts on console: left LED goes out, right LED comes on
2. Booted in OS, no cable attached, then insert cable:
   1. Right LED remains on; no change
3. Unplug the cable again
   1. Now the right LED goes out, and left comes on.
4. Plug cable back in
   1. LEDs swap again: left off, right on
   2. Steps 3 & 4 repeat now for each cable unplug/plug - left=no connect, right=connect.
5. Run t2 with cable plugged in:
   1. During this process, the LEDs swap from right to left a total of four times
6. After t2, cable plugged in:
   1. Right is now a dual connectivity and activity light!
   2. Mostly it remains solid, but it blinks occasionally and I've confirmed I can increase the frequency by sending lots of ping packets down the wire.
7. After t2, unplug and replug cable:
   1. As before, on unplug the right LED goes out, and left comes on
   2. After re-plug, the right loses its ability to be an activity light. It goes back to solid-on always, regardless of traffic. (However the port does remain at Gbit/FD.)
8. Another t2 will re-establish right as a dual connectivity/activity light.
9. Predictably, conf t>int man 1>disable & enable will also cause the LEDs to go left and back to right, and lose the activity ability (as well as dropping to 100mbit/HD of course.)

So in general it seems like left LED = no connection; right LED = connection, except in the case where you boot without a cable, in which case the right LED will always be on until the first time a cable is plugged then unplugged, at which point it goes into the left/right cycle.

Before t2, there is never an activity light. t2 makes the right LED do double duty: both connection light and activity light. Unless/until you re-plug the cable, at which point it reverts to connection only.

Finally I should note that earlier today I am sure I saw a situation where I had a cable connected to mgmt #1 and neither light was on. I can't re-create that now, though.

So.. there you go. More data. No idea if it's good for anything

 

[TheBloke]

So it's lying to me? The nerve!

I tried plugging mgmt#1 into another copper port of the 6M:

switch10g#show int man 1
GigEthernetmgmt1 is up, line protocol is up
  Hardware is GigEthernet, address is 089e.0193.0832 (bia 089e.0193.084e)
  Configured speed auto, actual 100Mbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none

switch10g#show interface ethernet 25
GigabitEthernet25 is up, line protocol is up
  Hardware is GigabitEthernet, address is 089e.0193.0832 (bia 089e.0193.084a)
  Configured speed auto, actual 100Mbit, configured duplex fdx, actual hdx
  Configured mdi mode AUTO, actual MDIX

So show int man 1 also appears to differ about the mdi mode - saying none, where the copper port shows MDIX. However, unlike FD/HD, this doesn't change after t2: when it's fixed to 1G/FD, show int man 1 still shows "actual none" where show int eth 25 still shows "actual MDIX".

... except, after I did reload, which preserves the port at 1G, show int eth 25 then showed "actual MDI" - not MDIX. show int man 1 still shows none. I then replugged the cable and it was back at MDIX, including after another reload.

To summarise:

1. show int man 1 always shows mdi mode as "actual none", and duplex as "actual fdx"
2. Connecting mgmt #1 to another copper port shows:
   1. Duplex hdx or fdx depending on if the port is 100mbit or 1G, respectively.
   2. mdi mode as either MDIX or MDI; usually MDIX, but I have seen MDI once, after a reload done when at 1G.

 

[AT S37=0]

When researching the LB6M for this: lb6m-ospf/lb6m.adoc at master · injinj/lb6m-ospf · GitHub

I found these things:

The pluto switch is a derivative of the the same Broadcom silicon:

Mystery Google Device Appears in Small-Town Iowa

This describes the Google network history, quite good:

http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p183.pdf

I believe the dual mgmt ports are a direct result of Google specing for redundant mgmt networks in the early days of their network.

 

[fohdeesha]

The main flash is a Cypress Spansion 256MB part.

should be a Cypress S29GL256P - "show flinfo" in bootloader gives some info. And yup, the 16 pin header is a JTAG connector, to use it you'll need a powerPC capable JTAG unit, like a Freescale Ethernet Tap or similar. I have one on order, should be here in a week or two.

that two pin jumper in my experience is a JTAG enable jumper

The LB6M was originally made en-mass by Quanta for Microsoft, hence the M. A friend was VP @ MS during this time and oversaw the purchase of a ton of them, and also remembers working with the team that wrote the MS specific software that they used to replace fastpath (that we'll never see). This was around 10 years ago at this point though, so at least a few years before google's implementation of the broadcom reference design.

It seems amazon started purchasing the LB6M design from Quanta several years later. I don't think many of the microsoft originated chassis made it to the secondhand market, which would explain why we haven't seen much of them, but a lot of amazon. a LOT of large scale providers used designs based on the BCM56820, the quanta is one of many. I believe dell has a model based on the same broadcom ref design, along with a couple others we looked at flashing that I can't remember offhand. They're all more expensive than the quanta with the same features though, so not much point in pursuing them

 

[mixmansc]

Will be interesting to see what you find once you get your hands on the JTAG. I'm guessing one of yall' have already binwalked and otherwise extracted the firmware to poke around a bit more. I guess the idea will be to look at both firmwares side by side to see where the Quanta firmware has the code for the BCM5482 and where the Brocade has its code for the extras we cannot use and figure if it there will be some viable way to insert the 5482 code into the Brocade firmware. Comparing the pictures of them externally it is immediately obvious that the narrow front upper led boards are different as the brocade one has both serial and rj45 mgmt ports and then only 4 copper ports to the right end. That also indicated that the mainboards themselves are at least slightly different. There is also a 10 pin header on the upper front board too. Yup - that is the exact part number on the flash too. I had to take pictures of all with my camera on macro so I could read them. Seems that once I hit 40 I started going blind and it gets worse each year. lol

I should also note that while I'm very good at poking around I am not a developer and am terrible in Linux... I came from an IT career many years back but in the admin side of things. So in poking around in code I'm usually very good at finding things but terrible at actually doing anything useful with my findings.

 

[fohdeesha]

Sadly the brocade image is not pure unobfuscated linux like fastpath is, so the best you'll get out of it is pure powerPC assembly. On top of that mess, the OS image is actually encrypted (which is why it can only be loaded by the brocade bootloader), so even getting to the unobfuscated assembly would be a nightmare. Not good news I know, but I have a couple other ways in mind to potentially fix this management port mess.

The 10 pin header on the LED daughterboard is the programming/JTAG interface for the CPLD. If I can get verygeeky to lend me his actual brocade TI, I'm going to read the code off the CPLD, and flash it to the quanta's CPLD - that could possibly bring the LED lights back, assuming the hardware is wired to the CPLD the same, and they're just running slightly different code. This is assuming they haven't set the "lock" bit on the CPLD, in which case I won't be able to pull anything off. Can't imagine why they'd do that on a little cpld for doing nothing but flashing LED's though. If that works, CPLD programmers are $7, so potentially something the rest of you could do at home