Turbocharge your Quanta LB6M - Flash To Brocade TurboIron - Big Update!

[fohdeesha]

well shit, that's actually what @verygeekys started doing before it stopped linking completely. Can you unplug the ethernet cable and plug it back in and see if it links back up at 1gbe? if it doesn't, do this:

while connected over serial (won't work via telnet/ssh), in the OS, hit ctrl + y then let go then press m. then press enter, should drop you to OS prompt. run "t2", select ethernet test, then hit i think 6 for "run all ethernet tests" when it finishes, escape out of all the test menus until your back at the OS prompt, then hit ctrl + z to return back to the normal OS prompt. Your port should now be at 1gbe. REALLY need to figure out wtf is causing this

 

[TheBloke]

OK all done. It remained 100mbit despite multiple cable re-plugs, then I did the Ethernet test, after which it did indeed come up as 1gbit.

I will keep an eye on it and let you know if anything else changes. For me it really doesn't matter even if the management port did die, it's only a home network and I'm fine using in-band, or even serial if need be. But I'd be glad to test anything that might help you resolve it for others.

Thanks again for all your hard work on this. It's a fantastic project!

 

[fohdeesha]

Thanks for testing. It looks like you have the same symptom, which is bad news considering you followed the proper flash routine. I don't want to be responsible for disabling 100 peoples management ports :O Will take some time off tomorrow and sit down and figure this out.

Can you think of anything you did in between the time it was working, and the time it linked up at 100mbit? If this happened right away after flashing without some trigger, you'd think I'd have 100 angry emails

 

[TheBloke]

Thanks for testing. It looks like you have the same symptom, which is bad news considering you followed the proper flash routine. I don't want to be responsible for disabling 100 peoples management ports :O Will take some time off tomorrow and sit down and figure this out.

Can you think of anything you did in between the time it was working, and the time it linked up at 100mbit? If this happened right away after flashing without some trigger, you'd think I'd have 100 angry emails

Unfortunately I didn't notice until I'd already done a few things with the switch. But I can tell you everything I did up until I noticed:

EDIT: It's the power cycle that does it every time; see later post.

1. Flashed it, rebooted, confirmed my 4 x 10G links worked immediately via pings/ssh's
2. Pinged/telnetted to mgmt#1 and confirmed that worked
3. I did your test for boot-parameter: reboot into bootloader, tried to "type boot-parameter"
4. In the BL I typed "reset" instead of "reload" by mistake, making it hang on that different prompt (date/time), and I had to hard power cycle to get it operational again
5. I followed your quick start guide, setting up an in-band IP and SSH and confirming those worked
6. I enabled Jumbo Frames, which requires a 'reload'
7. I named my 4 x 10G ports and set their MTUs to 9022, and then tested them
8. I then did the power-cycle test for you, to confirm mgmt #1 was still working
9. This is the point I noticed it was at 100mbit - though I didn't mention it until we'd done a couple more tests
   1. the reason I noticed it here is that after this hard power cycle, my LB6M fan started to make an annoying chirpy sound - which caused me to go into the server area to listen closer.
   2. I assume this can't be FW/software related! I've barely used the switch since I got it, and know nothing of its used history, so I figure the sudden power cycle on/off has just tipped the fan over to need replacing sooner. It's only making the noise intermittently now.
   3. (I have remote-controlled power sockets on the server and switch, so the earlier power cycles didn't require me to go into the server area - I only did so to hear the fan sound better.)

So the items that jump out to me as unusual steps are the drop to bootloader, the 2 x hard power cycles, and the hang caused by doing 'reset' instead of 'reload' in the BL, necessitating one of those hard cycles.

That said, I suppose theoretically some of those other 100 might also have the issue but just not noticed the port is now 100mbit, given there's no loss of functionality (at least not yet for me)? I wouldn't have noticed for a long time had I not had the fan noise, and the only reason it jumped out at me is because I had only plugged in the management cable for the first time 30 mins earlier, and in checking for connectivity I thought "Oh, it's gigabit - I expected a management port to be 100mbit." So it was fresh in my mind, where others might not notice it for ages - at least unless until it actually stops working.

 

[TheBloke]

One correction: that 'weird date prompt' I mentioned after 'reset' is nothing to do with the switch! I just realised that was the terminal program I was using. I've now switched to Putty and after 'reset' it's just an immediate hang, no further output from the switch, as no doubt you saw too.

Sorry if that was confusing.

I'm now trying to re-create some of the steps I went through to see if I can cause mgmt#1 to drop back to 100mbit.

 

[TheBloke]

OK it's really simple - every time I hard power cycle the switch, the management port goes to 100mbit.

It does not do so after a 'reload'.

I can't see a way to do a software shutdown, so not sure I can test if that's different. But a hard power drop & power on definitely causes 100mbit each time.

But I haven't so far ever lost connectivity or usability - it continues to work fine for telnet and SSH when at 100mbit.

Each time it goes to 100mbit I have been able to fix it back to 1G using the Ethernet Test.

 

[TheBloke]

Is it worth trying to re-flash the BL? Ie copy tftp flash 192.168.1.49 brocadeboot.bin bootrom

Or could that do no good?

 

[fohdeesha]

OK it's really simple - every time I hard power cycle the switch, the management port goes to 100mbit.

Right, that's the main symptom of this bug. That's why right after you cleanly flashed I asked you to cold power cycle it and check. But it seems by then it was already too late as you had ran "reset" when I first asked you to check boot-parameter. my fault!

edit: nothing to do with reset command, its a bug in the firmware

 

[fohdeesha]

Is it worth trying to re-flash the BL? Ie copy tftp flash 192.168.1.49 brocadeboot.bin bootrom

Or could that do no good?

perfectly safe. In fact that's what I'm doing right now, re-flashing both the bootloader and software image and doing a cold power cycle, and seeing if this bug goes away

 

[TheBloke]

OK I just tried reflashing the bootrom and the primary, and no that didn't fix it - next power cycle I have 100mbit on mgmt #1.

One more piece of data: I tried disabling and re-enabling "interface management 1" and this also causes it to come back at 100mbit - no power cycle required.

 

[fohdeesha]

OK figured some things out, some good:

1. It's quite easy to revert to fastpath, so it's no longer a 1 way trip. I'll add this to the guide with the proper memory read checks, but just so you know it's possible (I don't recommend actually following this yet, a lot of things need to be verified):

Quanta Revert - Pastebin.com
(EDIT: don't follow this. it's missing a lot of important uboot environment variables. wait until it's in the guide)

2. The unique base MAC of your switch (that each port/vlan gets iterated on) is stored in EEPROM. When flashing to brocade, it now has no idea how to read that EEPROM (Brocade expects it to be in a different part of EEPROM), so it defaults to 00e0.5200.0100. On it's own this isn't a problem, but if you have multiple flashed LB6M's on the same l2 network, you're obviously going to have some collision issues. There's an easy command to change the base MAC to resolve this, I will also be adding this to the guide (along with how to fix the weird serial number, and set it to what you want) - This makes it a good idea to note down the base MAC of your quanta before flashing. I don't think there's a sticker with this anywhere on them usually, so after flashing it's gone and you'll need to set it to a randomly generated MAC (nothing wrong with this, though)

3. Under Brocade, you can slow the fans down. at configure terminal level, do "fan-speed 1" and it'll slow them down to level 1 permanently instead of auto-cycling through the 3 fan speed levels. Just keep an eye on your temps, mine went from 30c to 45c which is still fine, but could be worse for people in different environments. There's also "set-pwr-fan-speed" which controls the speed of the fans in the power supplies, but I haven't messed with it yet

If any of you still have a stock unflashed LB6M, I *REALLY* need the output of the command "printenv" in the quanta bootloader

 

[TheBloke]

Great work @fohdeesha

That's great about the fan speed setting, I'm going to try that immediately. Between my server and JBOD I have 8 fans running at up to 4500RPM, and I can still clearly hear the LB6M over the top of those. It's much higher pitched for some reason.

Let me know the MAC address change command when you get a chance and I'll give that a go.

Presumably none of this is causing the management port to drop to 100mbit? Last night I was wondering - could the MAC address be the cause of one person's management port going dead completely? Because although mine drops to 100mbit on every power cycle, it hasn't ever yet stopped working. But if someone had MAC conflicts, that could cause the port to become inaccessible? Or do you expect that my management port will eventually progress to not working at all?

 

[fohdeesha]

I don't think any of it has to do with the mgmt port, I already tried changing mac with no change to that. It's possible it will only work correctly with the MAC the switch has from factory because it's stored in EEPROM somewhere I can't see, in which case I need to open my switch up and hope there's a sticker with the original MAC, and set it to that.

I honestly think it might be that "reset" command corrupting something, so now that I've flashed back to quanta succesfully I can start the flash procedure all over again and see if it's remedied. I'm inclined to think that you do have the issue as the phy eth test brings it back to 1gbe. I'll poke around some more tonight and get that figured out because it's bothering me! My mgmt port went right back to working perfectly once I got fastpath back on there, which tells me again it's a 100% software issue

 

[TheBloke]

OK fair enough. I'd be glad to go back to Quanta and then back to Brocade to test - I read your pastebin, it seems simple enough. But I don't have the FW lb6m.1.0.0.10.bin, the earliest I have is 1.0.0.13. Could you put that up somewhere?

Also an MD5 for uboot.bin would be great, just to confirm the uboot.bin file I just found I already have (but haven't ever used) is definitely the right one.

Thanks again.

 

[TheBloke]

On a different matter, could someone confirm that the following RJ45 SFP should work: AVAGO ABCU-5710RZ GbE SFP Copper RJ45 GBIC Transceiver 1000BASE-T

I know it was said any SFP works, it's just I've never used an RJ45 SFP before so I don't know if there's anything further I need to check in terms of specs/capabilities. I just want an SFP I can plug any normal Cat5 or Cat6 cable in and connect up to various devices and dumb switches.

I'm thinking that now the Brocade LB6M can use 1G RJ45s, maybe I don't need my LB4M any more. I was going to use the 4M to collect all my 1G ports around the house then uplink over 10G to the 6M. But it seems I can get RJ45 SFPs for £8-£10 each, so I could get six of those, giving me 10 x 1G ports (including the 4 built-in ports) in the same switch as my few 10G links.

That would by physically neater, and would also mean I'd only need to learn and configure the Brocade software, and not bother with two sets of switch software.

TIA.

 

[TheBloke]

@fohdeesha I have a MAC for you. It is indeed on a sticker inside. I've not (yet) opened mine, but I remembered I had downloaded a few internal photos posted by another forum member early in the main LB6M thread.

There's a sticker that shows: "E89A8F 138586", which I checked on macvendors.com as E8:9A:8F:13:85:86 and it showed vendor = Quanta Computer Inc.

Spoiler: Here's the photo for reference:

I will open my LB6M later to find out what my MAC is so that I can reset it correctly.

 

[fohdeesha]

Actually, this may be way simpler than all of that: Do you have your management port plugged into the same L2 network as the other quanta ports? If so, they both have the same MAC(the default vlan 1 MAC, anyway), and you're going to get collisions which are gunna screw that port up. Unplug everything from the quanta except management 1, and do a hard power cycle. See if it comes up as 1gbe and acts normally

 

[BeardedKiltGuy]

I Found the Issue with duplicate MAC addresses yesterday afternoon. I see that there is already a fix in the works. I have 2 flashed switches that i would like to try this on. They came from Unix+ and they have the MAC on the side of the case as well. If there is testing that i can do i will try and help. If i Could get the command to fix the MAC that would be great. Thanks

 

[TheBloke]

Actually, this may be way simpler than all of that: Do you have your management port plugged into the same L2 network as the other quanta ports? If so, they both have the same MAC(the default vlan 1 MAC, anyway), and you're going to get collisions which are gunna screw that port up. Unplug everything from the quanta except management 1, and do a hard power cycle. See if it comes up as 1gbe and acts normally

No, unfortunately not. I have my management port plugged into a dumb Netgear 1G switch, which connects to my general house 1G network. But there's no other cables from the LB6M into that network, and no software route between 10G and 1G networks.

Just for the hell of it I did try pulling the other 4 cables out of the Quanta and turned it on, such that the only cable out was mgmt#1, and it came up immediately as 100mbit like before.

To be specific: besides management, I'm using only four other LB6M ports, all at 10G: two DACs to my server, two fibre cables to my workstation. Two 10G ports are configured on 192.168.100.0/24 and the other two on 192.168.200.0/24, where the main 1G network is 192.168.0.0/24. There's no router on the 10G subnets, so no way for it to reach the 1G network even when the cables are plugged in.

Also, if it was collisions, wouldn't it tend to drop back to 100mbit other than at a power cycle? As soon as I run the Ethernet test it goes to 1G and stays there as long as I never power down - regardless of how much I use it. But on power-on it's instantly 100mbit.

By the way, I noted when I just tried this test that the port comes up (at 100mbit) almost immediately after I powered the switch on: like 1 or 2 seconds. Ie the switch is definitely at the bootloader stage (around when it asks 'press b to interrupt..'), and at least 30-45 seconds before the OS is properly loaded. Don't know if that helps at all, especially as the port also drops to 100 when it's disabled then re-enabled via the switch OS.

 

[TheBloke]

I do have a 1G smart switch here, another Quanta: the LB4M. So if it's any help I could get that set up and plug mgmt #1 into that, which would give me proper diagnostic abilities, ie looking for collisions, watching logs to see what happens when it syncs at 100mbit, etc. Let me know if that would be useful.

PS. I've also tested unplugging and re-plugging the mgmt #1 cable from the switch: as confirmed yesterday, when it's 100mbit it stays 100mbit. But also, when it's 1G it stays 1G. So it's not that any fresh connection causes it to drop to 100. It only drops on a power-cycle, or when the interface is disabled and re-enabled in the running switch OS.