Quanta LB6M (10GbE) -- Discussion

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Ryan Anstey

New Member
Jan 17, 2018
3
4
1
38
OK @Ryan Anstey I've just been your guinea pig.
Haha, I didn't seem to get notified of your post (or we both did this at the same time yesterday) so I just went at it cause the noise was getting to me. I don't usually work with hardware that much, so my goal was to just touch nothing and not die, lol.

A few days ago I did the main fans, that was easy (3x Noctua NF-A4x20 FLX):



The PSU (Delta DPSN-300DB (for the Google keywords)):

I used 2x Noctua NF-A4x10.



I used the rubber "screws" because the original screws nor the ones that came with the Noctua's didn't fit. I needed small needle-nose pliers to get them through.



I used the 3-to-2 pin adapter that came with the Noctua's and then matched the colors that were connected previously:



Everything is super silent now!
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Good stuff @Ryan Anstey . Those fans look really nice - but damn are they expensive over here. To buy three of the 20mm and two of the 10mm would cost me at least £75 (CAD$130). That's the same as I paid for my 48-port Gigabit LB4M switch! :D

I see that the Noctua fans have some kind of modular cable thing, so it can work with any pin combination? That's a nice feature - I can see why they cost a bit more.

Personally if I do any mod I think I'll just do a couple of 12cm fans on top, which I already have plenty of. But the noise issue is hopefully only temporary - before too long I hope to be able to move all my gear into my garage. Just need to work out cabling through the garden, as the garage isn't attached to the house.

Anyway glad you're happy with your mod!

PS. how are your switch temperatures with the 3 x 20mm Noctua's? Do you know what they were before you did the swap?
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Damn you, Ryan, now I'm wondering if I do need to spend silly money on quiet fans :p :)

If I can find a suitable replacement for the three fans in my server, then that would be interesting. There's not much point silencing the switch while the server still sounds like a (muffled) jet engine.

The server fans are 3 x 80mm, and Noctua do have 80mm options, including PWM. But theirs are only 55 m^3/h, running at up to 2200 RPM. My current fans run at up to 5700 RPM, providing 136 m^3/h. That's quite a shortfall. Then again, the fans don't usually exceed 4000 RPM. But it's still going to be quite a bit lower airflow from the Noctuas, so I'll have to try and work out if that can be enough cooling. I am running pretty cool at the moment so I have a little leeway to increase temps.

For the switch, I just noticed that Noctua have a 200mm fan. It's pretty expensive, although I guess only double the price of one of their 40mm fans.

I'm wondering if one single 20cm fan mounted on the top of the Quanta could server as cooling for the main switch and one PSU.

It only runs at 800rpm, but that provides 150m^3/h of air, which is 15 times higher than a 40mm fan, 5 times more than three of them. And it should be quiet enough to be inaudible.

That'd still require massacring the case, but work out a lot cheaper - and perhaps even quieter - than buying five individual 40mm fans for the switch and PSU.

Although a worry is whether the PSU will even start up without any fan detected. It might refuse to for safety reasons. Then again yours works fine when connected to a 2-pin fan, so maybe it doesn't care as long as temps remain low. Or the Noctua wire does something to fool the third pin into thinking a fan is there, which I could emulate with a custom wire.

EDIT: Well, I'm not one to stop myself buying things. Ordered 3 x 80mm fans, 2 x 60mm, 1 x 120mm and 1 x 200mm. Who was it who said this was a silly amount to spend on fans? Can't remember :)

Only part of the purchase pertinent to this thread is the 200mm, which I shall attempt to mod into the switch as combined cooling for the motherboard and PSU. I shall report back in due course :) I may not need all the other fans for my server, so I might end up putting two fans into the switch if one provides incapable of properly cooling everything. My main concern is that there's no easy way to monitor the temperature of the PSU. But hopefully it shuts down well before damage can be done.
 
Last edited:

charlie

Member
Jan 27, 2016
58
3
8
Budapest, HU
Hi,

Is LB6M capable to be a "core" switch in small soho network (with 4 48 port gigabit switch). Including routing and OSPF to the uplink? (both ipv4 and ipv6)? Anybody using its for this porpuse?

(especially with this new brocade firmware)
 

segfault

New Member
Apr 5, 2016
29
13
3
43
hasno.info
I use it that way but I don’t do any ipv6. It’s my core with two other switches as the edge (ex4200 and 3750x).


Sent from my iPhone using Tapatalk
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
The new Brocade firmware is great and highly recommended to flash to. Yes it has OSPF and other routing protocols for ipv4. But no routing for ipv6 - a limitation of the hardware.
 

charlie

Member
Jan 27, 2016
58
3
8
Budapest, HU
What can be an alternative device if i need ipv6?

Edit: I found some IPv6 routing related thinks in the documentation, so is that not true?
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
there's a LOT of stuff in that docu that doesn't apply. no ipv6 routing. you can assign ipv6 addresses to management interfaces and stuff (on brocade fw anyway) but no routing
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Edit: I found some IPv6 routing related thinks in the documentation, so is that not true?
For LB6M on Brocade FastIron software, this is the doc that describes exactly what the TurboIron 24X - and therefore the LB6M once flashed - can do: TurboIron24x_08001_ConfigGuide.pdf.

Pre-flash, on FastPath, will be a bit different. Earlier in this thread someone has posted various docs, but they're not nearly as complete as the Brocade docs, and again probably don't exactly apply to the LB6M.

In either scenario there's no ipv6 routing as the switch lacks necessary hardware support.

The LB6M is a reference Brocade design that was never sold to the public, but rather sold by Quanta first to Microsoft and then later also into Amazon and maybe some other big companies. The models we all have were made for Amazon. So there's never been a proper published manual for it. But the Brocade TurboIron was sold publicly and was kept updated (at least until last year) and is well documented. It's identical in all respects to the LB6M, besides a couple of non-vital things like port LED lights and the management port. The result being that once the LB6M is flashed to Brocade FastIron, it works as per that doc above, except the LEDs on the 10G ports don't work, you only get one management port, and that management port is a bit odd (it will default to 100mbit half duplex on every hard power cycle, requiring a special command to fix.)
 
Last edited:

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
I'm going to re-write my previous post as it was a mess of updates and edits:

I have done a quiet(er) fan swap on my LB6M, running Brocade firmware. I got 3 x Noctua 40mm x 20mm 3-pin fans for the main switch. I wanted 4-pin PWMs but they were out of stock for a few days, so in my impatience I got 3-pin thinking they'd be fine. How annoying; they're not.

The core problem is this: most of the time (but not all) with 3 x 3-pin fans connected, show chass gives me no temperature information, and always shows all fans as OK:
Code:
Fan 1 ok, speed (manual): 1<->[[2]]<->3
Fan 2 ok, speed (manual): 1<->[[2]]<->3
Fan 3 ok, speed (manual): 1<->[[2]]<->3
Fan controlled temperature: 0.0 deg-C
...
Exhaust Side Temperature Readings:
        Current temperature : 0.0 deg-C
        Warning level.......: 80.0 deg-C
        Shutdown level......: 90.0 deg-C
Intake Side Temperature Readings:
        Current temperature : 0.0 deg-C
EDIT: as per next post, this is not specific to me, or Brocade. I've found an old post where another guy had the same problem when using 3-pin fans, according to his "show environment" output on FastPath, which showed Temp = 0 Celsius. And further, my testing has confirmed that manipulating the fourth pin does weird things to the temp monitoring of show chass.

If I plug in 3 x 4-pin PWM fans, show chass immediately works as normal. Tested both with the original 3 x 40mm fans, and also three Noctua 4-pin PWM fans. When 3 x 4-pin fans are connected, I get working temperatures and working fan detection.

But what is very strange is that the first time I connected my 3 x 3-pin fans, I got working temps, and over multiple power cycles. I connected the three fans, I booted, I checked show chass, it had temperatures as normal. I then power cycled several times while changing the PSU fans, and every time I booted I checked show chass and it was always working fine.

Once everything was setup, I let the switch idle for about an hour, monitoring the temperature. I then decided I would test temperatures at a lower fan speed, using the Noctua low-noise cables. I swapped all three Noctua fans out with their Low Noise Adapter (LNA), and power cycled.

This is when the problem started - after that boot, and most every subsequent boot, I got no temps in show chass. And if I disconnect a fan, it still shows as "Fan OK".

Since then I have tried the following without change:
  • Fiddling with wires/checking connections etc
  • Power cycling five, six, seven times in a row, with gaps in between
  • Pulling and re-plugging the fan tray
  • Unplugging each of the three Noctua fans in turn, to the point of having 0 fans connected
  • Booting with 1 x original fan + 2 x Noctua
  • Booting with 2 x original fan and 1 x Noctua
  • Booting with 1 x original fan or 2 x original fan - which is really weird, because this is a standard failure case! (EDIT: Ah, it depends which fan is connected. If the one nearest the PSU is connected - fan 3 - I get temp figures and that one fan behaves normally; if it's not, then any connected fans run at 100% speed and no temp figures are shown.)
  • Putting in the fan tray from my LB4M, which is the same model number, and trying the 3 x 3-pin Noctua fans in that instead.
However, some things did work, briefly:
  • As mentioned, having 3 x 4-pin PWM fans always seems to work
  • 3 x 3-pin worked when I first put them in, and continued to through a couple of power cycles
  • After that, I twice got it working again by:
    • Running with 1 x original + 2 x Noctua, booting = not working
    • Replacing the 1 x original with the third Noctua and booting again = then it worked.. but only to the next power cycle
    • This sequence worked twice in a row, but then the third and subsequent time I tried it, it stopped working.
I am really at a loss as to what is happening here. I thought the 4th pin was purely an output?

Then there's the question of why it worked perfectly the first time I connected everything up, and for at least an hour afterwards, and then almost never worked again after I first connected the 3 x LNA cables. But then did work twice more, very briefly, after I swapped 1 x original fan for a Noctua.

Maybe there's a way to connect the fourth pin to something, to 'fool' the motherboard, given it seems to be really affected by not having something on that pin?

Thanks in advance.
 
Last edited:

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Doh, it looks like this is not specific to me, and must be a HW issue. I just searched through the whole thread, and found this:

So I swapped out the 3 fans today.
....
New Fans
Sunon 40x40x20mm 3 pin fan #KDE1204PKVX
....
(FASTPATH Routing) #show environment

Temp (Celsius)................................. 0
It probably wasn't as obvious on FastPath, but it's clearly also showing 0 Celsius for that guy when 3-pin fans are used.

So I still don't understand why/how I got temp monitoring working on several boots when using three-pin fans.

I also don't understand how the lack of a connected 4th pin can mess up temp sensing. Maybe some low level issue? It's expecting a connection on that pin and that causes a HW bug?

Could I connect that pin to.. something? I don't really know how fans work. I know it's a PWM signal pin, so I suppose it sends a fluctuating voltage down it that controls the motor? I could put a high value resistor between it and ground? I don't want to risk damaging anything..

UPDATE: Ha! Yes that fourth pin sure is relevant. I got out my multimeter and, with the switch running and (Noctua 3-pin) fans operational, I (carefully!) probed the pins on one of the fan connectors:
  • Between pins 1 & 2 (Black and red / Ground & 12V): +12V, of course
  • Between pins 1 & 3 (Ground / Tach): Around 3 volts, slightly fluctuating
  • Between pins 1 & 4 (Ground / PWM): Around 0.5 volts, slightly fluctuating
Then I went back to show chass, planning to change the fan speed from 3 to down to 1, to see the effect on the PWM voltage. First thing I see:
Code:
Exhaust Side Temperature Readings:
        Current temperature : 128.0 deg-C
        Warning level.......: 80.0 deg-C
        Shutdown level......: 90.0 deg-C
Intake Side Temperature Readings:
        Current temperature : 128.0 deg-C
Hmm! Well that sure did something :) What the hell is it using this fourth pin for, and why is it even connected to temp monitoring? Are they somehow monitoring temp from the fourth pin?

Also notable is that 128 is well above the shutdown level, but of course it did not shutdown. So maybe shutdown is controlled using a separate temp sensor in the chip. Or more likely the code that controls shutdown knows the real temp, and it's just the temp I see that can be completely screwed by fourth-pin-manipulation. (Note that during all these tests I have a separate, external fan pointing right at the CPU to keep it cool regardless of how many fans are connected.)

This is really weird!

I tried connecting a few different resistors between ground and PWM with a 3-pin fan connected - 120k, 10k, 1k and 390Ω, all of which behaved the same: If connected on boot, I got the same 0.0 temps as usual. However if one of these resistors was connected between ground and PWM after booting (or disconnected and re-connected if already there from boot), show chass then showed the weird 128.0 temp I also saw when probing with my multimeter.

So.. I'm able to get some result? :) Not a very helpful one. I have no idea if I'm even on a useful path. Maybe it's actively monitoring this 4th PWM pin, despite it (I thought) being output only, and expects some specific and/or constantly varying resistance on it.

But can it even monitor the resistance of a pin it's also sending a voltage down? I suppose if it alternates between sending voltage and reading resistance. Maybe I should probe it with my oscilloscope. Not that there's much point; if it's doing anything complicated I'm not likely to be able to re-create it anyway.

My knowledge of electronics is sadly very limited. I'd really be grateful if anyone knows the first thing about fans/PWM etc and has any ideas. Maybe this concept of not working without a 4th pin is not uncommon?

I could just buy new fans, but now this feels like a challenge that needs to be solved :) Anyway they're out of stock, so I can't buy PWM fans for a few days regardless.

Or I could just run with 3-pin fans and unable to see temps and detect dead fans. I have some confidence that even though I can't see the temp, the shutdown routines probably can. Anyway if I just run the Noctuas at full speed, I don't expect to get anywhere near shutdown temps. It bothers me though, not being able to monitor :)
 
Last edited:

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Sorry for all the posts, last one I promise. Two more things.

First:
  • If I boot with a 4-pin PWM fan in the fan 3 position (closest to PSU), temperature monitoring always works, even with two 3-pins also connected
  • If I then swap this 4-pin fan for a 3-pin while the switch is running, temp monitoring continues to work from that point onwards.
  • It breaks again if the next power cycle has 3 x 3-pins again.
But there's also something else interesting about booting with a single 4-pin PWN fan alongside 2 x 3-pins. The 4-pin.. pulses. I can hear it constantly spinning faster and slower, faster and slower. It defaults to fan-speed 2, and in this fan-speed 2 I can hear the constant pulsing. If I then set it to "fan-speed 1", the pulsing stops. And so does the fan. It stops moving completely, and a few seconds later "show chass" shows it as failed. If I set it back to fan-speed 2, it starts working again, doing the pulsing.

fan-speed 3 also pulses, but at faster speeds. It makes a much more noticeable whirr-WHIRR-whirr-WHIRR type sound.

I used the multimeter to monitor the PWM during this pulsing, and in "fan-speed 3" the voltage was fluctuating between 0.05 and 3.5 volts in a regular cycle. So more or less an on-off cycle, where at fan-speed 3 it should have been full on, all the time.

This of course doesn't happen when I boot normally with 3 x 4-pin fans, or even 1 x 4-pin fan. The presence of the 2 x 3-pins is causing this behaviour in the 4-pin.

So.. are the fans inter-dependent? Does having 2 x 3-pin fans connected at the same time as 1 x 4-pin cause the 4-pin to behave differently? Well it clearly does, but why? Are the 4th pin of each connector inter-connected somehow?

Secondly, I tried booting with 3 x 3-pin (no working temps), then unplugging fan 3 and connecting a 4-pin instead:
  • Once the 4-pin was connected, show chass showed the weird 128-degrees 'reading'
  • The 4-pin fan put into position 3 ran permanently at full speed, ignoring my "fan-speed" setting. From boot it only does this if fan 3 is missing. Indicating that post-boot it doesn't really detect this fan properly; or that there's a specific at-boot check for fan 3.
  • Unplugging the 4-pin fan and re-plugging it did not clear out the 128-degrees, nor stop the full-speed.
The first and second points together seem to indicate that both working temps and the presence of fan 3 have to be established on boot, and once established will stay working, but cannot be established later. In the case of the 128-degree reading, this is the same as achieved by my earlier dumb resistor test. So connecting anything to pin-4 after boot seems to give the 128-degree reading to temp monitoring, even if it's a real fan.

And maybe the first point also indicates that the three fans are considered together somehow - or at least, missing fourth pins on one or two of them causes weird behaviour in fan(s) that do have it.

Finally, the first point above lead itself to a potential workaround, albeit a pretty crappy one: boot with 1 x 4-pin fan temporarily connected, then disconnect after boot and plug the intended 3-pin fan. Thereafter temp monitoring will work with 3-pin fans.

Maybe usable in the short term, at least if I knew the switch was going to be running without power loss for days or weeks. But not a long term solution.

(Sorry to keep posting about this, probably no-one else cares.. just buy the 4-pin PWM fans! But I find it fascinating - something so weird is going on here. Lucky me to stumble into it because Amazon were out of stock of the fan I wanted :D )
 
Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I believe it's a hardware/software watchdog. Both the fan speed and temperature sensors are all read through the same I2C bus. If it sees it's failed to get legitimate readings from multiple fans in X amount of time, it assumes the I2C bus has failed and the temperature readings it has are potentially bad, and forces them to 0 it looks like. This is one of several reasons why I always wondered why people put so much time and effort into fan modding enterprise gear - just stick the thing in a basement closet or something and forget about it :p

Also I really doubt it's going to attempt speed control on any fan if it can't get a legitimate RPM sense signal back from them (4th pin), so it probably just defaults them all to high speed

i dunno if you've dicked around with the power supply fans yet, but at least on the brocade's I've used, if the RPM reading from the power supply fans are less than 20 or 30% lower than what it expects (or no reading at all), it would refuse to boot and mark the power supply as failed. half the time it would fully boot, then shut off after a period of time marking the PSU as bad. There was a third strange failure mode where it would work fine, but if the power feed to one psu failed, it would not let the redundant 2nd PSU take over, because it was marked as failed, so it would just shut off

I went through the trouble of sourcing the exact delta fan replacements for the PSU's of a brocade FCX POE (which is about twice as loud at idle as an LB6M on full blast), except a 50% slower RPM model. They fit perfectly and had the correct pinout and everything, but it did not like the slower RPM. That was the first and last time I dicked around with modding fans

tldr: there are a LOT of software checks and logic to monitor all the various busses and devices in enterprise gear, a lot more than people realize. They're written on a set of assumptions about the data they'll get back (types of fans, speeds, etc). When you start modifying the hardware like this, expect them to start getting really ****y
 
Last edited:
  • Like
Reactions: TheBloke

Todd Lynch

New Member
Jan 26, 2018
3
1
3
46
I recently ordered an LB6M for my home lab and have been struggling with the CLI for the last couple of days. I'm just not "getting it." I'm no stranger to updating firmware and got both my LB6M and an LB4M updated to the most recent "stock" files I could find. That is to say, I'm familiar enough with the tools to do that particular job without much assistance. But the stock CLI is clunky and the Brocade interface is *definitely* something I'd prefer, even if I can only get it going on the LB6M.

I say all that because... I managed to brick my LB6M tonight trying to do the Brocade transition.

Long story short, I fat-fingered one of the commands when copying the Brocade bootloader - I wound up hitting an 'o' (letter, oh) where I needed a '0' (zero) and missed it before committing the command. Totally my fault. My terminal session went completely unresponsive for over an hour and still won't recover. I feel like an idiot admitting it but that's what happened.

It's one of the new, in-box units from UnixPlus and I'd rather not just toss it as a $300 mistake on my part and have to order a new one. I'd prefer to recover it, if possible, using the JTAG pins I've heard about but I'm a little lost getting that deep into the hardware and haven't used JTAG in any capacity before.

Can anyone offer some how-to or advice on what I need to do at this point? If there are parts I can buy to do this, I'd prefer that over soldering things on my own unless there's a crazy cost difference. Other than ordering another unit from UnixPlus, I don't have much to lose and, maybe turning this into a positive, I could serve as a good point of documentation for anyone else that gets stuck in my situation down the road.

Thank you for any help any of you can provide.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I say all that because... I managed to brick my LB6M tonight trying to do the Brocade transition.
We have our very first Quanta > Brocade brick! Congratulations - I'd give you a sticker but we haven't had any made :p

Long story short, I fat-fingered one of the commands when copying the Brocade bootloader - I wound up hitting an 'o' (letter, oh) where I needed a '0' (zero) and missed it before committing the command
This is *exactly* why one of the first things I put in the guide is to only copy and paste commands, do not try to type them out :)

If you did indeed brick it, the only solution is to use a JTAG unit to put the bootloader back onto the onboard flash. However PowerPC-capable JTAG units generally run about the price of a brand new LB6M. However I ordered a PPC capable unit last week to do some further hardware debugging.

I'll unbrick your LB6M for free if you pay shipping both ways (shipping it to me, then shipping it back after I JTAG the bootloader back on). If you want to calculate shipping I'm in indianapolis Indiana, 46280. If you're in a major metro area in the US I'd guess shipping with fedex would probably be about $20 each way.

First though, I would confirm you actually bricked the thing. The only way to brick it is erasing and/or overwriting the bootloader incorrectly, and doing that wouldn't immediately make the terminal unresponsive like you said yours did, since the bootloader program runs from RAM. it sounds like you might have accidentally overwritten a location in RAM it was running from. To check, just pull power from the switch and then power it back on, see if it comes back up
 
Last edited:

Todd Lynch

New Member
Jan 26, 2018
3
1
3
46
Re-flashing it via JTAG with your equipment would be mighty kind - thank you! I'm in the Minneapolis area so it shouldn't be too bad for shipping.

I know it's bricked. I had already issued the delete command before I sent the bad copy command. The putty window was unresponsive. I closed it and attempted to re-open it but the LB6M wouldn't give me any response. At that point, all I could do was pull the power and pray I didn't totally hose it. Unfortunately, while it now powers up, I still get nothing from the serial port nor do the fans spin down.