Quanta LB6M (10GbE) -- Discussion

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
  • If I boot with a 4-pin PWM fan in the fan 3 position (closest to PSU), temperature monitoring always works, even with two 3-pins also connected
  • If I then swap this 4-pin fan for a 3-pin while the switch is running, temp monitoring continues to work from that point onwards.
  • It breaks again if the next power cycle has 3 x 3-pins again.
There is a common mis-perception that the 3rd wire on 3-pin fans is always speed control and the 4th wire on 4-pin fans is always speed monitoring, because that is usually how PCs use 3- and 4-wire fans. That isn't the case and even if it was, there are many options just for speed monitoring. To get an idea of what sort of combinations there, look at this Nidec PDF. This is one reason I don't buy "modder-class" fans - I want fans with detailed datasheets, not (at best) an airflow graph or two.

As someone else pointed out, fan speed monitoring and temperature sensing are probably running over a single bus. The switch was designed with speed-monitoring fans, so by definition if no fans are reporting a speed signal it is more likely that there is a single failure affecting that bus than all 3 fans failing that way. The hardware and software designers didn't forsee users who wanted to change out all the fans for incompatible ones. :p
 

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
I got a hack working :D

Using a 3-pin fan with a 120k resistor between +12V and the empty fourth pin = working temps, working fan detection:
Code:
fan 1 failed
fan 2 failed
Fan 3 ok, speed (manual): 1<->[[2]]<->3
Fan controlled temperature: 32.0 deg-C
..
Exhaust Side Temperature Readings:
        Current temperature : 32.0 deg-C
        Warning level.......: 80.0 deg-C
        Shutdown level......: 90.0 deg-C
Intake Side Temperature Readings:
        Current temperature : 30.0 deg-C

@Terry Kennedy thanks so much for that PDF. I know you meant it to show me the vast variation in fans, but it was page 2 that got my hackaday solution working. It had the circuit diagrams I had been failing to find earlier, in particular showing a pull-up resistor between 12V and signal, in the Tach diagram.

I tried multiple resistors earlier against ground, but was a bit worried putting one to 12V. After seeing the diagram, I tried it, and boom! Not literally 'boom' :D

And the good news is this is only needed on one fan - the one in position three, nearest the PSU. This appears to be the key fan probed on boot, which has to pass the checks for temp monitoring and fan detection to work. I'm running now with this fan modded with the resistor, and the other two just connected direct with 3-pins, and it's fine.

I thought this was proof that, as you guys cautioned, the wires were not in the order I thought. I thought the fact that a pull-up resistor on pin 4 meant pin 4 must be actually the Tach pin, where such a resistor is included in the design. But as I have later discovered and will describe in my next post (volume 9 of this epic tome..), they're clearly not swapped, and swapping over pins 3 & 4 breaks things in the same way as using a 3-pin fan.

i dunno if you've dicked around with the power supply fans yet, but at least on the brocade's I've used, if the RPM reading from the power supply fans are less than 20 or 30% lower than what it expects (or no reading at all), it would refuse to boot and mark the power supply as failed. half the time it would fully boot, then shut off after a period of time marking the PSU as bad. There was a third strange failure mode where it would work fine, but if the power feed to one psu failed, it would not let the redundant 2nd PSU take over, because it was marked as failed, so it would just shut off
Already done the PSU and thankfully no such problems. In fact I'm running them in 2-pin mode, so the PSU is not getting any tach data. Works just fine. I tested this before re-wiring and confirmed that it would in fact run with no fans connected at all (with an external fan blowing in for safety during the brief test.)

That said, I'm only ever running with a single non-redundant PSU (I've not modded the second one.) So I could theoretically be hitting your third failure mode and just wouldn't notice. But if I am, that's the only one I could be hitting - I get "Power supply 2, status OK" on every boot.

In the case of the PSU fans, the wiring was already known to be in a different order to 'standard', and the Noctua fans I used come with adapters to adjust the pin ordering easily for this case . I could have connected the Tach pin as well, but as I knew it worked without, I didn't bother as wiring space within the PSU case is very limited.

- just stick the thing in a basement closet or something and forget about it :p
If I had a basement, I wouldn't be going to all this trouble :p My original plan was to put all my gear in my separate garage, but that requires a lot of cleanup work in the garage as well as planning a safe and tidy way to run multiple sets of 10G and 1G cables out through the wall of my house, through 10' or so of front garden, and then through the garage wall. I've investigated the necessary cabling, but working out where/how to dig up the garden - or a suitable route around the garden - is a bit more complex.

I will hopefully still do that in the end, but doing this noise reduction work for my switch and server means I don't have to tackle it at least until the late spring/summer, and can now run my HW 24/7 in the meantime without driving myself completely crazy with noise :)

Anyway, wasn't all this a far more interesting and engaging exercise than just running with stock fans? :) Well for me anyway :)
 
Last edited:

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
So during my previous tests, I thought maybe pins three and four were swapped. They're not:

I re-wired a 4-pin extension cable to swap pins three and four. Used with either my 3-pin replacement fan or a (non-original) 4-pin PWM fan, this exhibits the usual problems: no temp readings, no fan control, no fan monitoring.

The overall results are:
  • If I plug in a non-original 4-pin PWM fan (a normal PC case fan) directly: I get temp figures, I get fan detection, but I don't get speed control, and it seems to always run at a slow or slowest speed. Except on boot, where it briefly exhibits the normal behaviour of running at full speed during bootloader, then dropping to a lower speed.
    • Also, some fans will then go failed a short while after boot; eg my 12cm Noctua fans do. But my 20cm Noctua does not, nor does a 12cm non-quiet Corsair. Difference in reported RPM speeds maybe? But the 20cm is the slowest of the lot, max 800rpm, and I would have assumed low speed = more likely to be considered failed. But might well be more complicated than that.
  • If I plug in a non-original 4-pin PWM fan via a swapped 3rd & 4th pin: No temp figures, no fan detection, no speed control
  • If I plug in a non-original 3-pin fan, either directly or via the swapped cable: No temp figures, no fan detection, no speed control, but runs at full speed.
  • However, if I plug in a 3-pin fan directly with the addition of a 120k resistor between its +12V pin and the unconnected 4th pin of the connector: Working temp figures, working fan detection, full speed.

So it looks like my nasty resistor hack may actually be the best overall solution, barring any further discoveries. I could buy 40mm 4-pin PWM fans to get temp readings and fan detection, but they'd have no speed control and worse, would run at a fixed low speed, which on quiet 40mm fans is quite likely too little airflow.

I studied in more detail the lengthy fan catalogue published by AVC, whose fans I have in my LB6M. One thing that might relevant - I note that the fans I have are marked as having both a "Tacho" signal and "Rotational Detection" signal, which is described as: This is the simple “Go”/“No-Go” fan signal output to alert the system if the fan is not functioning. It doesn't say which wire this is on, but the obvious assumption would be the same Tacho wire. Though it would make more sense if it was actually on the PWM wire, as that would explain why 4-pin fans generally work and 3-pin don't, without a hack to the fourth wire.

Anyway, yes, I certainly do take you guys' points that these things are much more complicated than swapping out fans in a PC mod :)

There's also the fact that, as I think one of you said, the switch was designed for and tested only with very specific fans. They might have put in some quite specific checks, for example for particular current draws that these fans will always exhibit when working properly, and any other fan is unlikely to.

I think I shall end up making one hacked wire with a resistor for fan position 3, and running with my 3 x 3-pin fans and just leaving it at that. Not sure much more testing will bring any useful result :)

I am still surprised though that I can't seem to find a definitive answer. Docs like the AVC hint at possible differences - eg the Rotational Detection - but I would have thought I'd be able to find some doc specifying exactly what signals are sent. The PDF Terry gave showed that to some extent for one brand of fan, but despite the AVC catalogue being 138 pages they don't seem to feel the need to explain the full details. They don't even give a wiring diagram. I suppose there must be be other documentation that's not been made accessible without a purchase. I did find this useful Intel specification document on Tach and PWM, though that makes no mention of Rotational Detection nor monitoring the PWM pin.

I probably came into the switch mod with a bit too much (over-)confidence, especially as before doing the switch I swapped out all the fans in my server (Tyan LGA1366 motherboard) with Noctua fans and it worked just fine, interfacing perfectly with the motherboard and IPMI's fan control routines. I do get that integrated components are a somewhat different ball-game, and not intended to be anywhere near as plug-and-play.
 
Last edited:

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
I am still surprised though that I can't seem to find a definitive answer. Docs like the AVC hint at possible differences - eg the Rotational Detection - but I would have thought I'd be able to find some doc specifying exactly what signals are sent. The PDF Terry gave showed did that to some extent for one brand of fan, but despite the AVC catalogue being 138 pages they don't seem to feel the need to explain the full details. They don't even give a wiring diagram. I suppose there must be be other documentation that's not been made accessible without a purchase. I did find this useful Intel specification document on Tach and PWM, though that makes no mention of Rotational Detection nor monitoring the PWM pin.
I was trying to find where you mentioned me and had a question on open collector, and you edited your post between when I read it and I tried to reply. :p

Anyway, the speed sense output (which can be either the 3rd or 4th wire) can be a tachometer pulse (normally 2 pulses/rev, but this varies), locked-rotor alarm (fan not spinning at all), low-speed alarm (typically 70% of expected speed on a fixed-speed fan, etc.), some combination, or something completely different.

Open-collector just means that the output is (normally) pulled to ground to indicate one state of the signal, but the other state of the signal "floats" and it is the device's responsibility to pull that signal up to the desired level with a resistor. For a device using 12V fans and 5V logic, the pull-up would normally be to the 5V line and not the 12V line, as having the logic see 12V would not be a good idea.

Look at page 21 of the AVC catalog for their version of the Nidec "secret decoder ring". Note that none of these manufacturers will tell you in their catalogs which versions are stock and which versions are special-order. You need to look up part numbers on places like Digi-Key to find that out. Woe to anyone who has a special-order fan and doesn't need thousands of them (I used to maintain systems where the cooling fans were 35V AC, 70Hz, square-wave instead of sine-wave) o_O.
 
  • Like
Reactions: TheBloke

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
Well after all that messing about with 3-pin fans, I ended up going back to an earlier idea and used 1 x 4-pin 12cm :)



It works out both quieter and cooler than 3 x 40mm, and doesn't require a resistor hack to get temperature monitoring. When sitting idle in an ambient temperature of 21°C, it settles on 50°-52°C. Compared to 60°-62° with the 3 x 40mm at 3700 RPM (ultra-low-noise adapter), and around 56°C when they were at 4400 RPM (low-noise adapter.) I don't know what speed the 12cm is running at, but I'm guessing it's around 800.

Main thing that doesn't work is fan detection. For some reason the switch doesn't like something about the Noctua 12cm fans and marks it as failed shortly after boot. Not sure why - it's not related to low RPM, as my Noctua 20cm runs at 400-800RPM and that worked fine. And other 12cm fans worked OK. Another peculiarity of the fan monitoring in this switch.

But no big deal. With only one fan it's either working or not, and if it stops working the temperature will rise fairly quickly and I'll pick that up in monitoring.

I also can't control the speed of the fan - the switch always runs it at what looks like its slowest speed. But that's turned out fine as it's cool enough, and of course nice and quiet at this speed. Most or all of the noise I can hear is from the 2 x Noctua 40x10mm fans in the PSU, which I let run at their full speed.

Compared to stock it's like the soft breath of a whispering baby mouse. I'd say it sounds something like a basic desktop that has a single stock CPU fan running at low-mid speed. Not silent when it's right behind me, but barely noticeable. And when I have it back in my office's annex area it should be inaudible.

Going with the 12cm has also freed up the 3 x 40mm in case I decide to also use my LB4M 1GB switch daily, in which case I can quiet that as well.

Any road, I've spent far too much time - and more money than I care to admit to - on all this quieting business, so now it's time to go back to actually using all this stuff :) But I can now run my server, switch and disk tray in the house without it being a constant annoyance.
 
Last edited:

Todd Lynch

New Member
Jan 26, 2018
3
1
3
46
Just recovered @Todd Lynch 's LB6M - it's amazing how fast and easy that is once you spend the money on the right tools. Will ship it back out tomorrow! Hopefully that's our first and last brick :)

Imgur: The magic of the Internet
Confirmed received and in perfect shape. Again - thank you so much!

EDIT TO ADD: What was the outcome of the management port coming up at 100/hdx? I'm doing some basic setup and noticed my management port is showing up as 100/hdx while the the other end is connected to a Cisco SG200-08 (1000/fdx capable and advertised).
 
Last edited:
  • Like
Reactions: fohdeesha

BlueTip

New Member
Feb 4, 2017
5
6
3
Hi, All,

I have been designing a 3D printable fan tray that expands out to 2x80mm fans. The progress is slow as I'm new to 3D printing and I'm relying on the public library print services. However, after three prototype prints I'm really close to a first working design. Here's a rough picture that still needs bolt holes, a couple tweaks, and final fitting. I don't have a very good measure of running temps yet, but I'm confident they'll be <40C with fans pretty quiet on max.

@TheBloke... It's funny we were both messing with the fans around the same time and I can confirm your findings regarding the key fan for monitoring being fan number three next to the PSU - if that one isn't working, you won't see any temperature reading. Also, I can tell you that I have had ZERO luck with Noctua fans in that position (tried NF-A8 and NF-R8 with no luck). What DOES work well for me is the Arctic F8 PWM (5-pack purchased on Amazon). My plan is to populate two fan connnectors and share the tach for the third.

The great thing about this mod is that it's 100% reversible! No cutting! The original fan tray comes out, fan control board is transplanted, and the new expander tray goes in. The intention is to use bolts and washers to fasten the assembly to the existing fan bolt holes.

model1.jpg

Wish me luck, and stay tuned!
 
Last edited:

Andras

New Member
Mar 28, 2017
3
0
1
47
Hi,
Could someone help me?
I have 2 LB6M switches, stock fw, configured as I thought it should be.
Both switches are configured and all ports are set for mtu 9216 but for some reason all machines that are connected to the second one have normal connection and speed only if the mtu is set to 1500.
The first switch is configured the same way but on that one the mtu 9216 is working correctly, machines that are connected to that switch have normal net connection and speed if the mtu is set to 9000.
Here are the running configs of the switches
Thx
 

Attachments

TheBloke

Active Member
Feb 23, 2017
200
40
28
44
Brighton, UK
I have been designing a 3D printable fan tray that expands out to 2x80mm fans. The progress is slow as I'm new to 3D printing and I'm relying on the public library print services. However, after three prototype prints I'm really close to a first working design. Here's a rough picture that still needs bolt holes, a couple tweaks, and final fitting. I don't have a very good measure of running temps yet, but I'm confident they'll be <40C with fans pretty quiet on max.
Wow, that's an impressive solution :) I can see the appeal of not needing to cut a circle in the top panel, in case one later wanted to sell it.

Hope it all goes smoothly for you.

@TheBloke... It's funny we were both messing with the fans around the same time and I can confirm your findings regarding the key fan for monitoring being fan number three next to the PSU - if that one isn't working, you won't see any temperature reading. Also, I can tell you that I have had ZERO luck with Noctua fans in that position (tried NF-A8 and NF-R8 with no luck). What DOES work well for me is the Arctic F8 PWM (5-pack purchased on Amazon). My plan is to populate two fan connnectors and share the tach for the third.
Yeah the fan detection is weird. I have had one Noctua fan work - the NF-A20, their 200mm beast. But none of their 120mm ones worked. I didn't bother trying any others, but it's good to know the 80mm doesn't work either.

So with the Arctic fans, as well as having them detected OK, do you also have functioning PWM control? You can change their speed from the switch (eg set fan-speed [1-3] in Brocade FW)? That's broken along with detection with my Noctua fan, though that's turned out to be OK because without PWM control it runs at a low/lowest speed, and that's proved perfectly fine for temperature.

My 1 x 120mm fan mod is holding up well. Completely silent when the switch is located in the annex area of my office, about 5 metres and around a corner away from me. Temperature holds steady at 50-52°C when ambient is 20-23°C. And I can get it as low as 40° if I open a door in the annex that opens into an undeveloped roof crawl space, allowing air to circulate which is at 10-15°C (at least for now, in the current wintery climate.)

One thing I didn't mention before: as well as modding the fan I also changed the thermal paste on the main CPU heatsink, using expensive enthusiast/overclocker's stuff instead. For some reason I didn't properly record before/after temps, but it may be helping to reduce temp another 2-4°C or so.

Good luck with your mod!
 

BlueTip

New Member
Feb 4, 2017
5
6
3

BlueTip

New Member
Feb 4, 2017
5
6
3
Wow, that's an impressive solution :) I can see the appeal of not needing to cut a circle in the top panel, in case one later wanted to sell it.

Hope it all goes smoothly for you.



Yeah the fan detection is weird. I have had one Noctua fan work - the NF-A20, their 200mm beast. But none of their 120mm ones worked. I didn't bother trying any others, but it's good to know the 80mm doesn't work either.

So with the Arctic fans, as well as having them detected OK, do you also have functioning PWM control? You can change their speed from the switch (eg set fan-speed [1-3] in Brocade FW)? That's broken along with detection with my Noctua fan, though that's turned out to be OK because without PWM control it runs at a low/lowest speed, and that's proved perfectly fine for temperature.

My 1 x 120mm fan mod is holding up well. Completely silent when the switch is located in the annex area of my office, about 5 metres and around a corner away from me. Temperature holds steady at 50-52°C when ambient is 20-23°C. And I can get it as low as 40° if I open a door in the annex that opens into an undeveloped roof crawl space, allowing air to circulate which is at 10-15°C (at least for now, in the current wintery climate.)

One thing I didn't mention before: as well as modding the fan I also changed the thermal paste on the main CPU heatsink, using expensive enthusiast/overclocker's stuff instead. For some reason I didn't properly record before/after temps, but it may be helping to reduce temp another 2-4°C or so.

Good luck with your mod!

Hi, TheBloke,
To answer your question, I haven't hooked up the tach for the left fan (opposite the power supply), but I'm getting correct PWM control, reporting of fan status, and temperature with the Arctic F8 fans (33C intake, 36C idle exhaust at fan speed 3 - see below).

SSH@10g-sw1(config)#show chassis
Power supply 1 not present
Power supply 2 (NA - NA - Regular) present, status ok

fan 1 failed
Fan 2 ok, speed (manual): 1<->2<->[[3]]
Fan 3 ok, speed (manual): 1<->2<->[[3]]

Fan controlled temperature: 36.0 deg-C

Fan speed switching temperature thresholds:
Speed 1: NM<----->30 deg-C
Speed 2: 25<----->40 deg-C
Speed 3: 35<----->90 deg-C (shutdown)


Exhaust Side Temperature Readings:
Current temperature : 36.0 deg-C
Warning level.......: 80.0 deg-C
Shutdown level......: 90.0 deg-C
Intake Side Temperature Readings:
Current temperature : 33.0 deg-C
Boot Prom MAC: 00e0.52c1.ec8b
SSH@10g-sw1(config)#


Here's some ballpark measurements after 30 minutes idle at each speed 1-3 (updated after repeating testing).

Fan Speed 1: 36C intake, 40C exhaust
Fan Speed 2: 36C intake, 38C exhaust
Fan Speed 3: 33C intake, 36C exhaust

I've got most of the design worked out, and I'm 3D printing the fan assembly in PETG via 3DHubs for about $30 - should be here by the weekend to test. I ended up making the fan shroud reversible (vertically) to allow more flexibility in my rack space.
 
Last edited:

Corsaire

New Member
Feb 3, 2018
21
0
1
I'm using Cisco three meter copper SFP patch cables I got for $5 each on ebay last year... no issues.
Hmm, yeah I used a similar offer and also use some DAC and SFP cards on my computers...

But it's not what my question is about.
Especially when you have a piece of equipment with a 10G RJ45 port, not SFP.

It needs a 10G RJ45(base-T) port on your switch to interface, hence a SFP+ transciever base-T.

So did someone test some of those ?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
So ,what can be a good alternatives with same parameters (24 TGE ports, redundant PSU, not too high price), but with full IPv6 support?
I know I'm a little late here, but the cheapest thing that fits these requirements is a Dell 8024. It's essentially an LB6M (same ASIC) but with a newer faster management CPU and firmware that supports full ipv6 routing
 

alex1002

Member
Apr 9, 2013
519
19
18
I am having a serious problem. Please do not laugh at me. I configured the switch but forgot to write the mem. Now for some reason nothing happens when i plug in the console cable and try to connect. My other switch it works right away when I plug in the console cable.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
I am having a serious problem. Please do not laugh at me. I configured the switch but forgot to write the mem. Now for some reason nothing happens when i plug in the console cable and try to connect. My other switch it works right away when I plug in the console cable.
what do you mean you forgot to write the mem? do you mean run the "write memory" command? if so, that does not affect the switch booting and outputting data over the serial console port. Are you using the serial console port? did you try to flash or update the switch?
 
  • Like
Reactions: alex1002

alex1002

Member
Apr 9, 2013
519
19
18
what do you mean you forgot to write the mem? do you mean run the "write memory" command? if so, that does not affect the switch booting and outputting data over the serial console port. Are you using the serial console port? did you try to flash or update the switch?
I'm trying to connect via the serial port and won't connect. Screen stays blank.

Sent from my Pixel 2 using Tapatalk