Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

igotserved

New Member
Sep 2, 2020
4
0
1
Spent the last 3 days going through this monster of a thread (and still only 2/3 way through). I think I will pull the trigger soon on a 6450-24p but have a question:
Currently my main core switch is a Netgear 24 port managed switch with 2 ports trunked to Unifi 8-port POE switches (for locations with limited ethernet ports). I also have 3 VLANs managed by pfsense. If I replace the Netgear switch with the 6450-24p, and enable L3 switching, do I also need to replace the Unifi switches, since they're only L2? In other words, must the network consist of all L3 switches for proper VLAN routing?
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
If I replace the Netgear switch with the 6450-24p, and enable L3 switching, do I also need to replace the Unifi switches, since they're only L2? In other words, must the network consist of all L3 switches for proper VLAN routing?
No, it's not necessary, you can route through VLANs on the 6450, and the other switches simply relay traffic on the trunk ports. It's a common architecture on a "collapsed core campus network" (access on L2, trunking on L3 core routing).
Be aware that pfSense cannot lease out DHCP addresses on VLANs not directly connected to itself AFAIK, if that's a thing for you.
 
Last edited:

Jorge Perez

Active Member
Dec 8, 2019
104
44
28
Just FYI, the stock fan has the pinout changed. That might be why swapping the fan results in an fan failure status.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Mine seems to halt right after

..CPSS DxCh Version: cpss3.4p1 release

I let it run for several hours, nothing after the line above. Rebooted the switch and it pauses in the same location. It would appear as though something is not recreating the config files and it cannot proceed forward into the console.
if there's no config files it doesn't attempt to recreate any, it won't write a config until you configure the switch somehow and do "write mem". It seems like it's freezing up when it's trying to initialize the ASIC which is not a good sign (CPSS is Marvell's ASIC driver/SDK interface, Core Prestera Software Suite, and the " DxCh" is the line of prestera ASIC in the switch) - did you already flash the latest bootloader and OS image from the bootloader? If you've done that succesfully, and it's still doing that, I would probably return it. As a last hope you could try a different power supply, or if it has two, plug both in, but I'd be surprised if it was that
 

acpatel

New Member
Sep 8, 2020
5
0
1
So can one put any of these fans into a ICX7250? I read somewhere in the thread that there was an issue with a minimum RPM for the 7250? For example -- could one use the Comair Rotron "Gryphon" GDA4028-12BB in the ICX7250, or would it be too slow?

So in order...
- the best fans for this are unobtanium due to COVID-19, but don't use Noctua. Noctua is overpriced junk - it has the lowest flow rate and lowest static pressure. Anything from Sunon or NMB or Nidec or Delta is fine. They also cost far less.
- 08.0.30t for the 6450 is the maximum possible
- see first post for that, but if the switch was only configuration reset and not factory reset it may have them installed. Do a show licenses first always.

Edit: to expound upon my 'unobtanium' fan choice, bear in mind, I have been doing systems integration for 30 years and have VERY extensive rackmount equipment experience. I know why Brocade chose the fans they did, I have worked with those fans, and they are the same fans I would have chosen. (BTW, at speed 2 the 6450-40P is attempting to move 90.9CFM. Yes, each of those 40mm fans is rated for 30.3CFM free air.)
If you want to quiet down these units, the BIGGEST change you can make is to remove the stamped grill with a Dremel. It's all sharp edges, and significantly blocks the hub as well. Turbulence from things like grills are what generate serious noise. If you still need protection, tack on a wire grill on the OUTSIDE of the chassis. HOWEVER, this will result in slightly reduced cooling to the front right corner. I won't bother going into my unobtanium choice, because unless you're ready to order at least 250 of them, you can't buy them.

So here's some fans you can buy that I'd recommend.
  • Delta EFB0412VHD-F00 - $12.40/ea @ Digikey
    There's a REASON Delta's a top pick for ODM and OEM. 40x40x20mm, 10.1CFM, 0.416in H2O (which is insane,) 32.5dBA @ 1m, rated for 70k hours at 50C. Make sure it's THAT part number and not the Rev C, which is a vastly inferior part.
  • Comair Rotron "Gryphon" GDA4028-12BB - discontinued, alas
    If you can find these? BUY THESE. 40x40x28mm, but 11CFM, 0.34in H2O, 8800RPM, but just 31.4dBA! I am still mad they discontinued them. No, I am not selling any of my spares.
  • Mechatronics MR4020E12B1-RSR - $7.50 @ Digikey (when in stock)
    READ THE PAGE. These are non-stocked currently! However these offer excellent balance between flow and noise. 40x40x20mm, 15.8CFM, 0.45in H2O, and 39dBA @ 1m at 11000RPM. (Yes, it's about the quietest 11k fan ever made.) You need the E12B1 though - NOT the B2. The B1 has a tach, the B2 is a rotor lock wire.
  • Mechatronics MR4020H12B1-RSR - about $8
    The 'slightly slower' version pushing 13.6CFM at just 35.1dBA. Note that you cannot use the B1+6 that DigiKey stocks. The B1+6 is a 4-wire PWM, and the ICX's 12V control will burn up the motor.
  • Mechatronics MH4028L12B1-RSR - PLEASE TELL US ALL IF YOU FIND A SOURCE!
    These. Are. GLORIOUS. 40x40x28mm, 12.81CFM, 0.37in H2O, 39dBA, I just LOVE these fans. But nobody stocks the low speed version. And the next step up (the M) is already over 45dBA. But they have amazing harmonics due to the 4028L being less than half the speed the frame was designed for (Max 16,500RPM!)
  • NMB-MAT 1608VL-04W-B69-B00 - $10.47 @ Digikey
    These are an excellent 'middle of the middle' choice. 40x40x20mm, 11.3CFM, 0.399in H2O, 34.6dBA @ 1m at 9500RPM, and 40k hour lifetime. They're also generally stocked by multiple vendors. Really great fans for general use, but believe me, that 40k hour lifetime is near spot on - expect to replace every 3 years or so.
  • Sunon PSD1204PHB1-A(2).Z.F.PWM.GN "Tiny Terror" - also currently unobtainable
    These are new design MagLev parts, and really impressive. 40x40x15mm (so the thinnest here,) but 14CFM, 0.63in H2O, and 44.2dBA @ 1m at 12,000RPM. They're also a LOT easier than the Mechatronics; Sunon MOQ is just 30 for a non-custom part, expect around $10-12/ea. These are awesome fans if you can get your hands on them and can stand the noise.
And while we're here, let's talk about the Sunon KDE MagLevs. Which will be a very short talk: KDE MagLevs are terrible for this application. They're cheap, plentiful, and useless. They are NOT designed for applications like this. The KDE1204PKVX that you can find anywhere and everywhere? It is objectively the worst. 10.8CFM, 27.5dBA, sounds great, right? It also has a static pressure of 0.27in H2O. These were designed for very free flow applications. The ICX chassis grills are the opposite.
I also don't recommend SanAce because the PSUs you all complain about on the 6610's? Those are SanAce fans. SanAces have a very distinct harmonic that everyone hates. That harmonic is also why SanAces perform the way they do. You can't have one without the other. If you think those 6610's are bad, at least there is much, much worse.

As far as selecting fans? The closer to stock flow, the better, however you achieve it. Installing 10CFM peak fans in an ICX6450-48 or 48P is a net cooling reduction of over 70%! So, you know, don't do that unless you want to run very hot or are not using POE! This is why I make angry noises at people who think they can just swap fans around in equipment like this. For the love of god people, the ICX6610 fan modules? 2x Delta FFB0412UHN's per module, that's over 50CFM per module, over 100CFM excluding the power supplies! Nobody puts that kind of brute force in unless they NEED it. Especially as those FFBs are close to $20/each at quantity.
But as I said: how you get there doesn't matter as long as you get there or accept the trade-off. Ducting a pair of 45CFM 120mm fans on the side of your 6450-48P? That'd work too. No, seriously! (Won't work on the exhaust side because you just don't have enough opening and it's too much pressure loss.) Add more, quieter fans? Also valid. Strap tiny quiet fans to the heatsinks? Absolutely will help. Only care about performance at all costs? Throw in some contra-rotating fans! Want to get stock cooling performance out of a 6450-48? Cut out 2 more fan holes, drop in 3 x 9CFM, dead silent and ready to rock.
 

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
So can one put any of these fans into a ICX7250? I read somewhere in the thread that there was an issue with a minimum RPM for the 7250? For example -- could one use the Comair Rotron "Gryphon" GDA4028-12BB in the ICX7250, or would it be too slow?
There IS a minimum RPM floor on all models for alerting, and the 7000's will refuse to boot. (The 6000's will just yell at you.) The factory fan for the ICX7250 is most likely the same W40S or is 'very similar.' The RPM window is tuned somewhat broadly so that Brocade can change fan suppliers if necessary without needing to have fan-specific software.
And the W40S is why I would KILL to be able to change the FAN RPM expectations in the OS.

The Nidec W40S is rated for 25,000RPM at 12V with a peak that could reach 29,000RPM. It is not just 'high speed.' It was literally the highest speed 40mm fan period for quite some time. Sanyo Denki just introduced the 9HVA at 38,000RPM this year. (We don't count the 9CRH I use because it's a counter-rotating fan.)
But at 25,000RPM, we have options. The Nidec W40S, SanAce 9GAX, and Delta FFB0412. So we can reasonably assume the engineers tuned the window to accommodate these three fans. Especially as we have already observed the SanAce 9GAX in the ICX. This means that an ICX originally equipped with 'screamers' will have a VERY specific RPM window - a minimum of 11,500+-500 (or 1000) at 6V, and an overspeed trigger at around 26,500RPM.
Mind you, I am basing my assumption here on the 7250 using substantially the same fan as the 6450. Which would make the most sense from a sourcing, manufacturing, and code stability standpoint. (7450's and others with fan modules, totally different expectations. Especially as they use counter-rotating fans.) So 1 or 3 fans with 25kRPM @ 12V, and you need to hit 11kRPM at 6V.

Unfortunately that means the ONLY way you can get there is 1) faking the RPMs completely 2) connecting to a constant 12V supply with a 12kRPM fan and hoping it never asks for speed 2. Speed 2 will go into fan failure mode immediately.
This also, by the way, informs us as to what signals a tach faker needs to put out and what components. And it is a doozy - you need 50KHz square wave at about +-2.5% to prevent it reading fan failure.
 
  • Like
Reactions: tommybackeast

tommybackeast

Active Member
Jun 10, 2018
286
105
43
There IS a minimum RPM floor on all models for alerting, and the 7000's will refuse to boot. (The 6000's will just yell at you.) The factory fan for the ICX7250 is most likely the same W40S or is 'very similar.' The RPM window is tuned somewhat broadly so that Brocade can change fan suppliers if necessary without needing to have fan-specific software.
And the W40S is why I would KILL to be able to change the FAN RPM expectations in the OS.

The Nidec W40S is rated for 25,000RPM at 12V with a peak that could reach 29,000RPM. It is not just 'high speed.' It was literally the highest speed 40mm fan period for quite some time. Sanyo Denki just introduced the 9HVA at 38,000RPM this year. (We don't count the 9CRH I use because it's a counter-rotating fan.)
But at 25,000RPM, we have options. The Nidec W40S, SanAce 9GAX, and Delta FFB0412. So we can reasonably assume the engineers tuned the window to accommodate these three fans. Especially as we have already observed the SanAce 9GAX in the ICX. This means that an ICX originally equipped with 'screamers' will have a VERY specific RPM window - a minimum of 11,500+-500 (or 1000) at 6V, and an overspeed trigger at around 26,500RPM.
Mind you, I am basing my assumption here on the 7250 using substantially the same fan as the 6450. Which would make the most sense from a sourcing, manufacturing, and code stability standpoint. (7450's and others with fan modules, totally different expectations. Especially as they use counter-rotating fans.) So 1 or 3 fans with 25kRPM @ 12V, and you need to hit 11kRPM at 6V.

Unfortunately that means the ONLY way you can get there is 1) faking the RPMs completely 2) connecting to a constant 12V supply with a 12kRPM fan and hoping it never asks for speed 2. Speed 2 will go into fan failure mode immediately.
This also, by the way, informs us as to what signals a tach faker needs to put out and what components. And it is a doozy - you need 50KHz square wave at about +-2.5% to prevent it reading fan failure.
The idea of a small fan moving at 38,000 RPM is not something I can wrap my mind around.
 
  • Like
Reactions: kache

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
So I figured out my OSPF shenanigans (result, bug in the version of the FRR I was using on the non-ICX side. Go figure.) So now I've got a real challenge for you Brocade experts.
And I do mean a real challenge. Because reading the manuals, I don't know if this is possible.

I have a very obnoxious topology requirement due to my provider being about as competent with networking as a dog is with embroidery. And that's probably still overstating their capabilities. To deal with these thumb-lacking buffoons, I HAVE to apply some very interesting grease that I'm sure many of you will find perplexing. I assure you, this is less strange and far more common than you think. Just not typically in this scenario because normally companies hire someone intelligent enough to configure the goddamn CPEs correctly if only to avoid service calls.

SO! What is the problem? I need to create 'special' VLANs, and these 'special' VLANs cannot present or show any MAC except specifically configured ones. And when I say they cannot show anything? I mean ANYTHING. No switchport MACs, no switch MACs, only specifically configured MACs. Which wouldn't be so bad except I don't know the MACs on the DE and they change. (Seriously, if these people had thumbs, they'd try to pick their noses with them.) Now, on JunOS, this is actually pretty damn easy. I just make a bi-directional source MAC firewall. No more leak.

Which it seems like Brocade just cannot do. Because it literally has no concept whatsoever of someone wanting to filter multiple MACs. Even worse, when I apply the mac filter to the interface, it seems to be 'last-in wins' and still leaks the Brocade's MAC even with no ve defined. Is there any way to actually do what I need with Brocade's MAC filtering at all or am I going to have to move a bunch of ports back to JunOS?
 

sfrode

New Member
Mind you, I am basing my assumption here on the 7250 using substantially the same fan as the 6450. Which would make the most sense from a sourcing, manufacturing, and code stability standpoint. (7450's and others with fan modules, totally different expectations. Especially as they use counter-rotating fans.) So 1 or 3 fans with 25kRPM @ 12V, and you need to hit 11kRPM at 6V.
The 7250-48 (non-POE) happily accepts fans that do 2400 RPM at 4,5v. However, the fans I have at at that speed do not have sufficient airflow to cool the switch chip properly, which means that the switch will go to fan speed 2 from time to time. A hack like the one described in post #3671 solved that problem.
 

MoMeanMugs

Member
Apr 16, 2018
60
19
8
74
I tried doing a search in the thread (and also on Google), but I'm coming up short. I acquired another ICX6650, but it has the intake fans and power supplies instead of the exhaust. Is it possible to reverse the fans to make them exhaust? I'm guessing the way the switch knows what modules are plugged in is through resistors and that those would need to be changed as well.
 

klui

Well-Known Member
Feb 3, 2019
824
453
63
SO! What is the problem? I need to create 'special' VLANs, and these 'special' VLANs cannot present or show any MAC except specifically configured ones. And when I say they cannot show anything? I mean ANYTHING. No switchport MACs, no switch MACs, only specifically configured MACs. Which wouldn't be so bad except I don't know the MACs on the DE and they change. (Seriously, if these people had thumbs, they'd try to pick their noses with them.) Now, on JunOS, this is actually pretty damn easy. I just make a bi-directional source MAC firewall. No more leak.
Good to hear you've solved your original issue. That is interesting and not a use case I've had to deal with. Indeed it sounds like a 0.1%er. Could you post a general config required to do this on JunOS?

Thanks!
 

acpatel

New Member
Sep 8, 2020
5
0
1
OK. Here's what's confusing me. @sfrode put low speed fans into a 7250-48 without issues.
The 7250-48 (non-POE) happily accepts fans that do 2400 RPM at 4,5v. However, the fans I have at at that speed do not have sufficient airflow to cool the switch chip properly, which means that the switch will go to fan speed 2 from time to time. A hack like the one described in post #3671 solved that problem.
Similarly, in post #256375 , @RoachedCoach reports successfully replacing the fans in a ICX7250-48P with the MR4020X12B1-RSR -- which is a 12,000 RPM fan per the datasheet.

@neb50 replaced the fans with KDE1204PKVX -- 8,200 RPM fan. post #1320

But @rootwyrm reports problems with lower speed fans. As does @kyled although it looks like a different problem because his switch would boot, then the fans would stop spinning, which seems really bad.

How can all of these things be true at once / what is actually happening here? Is it working because of the additional fan to cool the ASIC --> not needing to go into speed 2? Doesn't seem so from @sfrode 's post, but it's all very unclear to me.

Is it the pinout post #254785 ?

@vangoose and @camper8080 report average fan speeds of 6500 (7250-24P) and ~7300 (7250-48P). post #251590

@fohdeesha reports fan 2 in a 7250 as ~6500 post #218240

What speed is truth and threshold? What can I replace the fans in a 7250-48 with?

(yes, I realize that most of the post numbers are board-wide and not thread, but don't want to go back and fix them)
 
Last edited:

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
Good to hear you've solved your original issue. That is interesting and not a use case I've had to deal with. Indeed it sounds like a 0.1%er. Could you post a general config required to do this on JunOS?

Thanks!
The 'one side unknown' part is a 0.1% scenario sort of - affects ~800,000 people. (No, seriously. Most just don't know or care.) But multiple MAC filtering is a VERY common scenario, especially if there's a downstream switch or you're dealing in virtualization. Especially virtualization. And on JunOS the configuration is very simple and straightforward:

Code:
family ethernet-switching {
    filter moron-trap {
        term 1 {
            from {
                source-mac-address {
                    00:9c:fc:a0:b0:01/48;
                    00:9c:fc:46:57:00/48;
                    00:e0:7b:00:00:00/24;
                }
                vlan moron;
            }
            then {
                accept;
                count firewall-mac-gateway;
            }
        }
        term 2 {
            then discard;
        }
    }
}
This forces any traffic from the listed MAC addresses onto the VLAN "moron" (yes I'm annoyed with having to do this why do you ask?) and discards all frames - not just packets, frames - not from one of the listed source-mac-address. Then on the VLAN itself:

Code:
vlans {
    moron {
        description "Strict isolation VLAN because morons"
        vlan-id 1010;
        no-local-switching;
        vlan-prune;
        filter {
            input moron;
            output moron;
        }
}
edit: Forgot to mention, the 'untagged' port ALSO requires "arp-resp restricted". Juniper explains that better.

That 'no-local-switching' is no small part of the magic; it specifies that access ports in this VLAN do not forward to each other, which is then combined with 'vlan-prune' so that the internal logic (VCP) uses the shortest possible path for broadcast, multicast, and unknown unicast (e.g. DHCP) to the egress interface. Meaning ge-0/0/0 will go direct to ge-0/0/10 without traversing the routing plane or being broadcast to any other port or device, assuming the target is directly connected to ge-0/0/10. And because a VLAN is specified, those packets will be tagged with the moron VLAN always. Then you just add the moron VLAN to whatever ports need to access it, and, that's it. It's really that simple.

What happens when you DON'T filter the MACs? Well, then the DE sees too many MAC addresses, starts adding them to the table, gets confused because of an asymmetric configuration, and doesn't correctly update the learning table on the distant end. And now you're ****ed with L2 asymmetry and nothing works.
And the fix I keep pushing them on? It won't work with the Brocade in it's current state. It does RARP and put traffic on the wire with it's own MAC even without a ve on the VLAN. Which immediately slams into the learning table's configured limit on one side, so subsequent devices run into the asymmetry issue.
 
Last edited:

tommybackeast

Active Member
Jun 10, 2018
286
105
43
So I figured out my OSPF shenanigans (result, bug in the version of the FRR I was using on the non-ICX side. Go figure.) So now I've got a real challenge for you Brocade experts.
And I do mean a real challenge. Because reading the manuals, I don't know if this is possible.

I have a very obnoxious topology requirement due to my provider being about as competent with networking as a dog is with embroidery. And that's probably still overstating their capabilities. To deal with these thumb-lacking buffoons, I HAVE to apply some very interesting grease that I'm sure many of you will find perplexing. I assure you, this is less strange and far more common than you think. Just not typically in this scenario because normally companies hire someone intelligent enough to configure the goddamn CPEs correctly if only to avoid service calls.

SO! What is the problem? I need to create 'special' VLANs, and these 'special' VLANs cannot present or show any MAC except specifically configured ones. And when I say they cannot show anything? I mean ANYTHING. No switchport MACs, no switch MACs, only specifically configured MACs. Which wouldn't be so bad except I don't know the MACs on the DE and they change. (Seriously, if these people had thumbs, they'd try to pick their noses with them.) Now, on JunOS, this is actually pretty damn easy. I just make a bi-directional source MAC firewall. No more leak.

Which it seems like Brocade just cannot do. Because it literally has no concept whatsoever of someone wanting to filter multiple MACs. Even worse, when I apply the mac filter to the interface, it seems to be 'last-in wins' and still leaks the Brocade's MAC even with no ve defined. Is there any way to actually do what I need with Brocade's MAC filtering at all or am I going to have to move a bunch of ports back to JunOS?
I am a network noob; and have absolutely no understanding on what you wrote technically;

I am merely replying to compliment the way you paint with words.
 

sfrode

New Member
Just for the sake of completeness since there might be differences between firmware releases:

Code:
SSH@swcore02#sh ver
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on Jun 25 2020 at 11:04:01 labeled as SPR08092d
      (32179532 bytes) from Primary SPR08092d.bin (UFI)
        SW: Version 08.0.92dT213
      Compressed Primary Boot Code size = 786944, Version:10.1.17T215 (spz10117)
       Compiled on Wed Oct  9 10:08:27 2019

  HW: Stackable ICX7250-48
==========================================================================
UNIT 1: SL 1: ICX7250-48 48-port Management Module
      Serial  #:SERIAL_REMOVED
      Software Package: ICX7250_L3_SOFT_PACKAGE   (LID: fwlINGFrFfJ)
      Current License: l3-prem-8X10G
      P-ASIC  0: type B344, rev 01  Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8192 KB boot flash memory
2048 MB code flash memory
2048 MB DRAM
STACKID 1  system uptime is 4 day(s) 7 hour(s) 28 minute(s) 59 second(s)
The system started at 15:05:49 CEST Sat Sep 05 2020

The system : started=warm start   reloaded=by "reload"
Code:
SSH@swcore02#dm fan-speed
Fan 1 Speed at 2551 RPM.
Fan 2 Speed at 2547 RPM.
Code:
SSH@swcore02#sh chassis
The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 66.3 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->98       deg-C
                Speed 2:       67<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 66.3 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Ambient temperature at 25C (77F), ASIC fan hack deployed.

How can all of these things be true at once / what is actually happening here? Is it working because of the additional fan to cool the ASIC --> not needing to go into speed 2? Doesn't seem so from @sfrode 's post, but it's all very unclear to me.
I tried Noctua A4x20 FLX fans first. The 7250-48 will boot with them, but as soon as the fan speed management kicks in, the fans stop spinning probably due to too low voltage. When the fans stop spinning, the switch will state that at least one functioning fan must be present and initiate a reboot. That is the same result you get if you just unplug the fans (as expected).

The ASIC fan hack helps with cooling the chip since the low-speed fans do not have enough airflow to properly cool it by themselves at ~4,5v (AFAIK, have not measured it). Without the fan hack, the temperature will crawl up to 98C, the switch will initiate fan speed 2 (full 12v) and the temperature will quickly drop back down to the fan speed 1 threshold (67C) - and the cycle repeats itself.

What speed is truth and threshold? What can I replace the fans in a 7250-48 with?
I have Chiefly CC4028B12M fans in my 7250-48 for now. Low noise, but they have an annoying resonance and/or high-pitch noise. I'm waiting for a couple of Sunon fans that will hopefully have a better sound profile and work with the low voltage at fan speed 1.
 
Last edited:

klui

Well-Known Member
Feb 3, 2019
824
453
63
The 'one side unknown' part is a 0.1% scenario sort of - affects ~800,000 people. (No, seriously. Most just don't know or care.) But multiple MAC filtering is a VERY common scenario, especially if there's a downstream switch or you're dealing in virtualization. Especially virtualization.
.
.
.
edit: Forgot to mention, the 'untagged' port ALSO requires "arp-resp restricted". Juniper explains that better.

That 'no-local-switching' is no small part of the magic
.
.
.
Thanks for sharing! That's my challenge with JunOS. It's so powerful that unless I live it day-in and day-out, it's a challenge to get really good with it. It also exacerbated by documents describing similar features but: some apply to switches, some to firewalls, and things might be different depending on the installed firmware.
 

ske4za

Member
Feb 4, 2019
80
45
18
The 'one side unknown' part is a 0.1% scenario sort of - affects ~800,000 people. (No, seriously. Most just don't know or care.) But multiple MAC filtering is a VERY common scenario, especially if there's a downstream switch or you're dealing in virtualization. Especially virtualization. And on JunOS the configuration is very simple and straightforward:



Code:
family ethernet-switching {

    filter moron-trap {

        term 1 {

            from {

                source-mac-address {

                    00:9c:fc:a0:b0:01/48;

                    00:9c:fc:46:57:00/48;

                    00:e0:7b:00:00:00/24;

                }

                vlan moron;

            }

            then {

                accept;

                count firewall-mac-gateway;

            }

        }

        term 2 {

            then discard;

        }

    }

}


This forces any traffic from the listed MAC addresses onto the VLAN "moron" (yes I'm annoyed with having to do this why do you ask?) and discards all frames - not just packets, frames - not from one of the listed source-mac-address. Then on the VLAN itself:



Code:
vlans {

    moron {

        description "Strict isolation VLAN because morons"

        vlan-id 1010;

        no-local-switching;

        vlan-prune;

        filter {

            input moron;

            output moron;

        }

}


edit: Forgot to mention, the 'untagged' port ALSO requires "arp-resp restricted". Juniper explains that better.



That 'no-local-switching' is no small part of the magic; it specifies that access ports in this VLAN do not forward to each other, which is then combined with 'vlan-prune' so that the internal logic (VCP) uses the shortest possible path for broadcast, multicast, and unknown unicast (e.g. DHCP) to the egress interface. Meaning ge-0/0/0 will go direct to ge-0/0/10 without traversing the routing plane or being broadcast to any other port or device, assuming the target is directly connected to ge-0/0/10. And because a VLAN is specified, those packets will be tagged with the moron VLAN always. Then you just add the moron VLAN to whatever ports need to access it, and, that's it. It's really that simple.



What happens when you DON'T filter the MACs? Well, then the DE sees too many MAC addresses, starts adding them to the table, gets confused because of an asymmetric configuration, and doesn't correctly update the learning table on the distant end. And now you're ****ed with L2 asymmetry and nothing works.

And the fix I keep pushing them on? It won't work with the Brocade in it's current state. It does RARP and put traffic on the wire with it's own MAC even without a ve on the VLAN. Which immediately slams into the learning table's configured limit on one side, so subsequent devices run into the asymmetry issue.


I always enjoy this technical challenges, partly because I'm always running into edge cases myself, and partly because part of me enjoys pouring through well-written documentation. I've also had some experience on the other side with the process of writing documentation. Anyway, I don't have an ICX currently so I can't validate these commands but I did spend some time reading the documentation for the 08.9.90 version.



References:

08.0.90 Security Configuration Guide (MAC filtering)
08.0.90 Layer 2 Switching Configuration Guide (VLAN-based static MAC entries, private VLANs)
08.0.90 Layer 3 Routing Configuration Guide (disabling RARP, proxy ARP)



VLAN-based static MAC entries
So provided you knew what the MAC addresses are, you'd think you'd be able to do this:
You can configure a VLAN to drop packets that have a particular source or destination MAC address.
You can configure a maximum of 2048 static MAC address drop entries on a Ruckus device.

To configure a VLAN to drop packets with a source or destination MAC address of 0000.0063.67FF, enter the following commands.
Code:
device(config)# vlan 2 
device(config-vlan-2)# static-mac-address 0000.0063.67FF drop
But, this only works by statically dropping MAC addresses you don't want, which is cumbersome. And there's a limit of 2048. So not ideal.

Now the security guide has a section on MAC address filtering (pg 171):
In source or destination mac-address field, you can specify a particular mac-address with exact mask or a range of mac-addresses with a variable mask or specify any to match any mac-address. Specify the mask using f (ones) and zeros. For example, to match on the first two bytes of the address 0010.0075.3676, use the mask ffff.0000.0000. In this case, the filter matches on all MAC addresses that contain "0010" as the first two bytes. The filter accepts any value for the remaining bytes of the MAC address.

Code:
 device(config)# mac filter 1 deny 0010.0075.3676 ffff.0000.0000

Ok, now that sounds better. But, it appears that we can only assign it to interfaces:
Apply previously created MAC filters to an interface using the mac filter-group command.
Code:
device(config)# interface ethernet 1/1/1 
device(config-if-e1000-1/1/1)# mac filter-group 1 to 5 1024
Not quite the same as your example from JunOS. But if your VLAN is the only one on that specific interface, this could work? Also of note, your filter-groups have to be committed on one line (you can't do multiple).

MAC address filters do not filter Layer 2 control protocols. Layer 2 control protocols, such as STP and LACP, are
processed by the device even when a "Deny All" MAC address filter has been applied on the interface.
This is probably worth noting.

But wait, about dynamic MAC-based VLANs that can be based on Source MAC address authentication? Well:
Beginning in FastIron release 08.0.20, MAC-based VLAN features are replaced on most FastIron platforms by Flexible Authentication (flexauth) features.
Ok, so that starts on page 177 on the security guide. There's about 30 pages worth of explaining the different types of authentication, but later on we come across the same solution as above:
If any of the clients need to be statically authenticated or denied access, the MAC addresses of such clients can be
configured through MAC filters and applied on authentication-enabled ports as authentication filters.
Code:
device(config)# mac filter 1 permit/deny xxxx.xxxx.xxxx FFFF.FFFF.FFFF any
Cool. And then how do we auto assign it to the VLAN we want? Well we go into the interface and say what the default authenticated VLAN is.
Enter the authentication auth-default-vlan command to configure the authentication default VLAN (auth-default VLAN).
Code:
device(config-if-e1000-1/1/1)# authentication auth-default-vlan 30
(Optional) Allow tagged packet processing when the port is not tagged, which may be the case when multiple VMs are
connected to the port so that they can be authenticated with MAC authentication, and automatic tagging of the port
helps. This option is disabled by default.
Code:
device(config-if-e1000-1/1/1)# authentication allow-tagged
(Optional) Enter the auth-filter command to apply the specified filter on the interface so that the MAC addresses defined
in the filter (MAC filter) need not go through authentication.
Code:
device(config-if-e1000-1/1/1)# authentication auth-filter 2 4
Syntax: authentication auth-filter filter-id vlan-id
The source MAC addresses defined using the mac-filter command are considered pre-authenticated and are not
subject to authentication. A client can be authenticated in an untagged VLAN or tagged VLAN using the MAC address
filter. If the authentication filter has a tagged VLAN configuration, the clients are authenticated in the auth-default VLAN
and the tagged VLAN provided in the auth-filter. The clients authorized in the auth-default VLAN allow both untagged
and tagged traffic. The auth-filter is defined using the mac-filter command.


Disabling RARP
This seems pretty straight forward on page 32 of the Layer 3 guide.
RARP is enabled by default.
To disable RARP, enter the following command at the global CONFIG level.
Code:
device(config)# no ip rarp
You can also statically set RARP if you need to do so.



Restricted Proxy ARP
Judging by the Juniper link you sent, it seems pretty similar to the local proxy ARP (not global) that's available, on page 26 of the Layer 3 guide. But again, only on a interface:
Enable the proxy ARP command on the specified interface.
Code:
device(config-if-e10000-1/1/1)# ip proxy-arp enable
By default, gratuitous ARP is disabled for local proxy ARP.


'no-local-switching'
This reads like isolated private VLANs which start on page 316 of the Layer 2 guide.
The pvlan type command specifies that this port-based VLAN is a PVLAN and can be of the following types:

• community - Broadcasts and unknown unicasts received on community ports are sent to the primary port and also are
flooded to the other ports in the community VLAN.

• isolated - Broadcasts and unknown unicasts received on isolated ports are sent only to the primary port. They are not
flooded to other ports in the isolated VLAN.

• primary - The primary PVLAN ports are "promiscuous". They can communicate with all the isolated PVLAN ports and
community PVLAN ports in the isolated and community VLANs that are mapped to the promiscuous port.
Additionally, in the notes:
• Normally, in any port-based VLAN, the device floods unknown unicast, unregistered multicast, and broadcast packets in
hardware, although selective packets, such as IGMP, may be sent only to the CPU for analysis, based on the IGMP
snooping configuration. When protocol is enabled, or if PVLAN mappings are enabled, the Ruckus ICX device will flood
unknown unicast, and unregistered multicast packets in software. The flooding of broadcast or unknown unicast from
the community or isolated VLANs to other secondary VLANs will be governed by the PVLAN forwarding rules. The
switching is done in hardware and thus the CPU does not enforce packet restrictions.

• Ruckus ICX devices forward broadcast, unregistered-multicast, and unknown unicast traffic in hardware if PVLAN
mappings are enabled. When PVLAN mappings are enabled, multiple MAC entries for the same MAC do not appear in
the MAC table, instead all the MAC entries are learned in the primary VLAN.


Great! Except the documentation suggests this is available only on ICX 7xxx switches, and there's no indication that 6xxx series switches will work. However, going back a few versions to the documentation for 08.0.30 says it will, so we'll cross our fingers here.



So after all that, what's the conclusion? To me, based on what I've read, it sounds like you need to create a PVLAN that's isolated and identify the primary VLAN used to forward traffic out. Then, create a MAC address filter based on the range you want. Apply that filter to the correct interface, and optionally use the Flexible authentication. Enable local proxy ARP on your interface, and disable RARP.

Of course, I could be totally off-base here. I could've totally wasted my time if you went through this same documentation. Like I said I don't have an ICX here to test, but let me know if I was way off the mark, or said something to turn on a light bulb in your head.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
The 'one side unknown' part is a 0.1% scenario sort of - affects ~800,000 people. (No, seriously. Most just don't know or care.) But multiple MAC filtering is a VERY common scenario, especially if there's a downstream switch or you're dealing in virtualization. Especially virtualization. And on JunOS the configuration is very simple and straightforward:

Code:
family ethernet-switching {
    filter moron-trap {
        term 1 {
            from {
                source-mac-address {
                    00:9c:fc:a0:b0:01/48;
                    00:9c:fc:46:57:00/48;
                    00:e0:7b:00:00:00/24;
                }
                vlan moron;
            }
            then {
                accept;
                count firewall-mac-gateway;
            }
        }
        term 2 {
            then discard;
        }
    }
}
This forces any traffic from the listed MAC addresses onto the VLAN "moron" (yes I'm annoyed with having to do this why do you ask?) and discards all frames - not just packets, frames - not from one of the listed source-mac-address. Then on the VLAN itself:

Code:
vlans {
    moron {
        description "Strict isolation VLAN because morons"
        vlan-id 1010;
        no-local-switching;
        vlan-prune;
        filter {
            input moron;
            output moron;
        }
}
edit: Forgot to mention, the 'untagged' port ALSO requires "arp-resp restricted". Juniper explains that better.

That 'no-local-switching' is no small part of the magic; it specifies that access ports in this VLAN do not forward to each other, which is then combined with 'vlan-prune' so that the internal logic (VCP) uses the shortest possible path for broadcast, multicast, and unknown unicast (e.g. DHCP) to the egress interface. Meaning ge-0/0/0 will go direct to ge-0/0/10 without traversing the routing plane or being broadcast to any other port or device, assuming the target is directly connected to ge-0/0/10. And because a VLAN is specified, those packets will be tagged with the moron VLAN always. Then you just add the moron VLAN to whatever ports need to access it, and, that's it. It's really that simple.

What happens when you DON'T filter the MACs? Well, then the DE sees too many MAC addresses, starts adding them to the table, gets confused because of an asymmetric configuration, and doesn't correctly update the learning table on the distant end. And now you're ****ed with L2 asymmetry and nothing works.
And the fix I keep pushing them on? It won't work with the Brocade in it's current state. It does RARP and put traffic on the wire with it's own MAC even without a ve on the VLAN. Which immediately slams into the learning table's configured limit on one side, so subsequent devices run into the asymmetry issue.
I have to ask - WTF service are you using that is forcing you to do this? I'd like to have a word with their engineers. However you're correct, this isn't really what fastiron was built for. There's some similar-ish mechanisms like what @ske4za posted above, but it's only going to get you 70% of the way there in your application. The proper router/service provider line (NetIron) will do complex MAC filtering and assignment like this quite happily, but obviously we're not running NetIron. That's one of my biggest pet peeves with brocade was the harsh market segmentation between fastiron and netiron - stuff like this is missing from fastiron not because the ASICs aren't capable of it, but because they didn't bother to think their "access switch" line would ever need it. Juniper is *much* better about this
 
  • Like
Reactions: kache

sth

Active Member
Oct 29, 2015
379
91
28
Thank you for the answer!
So if I understood correctly you're using it as Core switch and the APs are connected directly to it rather than to the edge switches, correct?
yes - or they will be when I migrate everything over to it, probably this weekend as the family is away.