Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[klui]

Ah I see. So I just had the order wrong? So have all mac address block as the last filter rule, and preceding filter rules are "allow" for each mac address?

Is there a more efficient way to do this than to manually make a new allow filter rule for each mac address?

Are you concerned with manually entering every MAC address to your filter list? Yeah, that will be a hassle but it's your use case.

You can inspect all MACs on your network and figure out their common OUIs (first 3) and maybe some of their remainder octets and use a wildcard or forgo filtering it in this way and use rules from your firewall/DHCP server instead. You may have more flexibility with not using MAC filters as in the video that device only supports 64, but I assume you can reconfigure a global setting to have more.

 

[koifish59]

Are you concerned with manually entering every MAC address to your filter list? Yeah, that will be a hassle but it's your use case.

You can inspect all MACs on your network and figure out their common OUIs (first 3) and maybe some of their remainder octets and use a wildcard or forgo filtering it in this way and use rules from your firewall/DHCP server instead. You may have more flexibility with not using MAC filters as in the video that device only supports 64, but I assume you can reconfigure a global setting to have more.

This is for small business and we have a new policy to block all devices by default and only allow a couple hundred company computers to be on our network. I thought about doing it via DHCP server, but that only prevents users from pulling an IP address. It doesn't stop them from setting up a static IP on their device. I'll look into firewall rules first.

I think Terry said there is a max of 512 mac filters, but that's for the ICX 7000 switches (hoping this ICX 6610 has the same max)

 

[fohdeesha]

This is for small business and we have a new policy to block all devices by default and only allow a couple hundred company computers to be on our network. I thought about doing it via DHCP server, but that only prevents users from pulling an IP address. It doesn't stop them from setting up a static IP on their device. I'll look into firewall rules first.

I think Terry said there is a max of 512 mac filters, but that's for the ICX 7000 switches (hoping this ICX 6610 has the same max)

dhcp snooping + ip source guard IP Source Guard

 

[infoMatt]

This is for small business and we have a new policy to block all devices by default

Hmmm... 802.1x enters the chat.

 

[klui]

Hmmm... 802.1x enters the chat.

How involved is it to implement 802.1x? Whenever I read supplicant and stuff like that it tells me I need to have an key infrastructure or else the set and forget will bite when certificates expire.

 

[koifish59]

Hmmm... 802.1x enters the chat.

Yep! We've been using it for our wifi (WPA2-enterprise) with unifi APs. About to switch over to ruckus R750s for this. I'm going to implement 802.1x for stuff physically plugged it now. I'm still learning this swich, so I may visit this thread for help if needed

How involved is it to implement 802.1x? Whenever I read supplicant and stuff like that it tells me I need to have an key infrastructure or else the set and forget will bite when certificates expire.

We're using Active Directory and RADIUS server. Then just point the RADIUS server to anything that uses authentication, like APs or this switch. And of course, clients that can read 802.1x. It wasn't too hard to setup.

dhcp snooping + ip source guard IP Source Guard

TY! I'll look into this feature

 

[MrGuvernment]

Well, I am going the cheaper route for now and will just have to accept the 4 SFP+ ports and got a Brocade ICX6450 for $100 CAD from a local recycler. 2 links for my LAB box, was going to do 10 links for my firewall, but since the switch can do some VLAN work I can jut do a single link and then 1 more link to my main desktop.

 

[infoMatt]

How involved is it to implement 802.1x? Whenever I read supplicant and stuff like that it tells me I need to have an key infrastructure or else the set and forget will bite when certificates expire.

Not too much of a pain if you have an AD domain and all client PCs are joined to the domain itself.

Deploy a GPO policy to enable authentication to the NICs, configure the switch to authenticate via RADIUS to the NPS service on the DC and hey presto you've effectively locked down your network. You can choose if you want to authenticate the user or the machine account to the network (and you can also dynamically assign VLANs based on the user account groups, if you want).

Do not enforce authentication on the uplink ports!

For sure is more manageable than a never ending list of MAC addresses.

 

[victimofareload]

Well boys. I flew too close to the sun.

Got a ICX-6610-48P. Reasonable, ~$200 to my door from a friend of a friend.

I read the stats. I knew it was loud (It's got 2x rev B PSU's and 2x fan trays). I told myself, it's not that loud surely. $DAYJOB taught me what a loud server/switch is. I ignored the fine people here.. And I paid for it.

For reference, my gear is all located in a walk-in closet about 8 ft away from my desk behind a closed door.

My old switch was a Cisco 3750G 24port with POE. Which wasn't silent. But I can't hear it on the other side of the door.

Fast forward to the 6610. It's loud, sure. The cisco is loud too before the fans spin down. What I hadn't considered is them spinning back up.

The closet gets to almost 100F during the summer. And even now that it's cooler, about 83F. Even at 83F the 6610 runs quietly until it hits a CPU temp of 76C. Then it goes full balls-to-the-wall for about 20 seconds, Where it reaches under 71C which trips back into fan mode 1. Rinse and repeat every 10 minutes or so. Fan mode 1 is silent to me (Behind a door). Fan mode 2 is so loud I'll have to explain it on phone calls. Man if the 6610 had a middle fan speed between "audible" and "F18 on a vertical climb", it would be perfect.

So the cisco is back in for now. I'm guessing I'm going to throw the 6610 up on Ebay and try to make back what I've got into it. Putting my 10G upgrade plans on hold for now. I think I'll pickup a 7250-48P and give that a try.

Thanks to @fohdeesha on the licensing help!

 

[ArmedAviator]

You might want to check for proper airflow. My ICX6610-48-PE is in ambient up to about 80F and the switch temps never get above 63C during the hardest use I can throw at it. Besides POST, I've never had the fans get above level 1 speed. I have 2x Revision A PSUs.

 

[victimofareload]

You might want to check for proper airflow. My ICX6610-48-PE is in ambient up to about 80F and the switch temps never get above 63C during the hardest use I can throw at it. Besides POST, I've never had the fans get above level 1 speed. I have 2x Revision A PSUs.

Good point. I just took it apart and everything seems correct. No excessive dust. All the fans spin and are the correct orientation (rear exhaust). I did slightly reroute the power cable to the POE daughter board. It was sitting between what I believe is one of the larger switch chips (Large black Heatsinks) and the CPU (Smaller light colored heatsink near the center and the RAM. I moved the power cable so it was situated *above* the heatsink instead of behind it.

It's powered up now in the closet on the floor. So far max temp I've seen is 57.5C. I've also noticed that the Speed 1 max temp is now 84C, Where it was 76C previously. Assuming this had something to do with POE load, I added 4 SIP phones I had laying around (Ran about 6 POE devices earlier in todays test). But the extra POE devices didn't change the switching temp thresholds.

Only thing different from this morning and now is the amount of POE load (About half, 12W vs 20W). And the physical location (On a shelf), with a warm synology and 2x Lenovo tiny PC's on top of it. Guess I'll let it heat soak a bit and see where it gets. I seriously doubt by cable move and slight press on the heatsink made any difference.

The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  xxx
        Firmware Ver:    B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  xxx
        Firmware Ver:    B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->84       deg-C
                Speed 2:       79<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 42.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 48.5 deg-C
CPU Temperature Readings:
        Current temperature : 57.5 deg-C
sensor A Temperature Readings:
        Current temperature : 47.0 deg-C
sensor B Temperature Readings:
        Current temperature : 46.0 deg-C
sensor C Temperature Readings:
        Current temperature : 35.5 deg-C
stacking card Temperature Readings:
        Current temperature : 51.0 deg-C
        Warning level.......: 77.0 deg-C
        Shutdown level......: 87.0 deg-C

 

[koifish59]

For reference, my gear is all located in a walk-in closet about 8 ft away from my desk behind a closed door.

Are you sure you want to stick it in a closed closet? It's dumping 100w+ so unless you have some way to exhaust that heat, it's going to be an oven in there

 

[victimofareload]

Are you sure you want to stick it in a closed closet? It's dumping 100w+ so unless you have some way to exhaust that heat, it's going to be an oven in there

Yeah, It's not ideal. But the cisco and the rest of the gear has lived through a few summers now. Max temp I've seen the room reach is 98F. I normally prop the door open when it's warm. Which lets cool air in (Office is about 74F normally). I did this earlier today, which made zero difference on the switch temp. But had the room as low as about 78F (Monitored with an ESP/Temp sensor).

 

[ArmedAviator]

Your temps now are similar to my experience. One thing that I have noticed with my ICX6610 that may be common among some or all others.....

When "coldbooted" the CPU temps stay roughly 10C cooler until I do a reload/warmboot. It will be around 54C for months and then I warm boot and then suddenly it's at 63C consistently with no other changes in ambient conditions or load. If I pull the plugs, sit for a minute, and coldboot it again, back to 54C.

I'm a bit curious if you see anything similar.

 

[victimofareload]

Your temps now are similar to my experience. One thing that I have noticed with my ICX6610 that may be common among some or all others.....

When "coldbooted" the CPU temps stay roughly 10C cooler until I do a reload/warmboot. It will be around 54C for months and then I warm boot and then suddenly it's at 63C consistently with no other changes in ambient conditions or load. If I pull the plugs, sit for a minute, and coldboot it again, back to 54C.

I'm a bit curious if you see anything similar.

After soaking all night it didn't climb higher then 61C. I did a warm reload and it maintained temp.

So then I placed the brocade on top of the cisco. With the warm devices on top of it. Let it soak for awhile more. Still ~60C.

I just now created a trunk port between the two switches and have migrated over all the POE devices. After a few more hours I'll migrate the rest of them. So far it's holding ~60-61C like a champ. Without coming anywhere near fan mode 2.

Curiously, the mode 1 max temp is still 84c (Up from 76c yesterday morning). I still don't know what caused it to change... Or why the switch is running so much cooler today. I refuse to believe my poking and prodding of it's innards made any difference.

 

[kapone]

It's the 40gb ports/stacking card in the rear that gets really hot. Airflow is really poor in that area. Here's my chassis info:

The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    091     
    Firmware Ver:      B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 51.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->78       deg-C
        Speed 2:       73<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 42.0 deg-C
CPU Temperature Readings:
    Current temperature : 43.5 deg-C
sensor A Temperature Readings:                                   
    Current temperature : 27.0 deg-C
sensor B Temperature Readings:
    Current temperature : 42.5 deg-C
sensor C Temperature Readings:
    Current temperature : 20.0 deg-C
sensor D Temperature Readings:
    Current temperature : 19.0 deg-C
stacking card Temperature Readings:
    Current temperature : 51.5 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C
Boot Prom MAC : 748e.f8e9.7fac
Management MAC: 748e.f8e9.7fac

Notice that the stacking card is almost 8-10C hotter than the CPU. The switch will go from mode 1 to 2 based on ANY of these sensors.

 

[dreamsin]

Brocade ICX-6450-48 48 Port Gigabit NETWORK L3 Switch with Rack Ears TESTED #2B | eBay

INCLUDES RACK EARS AND POWER CABLE. THIS IS NOT A POE SWITCH.

ebay.us

Great deal for ICX-6450-48P in the EU.

 

[rootwyrm]

The closet gets to almost 100F during the summer. And even now that it's cooler, about 83F. Even at 83F the 6610 runs quietly until it hits a CPU temp of 76C. Then it goes full balls-to-the-wall for about 20 seconds, Where it reaches under 71C which trips back into fan mode 1. Rinse and repeat every 10 minutes or so. Fan mode 1 is silent to me (Behind a door). Fan mode 2 is so loud I'll have to explain it on phone calls. Man if the 6610 had a middle fan speed between "audible" and "F18 on a vertical climb", it would be perfect.

This is because Brocade/Arris were basically lazy and there is no fan ramp.
ICX 6-series (probably 7 too, but I don't have one to check) has exactly two fan speeds. 6V, and 12V. Nothing between. No actual ramping. It does not even have basic logic. It is literally just a switch. At CPU threshold, it puts the fans to 12V. As soon as they're below threshold, it drops to 6V. So the CPU starts getting hot again. Lather, rinse, repeat. It is not capable of running the fans at any level between there which would fix the issue.
And the 6610 uses contra-rotating fans, which is literally the loudest possible configuration. (And the only way to move enough air in the chassis.) 7-series uses the same contra-rotating fans in some models. This is a contra-rotating fan:

And this is why you use a contra-rotating fan:

HOWEVER, the 6610 is a front-to-rear (NEBS) chassis. So if you want to get the temperatures down, you can increase extraction at the rear (ducting and extraction fans) or increase intake at the front (make the holes bigger, blocking ingest of hot air, directing more cold air.)

 

[infoMatt]

And this is why you use a contra-rotating fan:

Yes and no. Basically all the configurations that adopt two or more fans stacked toghether to increase static pressure will use counter-rotating rotors, as it will minimize the vortex effect (as the two units needs to have opposite pitched blade), helps straightening the flow, and it won't allow for "passive coupling" of the second fan, dragging and slowing down the air (it can be seen in the video, before the user applies "full power" thus blowing away the free fan) instead of adding more thrust/speed to it.
If you have ever seen one, this configuration is used also for multi-stage boat propellers, as the outer one rotates the other way around than the inner one.

BTW, most switches use this simple two-stages logic, basically those are small pizza-boxes that won't dump out much energy, and the lowest flow rate is basically adequate for most of the situations as the SoC can whitstand a wide temperature range while in operation; full speed is pretty much used "on emergercies only".

 

[victimofareload]

So as an update my switch has held ~62C all day now. I moved everything over a few hours ago. Guess we'll see how it does over the next few days.

It bothers me that I have zero idea why it got better. But it's working great for now.

Just replaced the batteries in my UPS. So barring a hurricane (Central Florida) or other extended power outage, this thing won't be reloaded for a few years... hahaha.