Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

klui

Well-Known Member
Feb 3, 2019
824
453
63
Hmmm... 802.1x enters the chat.

:)
How involved is it to implement 802.1x? Whenever I read supplicant and stuff like that it tells me I need to have an key infrastructure or else the set and forget will bite when certificates expire.
 

koifish59

Member
Sep 30, 2020
66
19
8
Hmmm... 802.1x enters the chat.
Yep! We've been using it for our wifi (WPA2-enterprise) with unifi APs. About to switch over to ruckus R750s for this. I'm going to implement 802.1x for stuff physically plugged it now. I'm still learning this swich, so I may visit this thread for help if needed :p

How involved is it to implement 802.1x? Whenever I read supplicant and stuff like that it tells me I need to have an key infrastructure or else the set and forget will bite when certificates expire.
We're using Active Directory and RADIUS server. Then just point the RADIUS server to anything that uses authentication, like APs or this switch. And of course, clients that can read 802.1x. It wasn't too hard to setup.

dhcp snooping + ip source guard IP Source Guard
TY! I'll look into this feature
 
  • Like
Reactions: klui

MrGuvernment

Member
Nov 16, 2020
39
7
8
Well, I am going the cheaper route for now and will just have to accept the 4 SFP+ ports and got a Brocade ICX6450 for $100 CAD from a local recycler. 2 links for my LAB box, was going to do 10 links for my firewall, but since the switch can do some VLAN work I can jut do a single link and then 1 more link to my main desktop.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
How involved is it to implement 802.1x? Whenever I read supplicant and stuff like that it tells me I need to have an key infrastructure or else the set and forget will bite when certificates expire.
Not too much of a pain if you have an AD domain and all client PCs are joined to the domain itself.

Deploy a GPO policy to enable authentication to the NICs, configure the switch to authenticate via RADIUS to the NPS service on the DC and hey presto you've effectively locked down your network. You can choose if you want to authenticate the user or the machine account to the network (and you can also dynamically assign VLANs based on the user account groups, if you want).

Do not enforce authentication on the uplink ports! :)

For sure is more manageable than a never ending list of MAC addresses. ;)
 
  • Like
Reactions: klui and koifish59

victimofareload

New Member
Nov 10, 2020
6
0
1
Well boys. I flew too close to the sun.

Got a ICX-6610-48P. Reasonable, ~$200 to my door from a friend of a friend.

I read the stats. I knew it was loud (It's got 2x rev B PSU's and 2x fan trays). I told myself, it's not that loud surely. $DAYJOB taught me what a loud server/switch is. I ignored the fine people here.. And I paid for it.

For reference, my gear is all located in a walk-in closet about 8 ft away from my desk behind a closed door.

My old switch was a Cisco 3750G 24port with POE. Which wasn't silent. But I can't hear it on the other side of the door.

Fast forward to the 6610. It's loud, sure. The cisco is loud too before the fans spin down. What I hadn't considered is them spinning back up.

The closet gets to almost 100F during the summer. And even now that it's cooler, about 83F. Even at 83F the 6610 runs quietly until it hits a CPU temp of 76C. Then it goes full balls-to-the-wall for about 20 seconds, Where it reaches under 71C which trips back into fan mode 1. Rinse and repeat every 10 minutes or so. Fan mode 1 is silent to me (Behind a door). Fan mode 2 is so loud I'll have to explain it on phone calls. Man if the 6610 had a middle fan speed between "audible" and "F18 on a vertical climb", it would be perfect.

So the cisco is back in for now. I'm guessing I'm going to throw the 6610 up on Ebay and try to make back what I've got into it. Putting my 10G upgrade plans on hold for now. I think I'll pickup a 7250-48P and give that a try.

Thanks to @fohdeesha on the licensing help!
 

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
You might want to check for proper airflow. My ICX6610-48-PE is in ambient up to about 80F and the switch temps never get above 63C during the hardest use I can throw at it. Besides POST, I've never had the fans get above level 1 speed. I have 2x Revision A PSUs.
 

victimofareload

New Member
Nov 10, 2020
6
0
1
You might want to check for proper airflow. My ICX6610-48-PE is in ambient up to about 80F and the switch temps never get above 63C during the hardest use I can throw at it. Besides POST, I've never had the fans get above level 1 speed. I have 2x Revision A PSUs.
Good point. I just took it apart and everything seems correct. No excessive dust. All the fans spin and are the correct orientation (rear exhaust). I did slightly reroute the power cable to the POE daughter board. It was sitting between what I believe is one of the larger switch chips (Large black Heatsinks) and the CPU (Smaller light colored heatsink near the center and the RAM. I moved the power cable so it was situated *above* the heatsink instead of behind it.

It's powered up now in the closet on the floor. So far max temp I've seen is 57.5C. I've also noticed that the Speed 1 max temp is now 84C, Where it was 76C previously. Assuming this had something to do with POE load, I added 4 SIP phones I had laying around (Ran about 6 POE devices earlier in todays test). But the extra POE devices didn't change the switching temp thresholds.

Only thing different from this morning and now is the amount of POE load (About half, 12W vs 20W). And the physical location (On a shelf), with a warm synology and 2x Lenovo tiny PC's on top of it. Guess I'll let it heat soak a bit and see where it gets. I seriously doubt by cable move and slight press on the heatsink made any difference.

Code:
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  xxx
        Firmware Ver:    B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 (AC - PoE) present, status ok
        Model Number:   23-0000142-02
        Serial Number:  xxx
        Firmware Ver:    B
Power supply 2 Fan Air Flow Direction:  Front to Back

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 57.5 deg-C

Fan speed switching temperature thresholds:
                Speed 1: NM<----->84       deg-C
                Speed 2:       79<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
        Current temperature : 42.5 deg-C
MAC 2 Temperature Readings:
        Current temperature : 48.5 deg-C
CPU Temperature Readings:
        Current temperature : 57.5 deg-C
sensor A Temperature Readings:
        Current temperature : 47.0 deg-C
sensor B Temperature Readings:
        Current temperature : 46.0 deg-C
sensor C Temperature Readings:
        Current temperature : 35.5 deg-C
stacking card Temperature Readings:
        Current temperature : 51.0 deg-C
        Warning level.......: 77.0 deg-C
        Shutdown level......: 87.0 deg-C
 

koifish59

Member
Sep 30, 2020
66
19
8
For reference, my gear is all located in a walk-in closet about 8 ft away from my desk behind a closed door.
Are you sure you want to stick it in a closed closet? It's dumping 100w+ so unless you have some way to exhaust that heat, it's going to be an oven in there
 

victimofareload

New Member
Nov 10, 2020
6
0
1
Are you sure you want to stick it in a closed closet? It's dumping 100w+ so unless you have some way to exhaust that heat, it's going to be an oven in there
Yeah, It's not ideal. But the cisco and the rest of the gear has lived through a few summers now. Max temp I've seen the room reach is 98F. I normally prop the door open when it's warm. Which lets cool air in (Office is about 74F normally). I did this earlier today, which made zero difference on the switch temp. But had the room as low as about 78F (Monitored with an ESP/Temp sensor).
 

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Your temps now are similar to my experience. One thing that I have noticed with my ICX6610 that may be common among some or all others.....

When "coldbooted" the CPU temps stay roughly 10C cooler until I do a reload/warmboot. It will be around 54C for months and then I warm boot and then suddenly it's at 63C consistently with no other changes in ambient conditions or load. If I pull the plugs, sit for a minute, and coldboot it again, back to 54C.

I'm a bit curious if you see anything similar.
 

victimofareload

New Member
Nov 10, 2020
6
0
1
Your temps now are similar to my experience. One thing that I have noticed with my ICX6610 that may be common among some or all others.....

When "coldbooted" the CPU temps stay roughly 10C cooler until I do a reload/warmboot. It will be around 54C for months and then I warm boot and then suddenly it's at 63C consistently with no other changes in ambient conditions or load. If I pull the plugs, sit for a minute, and coldboot it again, back to 54C.

I'm a bit curious if you see anything similar.
After soaking all night it didn't climb higher then 61C. I did a warm reload and it maintained temp.

So then I placed the brocade on top of the cisco. With the warm devices on top of it. Let it soak for awhile more. Still ~60C.

I just now created a trunk port between the two switches and have migrated over all the POE devices. After a few more hours I'll migrate the rest of them. So far it's holding ~60-61C like a champ. Without coming anywhere near fan mode 2.

Curiously, the mode 1 max temp is still 84c (Up from 76c yesterday morning). I still don't know what caused it to change... Or why the switch is running so much cooler today. I refuse to believe my poking and prodding of it's innards made any difference.
 

kapone

Well-Known Member
May 23, 2015
1,095
642
113
It's the 40gb ports/stacking card in the rear that gets really hot. Airflow is really poor in that area. Here's my chassis info:

Code:
The stack unit 1 chassis info:

Power supply 1 (AC - Regular) present, status ok
     Model Number:    23-0000144-01
    Serial Number:    091     
    Firmware Ver:      B
Power supply 1 Fan Air Flow Direction:  Front to Back
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present

Fan controlled temperature: 51.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->78       deg-C
        Speed 2:       73<-----> 87 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
MAC 1 Temperature Readings:
    Current temperature : 42.0 deg-C
CPU Temperature Readings:
    Current temperature : 43.5 deg-C
sensor A Temperature Readings:                                   
    Current temperature : 27.0 deg-C
sensor B Temperature Readings:
    Current temperature : 42.5 deg-C
sensor C Temperature Readings:
    Current temperature : 20.0 deg-C
sensor D Temperature Readings:
    Current temperature : 19.0 deg-C
stacking card Temperature Readings:
    Current temperature : 51.5 deg-C
    Warning level.......: 84.0 deg-C
    Shutdown level......: 87.0 deg-C
Boot Prom MAC : 748e.f8e9.7fac
Management MAC: 748e.f8e9.7fac
Notice that the stacking card is almost 8-10C hotter than the CPU. The switch will go from mode 1 to 2 based on ANY of these sensors.
 

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
The closet gets to almost 100F during the summer. And even now that it's cooler, about 83F. Even at 83F the 6610 runs quietly until it hits a CPU temp of 76C. Then it goes full balls-to-the-wall for about 20 seconds, Where it reaches under 71C which trips back into fan mode 1. Rinse and repeat every 10 minutes or so. Fan mode 1 is silent to me (Behind a door). Fan mode 2 is so loud I'll have to explain it on phone calls. Man if the 6610 had a middle fan speed between "audible" and "F18 on a vertical climb", it would be perfect.
This is because Brocade/Arris were basically lazy and there is no fan ramp.
ICX 6-series (probably 7 too, but I don't have one to check) has exactly two fan speeds. 6V, and 12V. Nothing between. No actual ramping. It does not even have basic logic. It is literally just a switch. At CPU threshold, it puts the fans to 12V. As soon as they're below threshold, it drops to 6V. So the CPU starts getting hot again. Lather, rinse, repeat. It is not capable of running the fans at any level between there which would fix the issue.
And the 6610 uses contra-rotating fans, which is literally the loudest possible configuration. (And the only way to move enough air in the chassis.) 7-series uses the same contra-rotating fans in some models. This is a contra-rotating fan:

And this is why you use a contra-rotating fan:

HOWEVER, the 6610 is a front-to-rear (NEBS) chassis. So if you want to get the temperatures down, you can increase extraction at the rear (ducting and extraction fans) or increase intake at the front (make the holes bigger, blocking ingest of hot air, directing more cold air.)
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
And this is why you use a contra-rotating fan:
Yes and no. Basically all the configurations that adopt two or more fans stacked toghether to increase static pressure will use counter-rotating rotors, as it will minimize the vortex effect (as the two units needs to have opposite pitched blade), helps straightening the flow, and it won't allow for "passive coupling" of the second fan, dragging and slowing down the air (it can be seen in the video, before the user applies "full power" thus blowing away the free fan) instead of adding more thrust/speed to it.
If you have ever seen one, this configuration is used also for multi-stage boat propellers, as the outer one rotates the other way around than the inner one.

BTW, most switches use this simple two-stages logic, basically those are small pizza-boxes that won't dump out much energy, and the lowest flow rate is basically adequate for most of the situations as the SoC can whitstand a wide temperature range while in operation; full speed is pretty much used "on emergercies only".
 
Last edited:

victimofareload

New Member
Nov 10, 2020
6
0
1
So as an update my switch has held ~62C all day now. I moved everything over a few hours ago. Guess we'll see how it does over the next few days.

It bothers me that I have zero idea why it got better. But it's working great for now.

Just replaced the batteries in my UPS. So barring a hurricane (Central Florida) or other extended power outage, this thing won't be reloaded for a few years... hahaha.
 

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
Glad you got the temps back to normal. I do hope you find a way to keep the closet cooler - perhaps an exhaust fan fitted in the ceiling to the attic and a small grill on the bottom of the door for cool air ingress? Your switch and other devices in your closet may not work better because of it, but they may last longer and avoid further cooling noise.
 

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
2 questions in this post.

1) I could use some help with QoS, due to limited knowledge on the subject thus far.

I have VoIP and security camera traffic that I want to ensure gets high priority on the network. Not sure if SAN/NAS traffic should get afforded any special QoS, so comments on that welcome.

Anyway, how do I go about setting and enforcing QoS for VoIP and streaming camera traffic on my network? Each are seperated into their own VLAN and subnet:

VoIP - VLAN 2 - 10.1.2.0/24
Cams - VLAN 3 - 10.1.3.0/24
SAN - VLAN 4 - 10.1.4.0/24

I'm having trouble understanding where, how, and what to apply to correctly apply QoS shaping.

2) Is there any benefit to using the Switch image over the Router image on an ICX6450 doing only switch duty?
 
Last edited:

ArmedAviator

Member
May 16, 2020
91
56
18
Kansas
How's this for a start on QoS settings?

I have VoIP and streaming security cameras I want to give priority across the network. Is this looking correct so far?

This is on my ICX6450 setup as an access switch.
Code:
ip access-list extended markqos
permit tcp 10.1.3.0 0.0.0.255 eq 9000 any dscp-marking 32 
permit tcp any 10.1.3.0 0.0.0.255 eq 9000 dscp-marking 32
permit tcp any any eq 554 dscp-marking 32
permit tcp any eq 554 any dscp-marking 32
permit udp any any range 6970 6979 dscp-marking 32
permit udp any range 6970 6979 any dscp-marking 32
permit tcp any any eq 5060 dscp-marking 24
permit tcp any eq 5060 any dscp-marking 24
permit udp any eq 5060 any dscp-marking 24
permit udp any any eq 5060 dscp-marking 24
permit udp 10.1.2.0 0.0.0.255 range 10000 20000 any dscp-marking 46
permit udp any 10.1.2.0 0.0.0.255 range 10000 20000 dscp-marking 46
permit ip any any
Now do I add this ACL to each interface?

On my ICX6610, which also has edge devices on it, do I add these rules to all ports as well, or add them to the ACLs already in use on the VEs?

On both switches, the trust dscp option should be set on the uplink ports only, correct?


Also going to mention that I switched my ICX6450 to the ICX64S switching-only firmware. It only required some minor configuration changes, but works as expected. I still don't think there's any benefits to doing so, except slightly less complicated configuration.
 
  • Like
Reactions: tommybackeast

Eru0194

New Member
Jun 13, 2019
11
2
3
Just got a 7150-C12P from ebay.
Code:
ICX7150-Boot>nand bad

Device 0 bad blocks:
  05a00000
  05b00000

ICX7150-Boot>nand info

Device 0: nand0, sector size 1024 KiB, Micron NAND 2GiB
  Page size       4096 b
  OOB size         224 b
  Erase size   1048576 b
  subpagesize     4096 b
  options     0x   10200
  bbt options 0x       0
Should I be concerned about the bad blocks? I'm guessing it's fine but it'd be nice to get a second opinion while I can still return it.

Everything seems to be working normally.