Well, I am going the cheaper route for now and will just have to accept the 4 SFP+ ports and got a Brocade ICX6450 for $100 CAD from a local recycler. 2 links for my LAB box, was going to do 10 links for my firewall, but since the switch can do some VLAN work I can jut do a single link and then 1 more link to my main desktop.
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Not too much of a pain if you have an AD domain and all client PCs are joined to the domain itself.
Deploy a GPO policy to enable authentication to the NICs, configure the switch to authenticate via RADIUS to the NPS service on the DC and hey presto you've effectively locked down your network. You can choose if you want to authenticate the user or the machine account to the network (and you can also dynamically assign VLANs based on the user account groups, if you want).
Do not enforce authentication on the uplink ports!
For sure is more manageable than a never ending list of MAC addresses.
Well boys. I flew too close to the sun.
Got a ICX-6610-48P. Reasonable, ~$200 to my door from a friend of a friend.
I read the stats. I knew it was loud (It's got 2x rev B PSU's and 2x fan trays). I told myself, it's not that loud surely. $DAYJOB taught me what a loud server/switch is. I ignored the fine people here.. And I paid for it.
For reference, my gear is all located in a walk-in closet about 8 ft away from my desk behind a closed door.
My old switch was a Cisco 3750G 24port with POE. Which wasn't silent. But I can't hear it on the other side of the door.
Fast forward to the 6610. It's loud, sure. The cisco is loud too before the fans spin down. What I hadn't considered is them spinning back up.
The closet gets to almost 100F during the summer. And even now that it's cooler, about 83F. Even at 83F the 6610 runs quietly until it hits a CPU temp of 76C. Then it goes full balls-to-the-wall for about 20 seconds, Where it reaches under 71C which trips back into fan mode 1. Rinse and repeat every 10 minutes or so. Fan mode 1 is silent to me (Behind a door). Fan mode 2 is so loud I'll have to explain it on phone calls. Man if the 6610 had a middle fan speed between "audible" and "F18 on a vertical climb", it would be perfect.
So the cisco is back in for now. I'm guessing I'm going to throw the 6610 up on Ebay and try to make back what I've got into it. Putting my 10G upgrade plans on hold for now. I think I'll pickup a 7250-48P and give that a try.
Thanks to @fohdeesha on the licensing help!
Got a ICX-6610-48P. Reasonable, ~$200 to my door from a friend of a friend.
I read the stats. I knew it was loud (It's got 2x rev B PSU's and 2x fan trays). I told myself, it's not that loud surely. $DAYJOB taught me what a loud server/switch is. I ignored the fine people here.. And I paid for it.
For reference, my gear is all located in a walk-in closet about 8 ft away from my desk behind a closed door.
My old switch was a Cisco 3750G 24port with POE. Which wasn't silent. But I can't hear it on the other side of the door.
Fast forward to the 6610. It's loud, sure. The cisco is loud too before the fans spin down. What I hadn't considered is them spinning back up.
The closet gets to almost 100F during the summer. And even now that it's cooler, about 83F. Even at 83F the 6610 runs quietly until it hits a CPU temp of 76C. Then it goes full balls-to-the-wall for about 20 seconds, Where it reaches under 71C which trips back into fan mode 1. Rinse and repeat every 10 minutes or so. Fan mode 1 is silent to me (Behind a door). Fan mode 2 is so loud I'll have to explain it on phone calls. Man if the 6610 had a middle fan speed between "audible" and "F18 on a vertical climb", it would be perfect.
So the cisco is back in for now. I'm guessing I'm going to throw the 6610 up on Ebay and try to make back what I've got into it. Putting my 10G upgrade plans on hold for now. I think I'll pickup a 7250-48P and give that a try.
Thanks to @fohdeesha on the licensing help!
You might want to check for proper airflow. My ICX6610-48-PE is in ambient up to about 80F and the switch temps never get above 63C during the hardest use I can throw at it. Besides POST, I've never had the fans get above level 1 speed. I have 2x Revision A PSUs.
Good point. I just took it apart and everything seems correct. No excessive dust. All the fans spin and are the correct orientation (rear exhaust). I did slightly reroute the power cable to the POE daughter board. It was sitting between what I believe is one of the larger switch chips (Large black Heatsinks) and the CPU (Smaller light colored heatsink near the center and the RAM. I moved the power cable so it was situated *above* the heatsink instead of behind it.
It's powered up now in the closet on the floor. So far max temp I've seen is 57.5C. I've also noticed that the Speed 1 max temp is now 84C, Where it was 76C previously. Assuming this had something to do with POE load, I added 4 SIP phones I had laying around (Ran about 6 POE devices earlier in todays test). But the extra POE devices didn't change the switching temp thresholds.
Only thing different from this morning and now is the amount of POE load (About half, 12W vs 20W). And the physical location (On a shelf), with a warm synology and 2x Lenovo tiny PC's on top of it. Guess I'll let it heat soak a bit and see where it gets. I seriously doubt by cable move and slight press on the heatsink made any difference.
Code:
The stack unit 1 chassis info:
Power supply 1 (AC - PoE) present, status ok
Model Number: 23-0000142-02
Serial Number: xxx
Firmware Ver: B
Power supply 1 Fan Air Flow Direction: Front to Back
Power supply 2 (AC - PoE) present, status ok
Model Number: 23-0000142-02
Serial Number: xxx
Firmware Ver: B
Power supply 2 Fan Air Flow Direction: Front to Back
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2
Fan controlled temperature: 57.5 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->84 deg-C
Speed 2: 79<-----> 87 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Fan 2 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 42.5 deg-C
MAC 2 Temperature Readings:
Current temperature : 48.5 deg-C
CPU Temperature Readings:
Current temperature : 57.5 deg-C
sensor A Temperature Readings:
Current temperature : 47.0 deg-C
sensor B Temperature Readings:
Current temperature : 46.0 deg-C
sensor C Temperature Readings:
Current temperature : 35.5 deg-C
stacking card Temperature Readings:
Current temperature : 51.0 deg-C
Warning level.......: 77.0 deg-C
Shutdown level......: 87.0 deg-CAre you sure you want to stick it in a closed closet? It's dumping 100w+ so unless you have some way to exhaust that heat, it's going to be an oven in there
Yeah, It's not ideal. But the cisco and the rest of the gear has lived through a few summers now. Max temp I've seen the room reach is 98F. I normally prop the door open when it's warm. Which lets cool air in (Office is about 74F normally). I did this earlier today, which made zero difference on the switch temp. But had the room as low as about 78F (Monitored with an ESP/Temp sensor).
Your temps now are similar to my experience. One thing that I have noticed with my ICX6610 that may be common among some or all others.....
When "coldbooted" the CPU temps stay roughly 10C cooler until I do a reload/warmboot. It will be around 54C for months and then I warm boot and then suddenly it's at 63C consistently with no other changes in ambient conditions or load. If I pull the plugs, sit for a minute, and coldboot it again, back to 54C.
I'm a bit curious if you see anything similar.
When "coldbooted" the CPU temps stay roughly 10C cooler until I do a reload/warmboot. It will be around 54C for months and then I warm boot and then suddenly it's at 63C consistently with no other changes in ambient conditions or load. If I pull the plugs, sit for a minute, and coldboot it again, back to 54C.
I'm a bit curious if you see anything similar.
After soaking all night it didn't climb higher then 61C. I did a warm reload and it maintained temp.
So then I placed the brocade on top of the cisco. With the warm devices on top of it. Let it soak for awhile more. Still ~60C.
I just now created a trunk port between the two switches and have migrated over all the POE devices. After a few more hours I'll migrate the rest of them. So far it's holding ~60-61C like a champ. Without coming anywhere near fan mode 2.
Curiously, the mode 1 max temp is still 84c (Up from 76c yesterday morning). I still don't know what caused it to change... Or why the switch is running so much cooler today. I refuse to believe my poking and prodding of it's innards made any difference.
It's the 40gb ports/stacking card in the rear that gets really hot. Airflow is really poor in that area. Here's my chassis info:
Notice that the stacking card is almost 8-10C hotter than the CPU. The switch will go from mode 1 to 2 based on ANY of these sensors.
Code:
The stack unit 1 chassis info:
Power supply 1 (AC - Regular) present, status ok
Model Number: 23-0000144-01
Serial Number: 091
Firmware Ver: B
Power supply 1 Fan Air Flow Direction: Front to Back
Power supply 2 not present
Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 not present
Fan controlled temperature: 51.5 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->78 deg-C
Speed 2: 73<-----> 87 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
MAC 1 Temperature Readings:
Current temperature : 42.0 deg-C
CPU Temperature Readings:
Current temperature : 43.5 deg-C
sensor A Temperature Readings:
Current temperature : 27.0 deg-C
sensor B Temperature Readings:
Current temperature : 42.5 deg-C
sensor C Temperature Readings:
Current temperature : 20.0 deg-C
sensor D Temperature Readings:
Current temperature : 19.0 deg-C
stacking card Temperature Readings:
Current temperature : 51.5 deg-C
Warning level.......: 84.0 deg-C
Shutdown level......: 87.0 deg-C
Boot Prom MAC : 748e.f8e9.7fac
Management MAC: 748e.f8e9.7fac
Brocade ICX-6450-48 48 Port Gigabit NETWORK L3 Switch with Rack Ears TESTED #2B | eBay
INCLUDES RACK EARS AND POWER CABLE. THIS IS NOT A POE SWITCH.
ebay.us
Great deal for ICX-6450-48P in the EU.
This is because Brocade/Arris were basically lazy and there is no fan ramp.
ICX 6-series (probably 7 too, but I don't have one to check) has exactly two fan speeds. 6V, and 12V. Nothing between. No actual ramping. It does not even have basic logic. It is literally just a switch. At CPU threshold, it puts the fans to 12V. As soon as they're below threshold, it drops to 6V. So the CPU starts getting hot again. Lather, rinse, repeat. It is not capable of running the fans at any level between there which would fix the issue.
And the 6610 uses contra-rotating fans, which is literally the loudest possible configuration. (And the only way to move enough air in the chassis.) 7-series uses the same contra-rotating fans in some models. This is a contra-rotating fan:
And this is why you use a contra-rotating fan:
HOWEVER, the 6610 is a front-to-rear (NEBS) chassis. So if you want to get the temperatures down, you can increase extraction at the rear (ducting and extraction fans) or increase intake at the front (make the holes bigger, blocking ingest of hot air, directing more cold air.)
Yes and no. Basically all the configurations that adopt two or more fans stacked toghether to increase static pressure will use counter-rotating rotors, as it will minimize the vortex effect (as the two units needs to have opposite pitched blade), helps straightening the flow, and it won't allow for "passive coupling" of the second fan, dragging and slowing down the air (it can be seen in the video, before the user applies "full power" thus blowing away the free fan) instead of adding more thrust/speed to it.
If you have ever seen one, this configuration is used also for multi-stage boat propellers, as the outer one rotates the other way around than the inner one.
BTW, most switches use this simple two-stages logic, basically those are small pizza-boxes that won't dump out much energy, and the lowest flow rate is basically adequate for most of the situations as the SoC can whitstand a wide temperature range while in operation; full speed is pretty much used "on emergercies only".
Last edited:
So as an update my switch has held ~62C all day now. I moved everything over a few hours ago. Guess we'll see how it does over the next few days.
It bothers me that I have zero idea why it got better. But it's working great for now.
Just replaced the batteries in my UPS. So barring a hurricane (Central Florida) or other extended power outage, this thing won't be reloaded for a few years... hahaha.
It bothers me that I have zero idea why it got better. But it's working great for now.
Just replaced the batteries in my UPS. So barring a hurricane (Central Florida) or other extended power outage, this thing won't be reloaded for a few years... hahaha.
Glad you got the temps back to normal. I do hope you find a way to keep the closet cooler - perhaps an exhaust fan fitted in the ceiling to the attic and a small grill on the bottom of the door for cool air ingress? Your switch and other devices in your closet may not work better because of it, but they may last longer and avoid further cooling noise.
2 questions in this post.
1) I could use some help with QoS, due to limited knowledge on the subject thus far.
I have VoIP and security camera traffic that I want to ensure gets high priority on the network. Not sure if SAN/NAS traffic should get afforded any special QoS, so comments on that welcome.
Anyway, how do I go about setting and enforcing QoS for VoIP and streaming camera traffic on my network? Each are seperated into their own VLAN and subnet:
VoIP - VLAN 2 - 10.1.2.0/24
Cams - VLAN 3 - 10.1.3.0/24
SAN - VLAN 4 - 10.1.4.0/24
I'm having trouble understanding where, how, and what to apply to correctly apply QoS shaping.
2) Is there any benefit to using the Switch image over the Router image on an ICX6450 doing only switch duty?
1) I could use some help with QoS, due to limited knowledge on the subject thus far.
I have VoIP and security camera traffic that I want to ensure gets high priority on the network. Not sure if SAN/NAS traffic should get afforded any special QoS, so comments on that welcome.
Anyway, how do I go about setting and enforcing QoS for VoIP and streaming camera traffic on my network? Each are seperated into their own VLAN and subnet:
VoIP - VLAN 2 - 10.1.2.0/24
Cams - VLAN 3 - 10.1.3.0/24
SAN - VLAN 4 - 10.1.4.0/24
I'm having trouble understanding where, how, and what to apply to correctly apply QoS shaping.
2) Is there any benefit to using the Switch image over the Router image on an ICX6450 doing only switch duty?
Last edited:
How's this for a start on QoS settings?
I have VoIP and streaming security cameras I want to give priority across the network. Is this looking correct so far?
This is on my ICX6450 setup as an access switch.
Now do I add this ACL to each interface?
On my ICX6610, which also has edge devices on it, do I add these rules to all ports as well, or add them to the ACLs already in use on the VEs?
On both switches, the trust dscp option should be set on the uplink ports only, correct?
Also going to mention that I switched my ICX6450 to the ICX64S switching-only firmware. It only required some minor configuration changes, but works as expected. I still don't think there's any benefits to doing so, except slightly less complicated configuration.
I have VoIP and streaming security cameras I want to give priority across the network. Is this looking correct so far?
This is on my ICX6450 setup as an access switch.
Code:
ip access-list extended markqos
permit tcp 10.1.3.0 0.0.0.255 eq 9000 any dscp-marking 32
permit tcp any 10.1.3.0 0.0.0.255 eq 9000 dscp-marking 32
permit tcp any any eq 554 dscp-marking 32
permit tcp any eq 554 any dscp-marking 32
permit udp any any range 6970 6979 dscp-marking 32
permit udp any range 6970 6979 any dscp-marking 32
permit tcp any any eq 5060 dscp-marking 24
permit tcp any eq 5060 any dscp-marking 24
permit udp any eq 5060 any dscp-marking 24
permit udp any any eq 5060 dscp-marking 24
permit udp 10.1.2.0 0.0.0.255 range 10000 20000 any dscp-marking 46
permit udp any 10.1.2.0 0.0.0.255 range 10000 20000 dscp-marking 46
permit ip any anyOn my ICX6610, which also has edge devices on it, do I add these rules to all ports as well, or add them to the ACLs already in use on the VEs?
On both switches, the trust dscp option should be set on the uplink ports only, correct?
Also going to mention that I switched my ICX6450 to the ICX64S switching-only firmware. It only required some minor configuration changes, but works as expected. I still don't think there's any benefits to doing so, except slightly less complicated configuration.
Just got a 7150-C12P from ebay.
Should I be concerned about the bad blocks? I'm guessing it's fine but it'd be nice to get a second opinion while I can still return it.
Everything seems to be working normally.
Code:
ICX7150-Boot>nand bad
Device 0 bad blocks:
05a00000
05b00000
ICX7150-Boot>nand info
Device 0: nand0, sector size 1024 KiB, Micron NAND 2GiB
Page size 4096 b
OOB size 224 b
Erase size 1048576 b
subpagesize 4096 b
options 0x 10200
bbt options 0x 0Everything seems to be working normally.
