Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[shremi]

Hey guys i am a noob when it comes to switches and i am still learning maybe its very stupid but can someone help me with setting up some basic ACL:

Here is the setup
Russound Amp vlan 40 with an ip of 10.0.40.40
Home Assistant with 10.0.0.40 on the main Vlan
Pfsense router with 10.0.0.1 on the main Vlan
Computer to test with 10.0.0.23 on the main vlan

The goal is for the russound amp 10.0.40.40 to only talk to the specific devices and to the pfsense for dhcp and dns. The issue is that the russound loses connectivity after a while and people on the HA forums have managed to fix the issue by creating an ACL and letting the amp to only talk to specific clients and pfsense.

I have created the ACL (standard don't know if that matters ) and assigned to the port of the amp

access-list 1 permit host 10.0.0.1
access-list 1 permit host 10.0.0.40
access-list 1 permit host 10.0.0.23

Assigned the ACL to the port in IN ( i know it has VLAN but is running dual mode)

I cant ping the AMP from my test computer

Main goal is to make 10.0.40.40 to talk to the clients i specify on the main LAN

PS: Stay tuned for a dual 120mm fan mod

 

[Nikotine]

I'm sorry if this has been answered already, this thread is huge.
I just received my ICX 6450-48P and it didn't come with a console cable.
Is the console port just an RS-232 interface (Tx, Rx and GND)? Because I have in my box an RS-232 to usb interface.
If yes, which pins of an RJ45 do what?

 

[BlackCat]

I'm sorry if this has been answered already, this thread is huge.
I just received my ICX 6450-48P and it didn't come with a console cable.
Is the console port just an RS-232 interface (Tx, Rx and GND)? Because I have in my box an RS-232 to usb interface.
If yes, which pins of an RJ45 do what?

You can check standard Cisco serial cable. Most of time it works brilliant.

 

[tubs-ffm]

So I'm going to totally disagree and say you can do it safely. I modified a 48P and quieted it down quite a bit. It's been running for a year continuously in a poorly ventilated closet under moderate POE load with no issues. That said, I feel terrible for tubs - that is NOT something I'd want to wake up to.

In my original post I was writing that I cannot prove that it is because of the fan modification I did. But a single case that something is running I would not consider as a prove to say it is save. A single case that fails gives me enough confidence to at least give a warning.

I am not an expert for cooling and not an expert for electronics neither I have technical data to judge what value is save and where it is getting dangerous. Maybe your 3 Mechatronics MR4020X12B1-RSR are sufficient and my 2 Delta EFB0412VHD-F00 are not. I do not know.

 

[asceca]

I'll try using the RPS9 power supply for broacade net-iron - the connector appears to be the same but can't seems to find affordable fans refurbished or new anywhere...

I was hoping the 6610 fans would work but not they don't

Hello, have you tried it?
Can RPS9 power supply work on icx7750?

 

[Nikotine]

You can check standard Cisco serial cable. Most of time it works brilliant.

Thanks, I just went ahead and tried it.
Indeed only 4 connections needed.

RJ45 pins:
3 -> TX
4 & 5 -> GND
6 -> RX

I initially had TX and RX inverted, because of the pinouts I found on Google, but I had to invert them.

USB-serial interface is an ICP CON I-7561 that I bought on eBay.
It's probably cheaper to just buy the damn console cable, but now I used what I had laying around.

Now comes the fun part, configuring this beast.

 

[richtj99]

Any thoughts on 6450vs7250 for homelab usage? Two 6450's is $300ish & two 7250's is closer to $850. I am looking to keep these for years so power usage is a concern also to me. Is it worth it to spring for the newer switch?

 

[RoachedCoach]

In my original post I was writing that I cannot prove that it is because of the fan modification I did. But a single case that something is running I would not consider as a prove to say it is save. A single case that fails gives me enough confidence to at least give a warning.

I am not an expert for cooling and not an expert for electronics neither I have technical data to judge what value is save and where it is getting dangerous. Maybe your 3 Mechatronics MR4020X12B1-RSR are sufficient and my 2 Delta EFB0412VHD-F00 are not. I do not know.

Agreed, we have no way of knowing for sure. I remember reading your post about the PSU popping and letting out an audible groan.

For anyone going forward, my best rule of thumb is to match the replacement components as closely to the originals in terms of performance. I'm going off the assumption Brocade knew what they were doing when they built these, so any specs used on the original parts will be sufficient for the machine to function as intended.

The original fans in this system moved an airflow of around 25 CFM according to my research. You were rocking two Deltas with a max CFM of around 10 CFM; the Deltas I initially tried were 24 CFM, and then the Mechatronics I settled on are around 17 CFM.

Obviously no data to support, but my best guess would be the much higher airflow of the Mechatronics fans (plus having 3 instead of 2) would play a factor. Then again, you were monitoring your temps and they seemed fine, so it very well could've been a faulty cap, or maybe the temp sensors simply don't tell the whole story with this stuff.

I don't know - best advice to anyone on these boards once they get modding is understand the risks involved. And I hope we don't come to find these have crap PSUs longterm.

 

[kousuke]

Hello, have you tried it?
Can RPS9 power supply work on icx7750?

Yes, I confirm they are running with no alarm whatsoever

 

[eduncan911]

So I'm going to totally disagree and say you can do it safely. I modified a 48P and quieted it down quite a bit. It's been running for a year continuously in a poorly ventilated closet under moderate POE load with no issues. That said, I feel terrible for tubs - that is NOT something I'd want to wake up to.

Here's my fan mod post. The weird pinouts for fans are in the post that the below post quotes.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-159#post-256375

I would also add, if you feel the PSU sensor is running on the hot side, you could always side mount a fan next to PSU externally (there's vents there). Not the cleanest look in the world, but it'd be effective. I tried it for kicks and it dropped around 5 - 7 C off the PSU, but didn't stick with it since mine never climbed to a point that was anywhere close to unsafe for its tolerances.

The OP that asked to quiet it down has a 7450. And the follow-up person that said his PSU exploded, also had a 7450.

The post you linked to above is for the 7250 - different PSU (there can be two in the 7450), and I think different fan using "Fan Modules" that are reversible.

Nevermind, nothing to see here. Please drive through.

 

[RoachedCoach]

The OP that asked to quiet it down has a 7450. And the follow-up person that said his PSU exploded, also had a 7450.

The post you linked to above is for the 7250 - different PSU (there can be two in the 7450), and I think different fan using "Fan Modules" that are reversible.

What? Tubs had a 7250-24P

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-256#post-297209

OP also has a 7250-24P

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-257#post-297899

 

[eduncan911]

What? Tubs had a 7250-24P

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-256#post-297209

OP also has a 7250-24P

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-257#post-297899

How did I... What the f...

Nevermind! I must have 7450 on my brain. *slaps self*

 

[D4M4EVER]

On my ICX 6610-48P the fans started ramping up to fighter jet speed, the same level as it does at first boot.

Started looking at my switch temps and the CPU seems kind of hot to me, are these temps concerning?

The fans are kicking off full speed to get the CPU temp back down and seems to slow back down once CPU temp is around 71deg C

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature: 75.5 deg-C

Fan speed switching temperature thresholds:
        Speed 1: NM<----->76       deg-C
        Speed 2:       71<-----> 80 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back                        
MAC 1 Temperature Readings:                                      
        Current temperature : 54.5 deg-C                        
MAC 2 Temperature Readings:                                      
        Current temperature : 64.0 deg-C                        
CPU Temperature Readings:                                        
        Current temperature : 75.0 deg-C                        
sensor A Temperature Readings:                                  
        Current temperature : 61.5 deg-C                        
sensor B Temperature Readings:                                  
        Current temperature : 61.5 deg-C                        
sensor C Temperature Readings:                                  
        Current temperature : 47.0 deg-C                        
stacking card Temperature Readings:                              
        Current temperature : 54.0 deg-C                        
        Warning level.......: 77.0 deg-C                        
        Shutdown level......: 80.0 deg-C

 

[ArmedAviator]

What's the ambient temp of that room? If the fans are ramping up to level 2, it's probably too hot for average use case.

I have two ICX6610s in a stack. One runs 10C cooler than the other both MACs and the CPU. No idea why the difference despite same load/PoE/devices plugged in.

 

[Nikotine]

I followed @fohdeesha 's guide to do the initial setup of the ICX 6450.
It all works, got it updated, has L3 firmware now and a static IP.

I would like it to get a reserved IP via dhcp from my router.
So I removed the
ip dhcp-client disable
and
ip address 192.168.1.55/24
lines from config, but the interface doesn't get an IP address.

ICX6450-48P-Router(config)#show run
Current configuration:
!
ver 08.0.30tT313
!
stack unit 1
  module 1 icx6450-48p-poe-port-management-module
  module 2 icx6450-sfp-plus-4port-40g-module
!
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable telnet authentication
hostname ICX6450-48P-Router
!
username root password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT+01
!
!
ntp
 disable serve
 server 192.168.1.1
!
!
!
!
!
interface ve 1
!
!
!
!
!
!
!
!
!
end

ICX6450-48P-Router(config)#show ip
Global Settings
  ttl: 64, arp-age: 10, bootp-relay-max-hops: 4
  router-id : 0.0.0.0
  enabled : UDP-Broadcast-Forwarding  Source-Route  Load-Sharing  RARP  VRRP-Extended  VSRP
  disabled: Route-Only  Directed-Broadcast-Forwarding  BGP4 IRDP  Proxy-ARP  RIP  OSPF  VRRP  ICMP-Redirect  add-host-route-first

ICX6450-48P-Router(config)#show logging
Syslog logging: enabled ( 0 messages dropped, 0 flushes, 0 overruns)
    Buffer logging: level ACDMEINW, 14 messages logged
    level code: A=alert C=critical D=debugging M=emergency E=error
                I=informational N=notification W=warning

Static Log Buffer:
Jan  1 01:00:42:I:System: Stack unit 1 POE  PS 1, Internal Power supply  with 740000 mwatts capacity is up

Dynamic Log Buffer (50 lines):
Jan  1 01:02:02:D:DHCPC: Stopped DHCP Client service
Jan  1 01:01:58:I:Security: console login by root to PRIVILEGED EXEC mode
Jan  1 01:00:42:I:System: Stack unit 1 POE  PS 1, Internal Power supply  with 740000 mwatts capacity is up
Jan  1 01:00:42:I:System: Interface ethernet 1/1/47, state up
Jan  1 01:00:42:I:System: Interface ve 1, state up
Jan  1 01:00:40:I:System: Warm start
Jan  1 01:00:38:I:PoD: No license present for port 1/2/4
Jan  1 01:00:38:I:PORT: 1/2/4 is ERR-DISABLED due to invalid license
Jan  1 01:00:30:I:PoD: No license present for port 1/2/2
Jan  1 01:00:30:I:PORT: 1/2/2 is ERR-DISABLED due to invalid license
Jan  1 01:00:22:D:DHCPC: starting dhcp client service on 53 port(s)
Jan  1 01:00:22:I:NTP: The system clock is not synchronized to any time source.
Jan  1 01:00:22:I:NTP: The system clock is not synchronized and does not have a reference configured.

While reading the fastiron manual, there is reference made to
ip dhcp-client enable
but that option isn't there:

ICX6450-48P-Router(config)#ip dhcp-client
  auto-update       Enable the DHCP client auto-update
  continuous-mode   In hours, default is 24-hrs
  disable           disable DHCP client globally on router

So what am I missing here?

 

[D4M4EVER]

What's the ambient temp of that room? If the fans are ramping up to level 2, it's probably too hot for average use case.

I have two ICX6610s in a stack. One runs 10C cooler than the other both MACs and the CPU. No idea why the difference despite same load/PoE/devices plugged in.

The room is ~22deg C (72deg F), i have the switch on top of my 42U rack but loaded in from the back side, the fans are suckers (Front to Back) I think they might be pulling in to much heat from the servers in the rack below it.

Both the front and back of rack is open, I have about 1ft from back of rack to a wall. My old swiotch i had did not have these kind of heat readings but also did not have the same fan power as this guy does.

 

[Harr_cami]

So I'm going to totally disagree and say you can do it safely. I modified a 48P and quieted it down quite a bit. It's been running for a year continuously in a poorly ventilated closet under moderate POE load with no issues. That said, I feel terrible for tubs - that is NOT something I'd want to wake up to.

Here's my fan mod post. The weird pinouts for fans are in the post that the below post quotes.

https://forums.servethehome.com/index.php?threads/brocade-icx-series-cheap-powerful-10gbe-40gbe-switching.21107/page-159#post-256375

I would also add, if you feel the PSU sensor is running on the hot side, you could always side mount a fan next to PSU externally (there's vents there). Not the cleanest look in the world, but it'd be effective. I tried it for kicks and it dropped around 5 - 7 C off the PSU, but didn't stick with it since mine never climbed to a point that was anywhere close to unsafe for its tolerances.

Thanks so much for your response mate, its greatly appreciated, ill purchase 4 of those fans right now!

 

[Harr_cami]

The wire switch needed is to swap the tachometer and ground pins. It's rather easy: use a flat-blade screwdriver or the end of a tweezer to push the pin in, pull the wires out, and reinsert them after swapping.

I found that the Delta FFB0412VHN (40x28, 15.8CFM, 9500RPM, 45 dbA) is a reasonable swap. They are loud at boot, but after switching to fan speed 1, they run at 3400RPM and aren't audible at that speed unless you listen for them. On my 7250-48P, ASIC temperature is currently about 77C and the POE sensor is only at 28.5C. The FFBs also have an operating voltage range down to 4.5V, while the EFB's are only specced to a 7V minimum voltage.

Btw, does anyone know what the temperature thresholds for the 7250P model POE sensor are supposed to be? My unit says 41/34, but the Ruckus docs say that's for the 7250-24P, and the -48P is supposed to be 57/49 -- is that a typo in their table?

CFM requirements from data sheet:
ICX 7250-24G: 14.3 CFM (Maximum), 5.7 CFM (Typical) ICX 7250-24 : 14.0 CFM (Maximum), 5.7 CFM (Typical) ICX 7250-24P: 34.4 CFM (Maximum), 13.7 CFM (Typical) ICX 7250-48: 34.8 CFM (Maximum), 13.8 CFM (Typical) ICX 7250-48P: 45.2 CFM (Maximum), 21.4 CFM (Typical)

Thanks for your advice and help!

 

[LodeRunner]

The room is ~22deg C (72deg F), i have the switch on top of my 42U rack but loaded in from the back side, the fans are suckers (Front to Back) I think they might be pulling in to much heat from the servers in the rack below it.

Both the front and back of rack is open, I have about 1ft from back of rack to a wall. My old swiotch i had did not have these kind of heat readings but also did not have the same fan power as this guy does.

Top of rack, with port-side intake and ports lined up with the outflow from your servers? Given the volume of air the fans move and the pressure they are capable of, you're definitely pulling the hot exhaust from your other gear into the switch. Either convert it to port-side exhaust, or move it to front of rack if you can.

 

[atomicwrites]

Hi all, I was about to pull the trigger on an ICX 6450 POE model, I'm currently limping along with a cheapo 8 port managed switch, and another 8 port POE switch both of which are full. My "future proof" (hopefully) requirements are 24 or more ports, 16 or more POE ports, at least two but preferably 4 10G sfp ports, and quiet enough to keep by my desk, although completely quiet is not necessary. Layer 3 routing would be nice, that way I don't have to ship all that traffic though a 1G link to the router. The problem is I read someone said the license unlocks don't work anymore, is that right? I know that means I'd lose 10G functionality on 2 of the 4 sfps, not sure what other issues I'd have. Are these still the "best" for an under $200 budget now?

P.S. is there a search in thread function? I couldn't find it.