Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

didnt take me long to get lost again. im baffled. i've started the whole setup from the beginning multiple times but just found its still referencing the original ip i gave it weeks ago the first time (10.0.0.203) while im logged into web interface from ip i set yesterday? 10.0.0.252....

am i just misunderstanding or is this actually a problem i need to figure out.

Not sure I see your problem. You still have Internet access from your client machines right? Your switch can ping the outside world right? (try ping 8.8.8.8, assuming you are allowing your switch's IP out to the real world - and am assuming you are since you've set NTP servers outside your network.)

your arp cache says it knows about 10.0.0.203 and that a user is logged into the switch's web interface from that IP and you can successfully ssh into 10.0.0.252?

I think you are also saying when you configured the switch you originally gave it the IP of 10.0.0.203...

Just curious, have you checked the system you are using to access the web console to see if it has the IP 10.0.0.203??? that's how I read what you posted.

 

[seany]

EDIT made it in. set password. sent write mem about 8 times. was able to get into serial and web. is there a way to backup this whole setup in a currently working state to revert back to this point when i screw it up again? im not really trusting it much at the moment since i didnt do anything to fix it over night it just started working on its own but locked me out....

got home from work this morning and suddenly the 7250 is connected to internet and working as a switch and i can get to the login page of the web gui but neither the default user/pw or one i set is working to login to the switch. anyway to get through the login or do i need to start over again?

The output of `show run` is the whole config. If you save your working config with that, you can always just paste it back into a reset switch and be back to where you want.

 

[BossHoss]

Not sure I see your problem. You still have Internet access from your client machines right? Your switch can ping the outside world right? (try ping 8.8.8.8, assuming you are allowing your switch's IP out to the real world - and am assuming you are since you've set NTP servers outside your network.)

your arp cache says it knows about 10.0.0.203 and that a user is logged into the switch's web interface from that IP and you can successfully ssh into 10.0.0.252?

I think you are also saying when you configured the switch you originally gave it the IP of 10.0.0.203...

Just curious, have you checked the system you are using to access the web console to see if it has the IP 10.0.0.203??? that's how I read what you posted.

im logged into the web interface and was using ssh to access the switch at 10.0.0.252 but the switch is saying its ip is 10.0.0.203. i've reset and started the whole reset and update proces from scratch litterally a dozen times since changing the ip weeks ago with the latest being earlier today. i've been under the impression i successfully changed the ip weeks ago to the 252. so its now 10.0.0.203 but identifies as 10.0.0.252? im sorry it may be perfectly normal but seems crazy to me and i dont understand how/why it could use both. i bought the switch to learn i just didnt think i would struggle this much.

it can reach the internet and pings 8888 100% with 32ms.

 

[itronin]

im logged into the web interface and was using ssh to access the switch at 10.0.0.252 but the switch is saying its ip is 10.0.0.203. i've reset and started the whole reset and update proces from scratch litterally a dozen times since changing the ip weeks ago with the latest being earlier today. i've been under the impression i successfully changed the ip weeks ago to the 252. so its now 10.0.0.203 but identifies as 10.0.0.252? im sorry it may be perfectly normal but seems crazy to me and i dont understand how/why it could use both. i bought the switch to learn i just didnt think i would struggle this much.

it can reach the internet and pings 8888 100% with 32ms.

Your switch is .252....

sorry if I was not clear... Did you check the IP address of the host you were using to access the web management page?

Your logs show that the SRC IP address of the device that accessed the web management page is .203.
Your ARP cache shows that the switch learned the .203 address on port 5 of your 7250t (1/2/5).

you grok enough to show the needed information and post it (awesome) but seem to have gotten yourself if in a bit of tizzy around what that information actually means.

Perhaps you have confused yourself because you started with the switch at .203 - but then you changed it... and (speculating though does it matter at this point?) maybe you forgot to write mem one of those times and so you lost access because your client device (or another device had .203) when the switch came back at one point with the IP you originally set it up with. However since then you've said its a .252 - your config shows .252 your logs show .252 and your LOGS and arp cache show that access is coming from (client device) .203 to your switch at .252.

 

[CIR-Engineering]

After a week or so with the ICX7450-32ZP in place there haven't been any issues, Engenius EWS377 link up on the 2.5 Ports without issues.

The only thing I am working on night might not be an issue on the switch, but my Asus ProArt Creator X670e will not link at 10Gb, only getting 5Gb and the switch shows the link at 10Gb. I tested another cable to no avail, I am hunting around for another 10-GBaseT SFP Modules in the hoard to see if I can rule that out as well. This was working previously on the ICX6610 at 10Gb.

Mine is running great as well even though it fails the diagnostics.

On a 10-GBase-T transceiver the switch will always report 10 GbE. There is no 2.5 or 5.0 GbE in SFP land so the transceiver negotiates the speed and connection with the device it's connected to and "converts" that to 10 GbE for the switch. The switch doesn't have anything to do with it.

 

[BossHoss]

Your switch is .252....

sorry if I was not clear... Did you check the IP address of the host you were using to access the web management page?

Your logs show that the SRC IP address of the device that accessed the web management page is .203.
Your ARP cache shows that the switch learned the .203 address on port 5 of your 7250t (1/2/5).

you grok enough to show the needed information and post it (awesome) but seem to have gotten yourself if in a bit of tizzy around what that information actually means.

Perhaps you have confused yourself because you started with the switch at .203 - but then you changed it... and (speculating though does it matter at this point?) maybe you forgot to write mem one of those times and so you lost access because your client device (or another device had .203) when the switch came back at one point with the IP you originally set it up with. However since then you've said its a .252 - your config shows .252 your logs show .252 and your LOGS and arp cache show that access is coming from (client device) .203 to your switch at .252.

feeling real dumb right about now lol. yes i switched the ip of the pc im using to match the tftp setup and when i was done and set it back to dhcp it was given 203 while i was still thinking i was working from 10.0.0.143. and yes it was bugging me that i couldnt understand what was going on and im still worried since the switch was working fine as a simple switch for weeks then stopped letting things connect to the internet and seemed to put everything plugged into it into a seperate network, after trying for 2 days to figure it out, i went to sleep and it was working when i woke up. luckily i still have all the old switches in the rack and was able to just move all the most important stuff to those quickly while figuring it out.

 

[BossHoss]

The output of `show run` is the whole config. If you save your working config with that, you can always just paste it back into a reset switch and be back to where you want.

thats great to know. do i need to delete any characters or anything before pasting?

 

[etakmit]

I have two 6450s (both PoE, one is 24p the other 48p). The 24p I've had for months and is fine. The 48p I just picked up this week.

Updated the firmware throughout (PoE included) and have done nothing else.

What I'm seeing is :

ICX6450-48P Router#sho cpu
4 percent busy, from 2 sec ago
1 sec avg: 99 percent busy
5 sec avg: 6 percent busy
60 sec avg: 7 percent busy
300 sec avg: 7 percent busy
ICX6450-48P Router#sho cpu
6 percent busy, from 3 sec ago
1 sec avg: 99 percent busy
5 sec avg: 5 percent busy
60 sec avg: 7 percent busy
300 sec avg: 7 percent busy
ICX6450-48P Router#sho cpu
7 percent busy, from 6 sec ago
1 sec avg: 99 percent busy
5 sec avg: 4 percent busy
60 sec avg: 7 percent busy
300 sec avg: 7 percent busy
ICX6450-48P Router#

This is very noticeable even on the serial console port (every keypress is delayed / slow to respond). If I add an IP to allow telnet/ssh I see this on pings:

Reply from 192.168.1.252: bytes=32 time=2038ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2067ms TTL=64
Reply from 192.168.1.252: bytes=32 time=26ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2033ms TTL=64
Reply from 192.168.1.252: bytes=32 time=3ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2035ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2072ms TTL=64
Reply from 192.168.1.252: bytes=32 time=26ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2040ms TTL=64
Reply from 192.168.1.252: bytes=32 time=3ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2037ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2061ms TTL=64

Looking for some debug direction to chase this down.

ICX6450-48P Router#sho cpu tasks
... Usage average for all tasks in the last 1 second ...
==========================================================
Name %

SigHdlrTsk 0
OsTsk 0
TimerTsk 0
FlashTsk 0
MainTsk 0
MportPollTsk 0
IntrTsk 0
stkKeepAlive 0
keygen 0
itc 0
poeFwdfsm 0
scp 0
appl 100
snms 0
rtm 0
rtm6 0
rip 0
bgp 0
bgp_io 0
ospf 0
ospf_r_calc 0
ipsec 0
dhcp6 0
snmp 0
rmon 0
web 0
acl 0
flexauth 0
ntp 0
rconsole 0
console 0
ospf_msg_task 0
auxTsk 0

 

[klui]

Instead of freaking out and unplugging the ethernet cable when the network went down, I waited. Everything came back up after ~30 seconds.

So... I'm not sure if the config changes were necessary. I can revert and test later later - my girlfriend is trying to run Final Fantasy XIV dungeons and I feel bad for interrupting her over and over.

Still, seems weird for the network to go down for 30 seconds on connection. But I can figure that out at my leisure.

STP uses the concept of roots and the algorithm determines which device should become the root of the tree if a new switch that supports the protocol is introduced into the network. The important attributes are priority and MAC address in that order. Most switches have spanning tree bridge priority set to the maximum of 32768. When multiple devices have the same priority then the MAC address is the tie breaker.

If your network goes down then becomes stable the root has changed. If STP was properly configured your network should never go down whenever a new STP-capable device is connected. The lower value for priority and MAC address determines who would become the root. Therefore your current root should have a priority less than the maximum. Then all new devices with max priority will never become the root unless you remove your current root device.

Properly setting STP bridge priority is one of the best practices when configuring spanning tree.

 

[etakmit]

I have two 6450s (both PoE, one is 24p the other 48p). The 24p I've had for months and is fine. The 48p I just picked up this week.

Updated the firmware throughout (PoE included) and have done nothing else.

What I'm seeing is :



This is very noticeable even on the serial console port (every keypress is delayed / slow to respond). If I add an IP to allow telnet/ssh I see this on pings:

Reply from 192.168.1.252: bytes=32 time=2038ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2067ms TTL=64
Reply from 192.168.1.252: bytes=32 time=26ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2033ms TTL=64
Reply from 192.168.1.252: bytes=32 time=3ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2035ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2072ms TTL=64
Reply from 192.168.1.252: bytes=32 time=26ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2040ms TTL=64
Reply from 192.168.1.252: bytes=32 time=3ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2037ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2ms TTL=64
Reply from 192.168.1.252: bytes=32 time=2061ms TTL=64

Looking for some debug direction to chase this down.

Interestingly I booted from the secondary flash partition (running 7.4) and the problem entirely disappears. I ran an update (copy tftp flash <ip> <file> primary) to just refresh the install - that didn't help either.

I may boot back to the secondary and wipe the primary partition and then reinstall and see if that helps

 

[clcorbin]

no

Crude... I guess I am going to reconfigure how I have ports distributed to the nodes as it does appear I can bond the 1/2/x ports or the 1/3/x ports, just can't bond a 1/2/x port with a 1/3/x port.

Thanks for the definitive answer!

 

[CircuitSlinger]

Mine is running great as well even though it fails the diagnostics.

On a 10-GBase-T transceiver the switch will always report 10 GbE. There is no 2.5 or 5.0 GbE in SFP land so the transceiver negotiates the speed and connection with the device it's connected to and "converts" that to 10 GbE for the switch. The switch doesn't have anything to do with it.

Good to know, first time I've ever run into that (I usually run DACs as much as possible), this is the only device that I have to use a transceiver for.

 

[etakmit]

Interestingly I booted from the secondary flash partition (running 7.4) and the problem entirely disappears. I ran an update (copy tftp flash <ip> <file> primary) to just refresh the install - that didn't help either.

I may boot back to the secondary and wipe the primary partition and then reinstall and see if that helps

And my testing continues:

If I run 7.4 - everything is fine
If I run 8.0.1 - everything is fine
If I run 8.0.30u - I see the original behavior. I've run it from both flash banks - doesn't matter.

I downloaded a fresh copy of the 8.0.30 - doesn't matter. Seems there's something here that's getting 'stuck' and pegging the CPU with absolutely nothing enabled (nothing plugged into the switch except power / serial cable) and config is entirely blank.

 

[etakmit]

So I've tried:

7.4 : works fine
8.0.1 : works fine
8.0.1c : pegged CPU 100%
8.0.10m : pegged CPU 100%
8.0.20d : pegged CPU 100%
8.0.30t : pegged CPU 100%
8.0.30u : pegged CPU 100%

verified that my 6450-28-PoE is running 8.0.30u and that hasn't had a single issue.

this is all with just a serial cable / power (unplug ethernet and wipe config then reload). I did a few upgrades via the bootloader / ethernet only in mgmt port as well.

only a single task

appl

has any CPU usage and its always at 100. I've read though that that is the generic Foundry process basically so it's no surprise (this behavior matches my 24 port as well - that shows 100 on appl at all times - but show cpu never breaks 1%)

 

[EmnaX]

Yep, just put a 1 G media converter back in to make sure and I'm getting about 940 Mbps on all devices (both on servers connected with 10 G DAC and laptops connected with 1 G Ethernet).

The 10 G media converter does supply full bandwidth (about 1500 Mbps) to the servers connected using 10 G DACs. In this configuration, the devices connected to the 1 G Ethernet ports on the Brocade switch only get about 600 Mbps instead if something closer to 1 Gbps and that's why I was suspecting the switch.

I have not tried a SFP+ to RJ45 transceiver yet. I'll order the one you have linked too and give it a shot.

I am also going to borrow a UniFi switch later this week with some SFP+ ports and some 1 G Ethernet ports to test and see if I get the same results across a different switch.

Thanks for your suggestions @CIR-Engineering !

I think I fixed my issue!

Quick recap:
ISP provides 1500 Mbps download speed. pfSense router has 10 Gbps link to Brocade ICX6450. On devices connected to 1 Gbps Ethernet ports, I was only getting around 600 Mbps instead of the expected 940 ish Mbps. Devices connected to the 10 Gbps ports on the switch were getting full speed from the ISP (1500 Mbps).

Fix:
Log into the switch and then do the following:

enable
conf t
symmetric-flow-control enable
show symmetric
write mem
exit

Results:
Running a speed test at www.speedtest.net now gives about 940 Mbps on 1 Gbps Ethernet links

I'm not a network guy so I'm not really sure what the difference is between the regular flow-control that is globally enabled by default and this other symmetric-flow-control. For whatever reason though, it seems to fix my issue. It is possible to check the status with show symmetric. And more reading on the setting can be found here: Commscope Technical Content Portal

Other things I tried:
@CIR-Engineering pointed me to a RJ45 transceiver and suggested removing my media converter so I picked that up and gave it a shot but it didn't fix this issue. I also tried removing the Brocade ICX 6450 and replaced it temporarily with a UniFi USW-Pro-8-POE to run the same tests across its 10 Gbps ports and 1 Gbps ports but I got the same strange results (low speeds across 1 Gbps Ethernet ports).

 

[DavidVermillion]

My stupid unifi switch has become a problem and I want it gone.

Am I missing something, because here's an ebay listing for a 100 dollar ICX725048P, which looks like the switch in the OP that has 8 sfp+ 10gb ports and 48 POE 1gb ports and draws 50 watts (plus POE of course.) Is this a different model or have they just dropped in price that much?

 

[etakmit]

My stupid unifi switch has become a problem and I want it gone.

Am I missing something, because here's an ebay listing for a 100 dollar ICX725048P, which looks like the switch in the OP that has 8 sfp+ 10gb ports and 48 POE 1gb ports and draws 50 watts (plus POE of course.) Is this a different model or have they just dropped in price that much?

Nah they've come down pretty well at this point. plenty in the 85-125 range (before shipping)

 

[Gerhen]

My stupid unifi switch has become a problem and I want it gone.

Am I missing something, because here's an ebay listing for a 100 dollar ICX725048P, which looks like the switch in the OP that has 8 sfp+ 10gb ports and 48 POE 1gb ports and draws 50 watts (plus POE of course.) Is this a different model or have they just dropped in price that much?

Just pointing this out in case it matters for you but the 7250-48P draws around 65 watts at idle with nothing connected. Confirmed by a couple folks in a separate thread.

 

[marshallm]

Hi folks! I've got a 6610 that I've been tinkering with and I've run into a conundrum that I'm trying to wrap my head around. When adding VEs to vlans on the switch with groups of tagged/untagged ports, it seems the MAC address for the 1/1/1 or MGMT port is duplicated across all of them and I can't wrap my head around why it is happening. It doesn't seem like it should be. Is this expected behavior?

Including a snippet of "show int br" before and after creating the VLANs/VEs:

Before:

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Up      Forward Full 1G    None  No  1    0   cc4e.24be.0850                
1/1/2      Down    None    None None  None  No  1    0   cc4e.24be.0851                
1/1/3      Down    None    None None  None  No  1    0   cc4e.24be.0852                
1/1/4      Down    None    None None  None  No  1    0   cc4e.24be.0853                
1/1/5      Down    None    None None  None  No  1    0   cc4e.24be.0854                
1/1/6      Down    None    None None  None  No  1    0   cc4e.24be.0855                
1/1/7      Down    None    None None  None  No  1    0   cc4e.24be.0856                
1/1/8      Down    None    None None  None  No  1    0   cc4e.24be.0857                
1/1/9      Down    None    None None  None  No  1    0   cc4e.24be.0858                
1/1/10     Down    None    None None  None  No  1    0   cc4e.24be.0859                
1/1/11     Down    None    None None  None  No  1    0   cc4e.24be.085a                
1/1/12     Down    None    None None  None  No  1    0   cc4e.24be.085b                
1/1/13     Down    None    None None  None  No  1    0   cc4e.24be.085c                
1/1/14     Down    None    None None  None  No  1    0   cc4e.24be.085d                
1/1/15     Down    None    None None  None  No  1    0   cc4e.24be.085e                
1/1/16     Down    None    None None  None  No  1    0   cc4e.24be.085f                
1/1/17     Down    None    None None  None  No  1    0   cc4e.24be.0860                
1/1/18     Down    None    None None  None  No  1    0   cc4e.24be.0861                
1/1/19     Down    None    None None  None  No  1    0   cc4e.24be.0862                
1/1/20     Down    None    None None  None  No  1    0   cc4e.24be.0863

After:

Port       Link    State   Dupl Speed Trunk Tag Pvid Pri MAC             Name
1/1/1      Up      Forward Full 1G    None  No  1    0   cc4e.24be.0850                
1/1/2      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/3      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/4      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/5      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/6      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/7      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/8      Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/9      Down    None    None None  None  No  100  0   cc4e.24be.0850                
1/1/10     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/11     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/12     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/13     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/14     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/15     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/16     Down    None    None None  None  Yes N/A  0   cc4e.24be.0850                
1/1/17     Down    None    None None  None  No  205  0   cc4e.24be.0850                
1/1/18     Down    None    None None  None  No  1    0   cc4e.24be.0861                
1/1/19     Down    None    None None  None  No  1    0   cc4e.24be.0862                
1/1/20     Down    None    None None  None  No  1    0   cc4e.24be.0863

 

[fohdeesha]

Just pointing this out in case it matters for you but the 7250-48P draws around 65 watts at idle with nothing connected. Confirmed by a couple folks in a separate thread.

updated OP