Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Hi guys. I opened a new thread regarding some L3 questions and moving parts or all of my inter VLAN routing tfrom pfSense to the switch.
If you have time to help, heres the link. Thanks!
If you have time to help, heres the link. Thanks!
Hey guys, I'm brand new to enterprise switches and was able to snag an ICX 7250-24 off ebay recently. Running through the setup today with fohdeesha's guide and this is what I'm running into. I think this was the second time running through the steps because I thought I missed something. Maybe I'm missing something obvious, but I'm getting this error when attempting to boot from primary:
Below is the full log including commands sent. IPs have been changed back to fohdeesha's defaults for security--I don't know if this is necessary, but I'm not having any issues connecting to tftp so I don't think the IPs are the problem.
Any thoughts? Thanks in advance.
Code:
ICX7250-24 Copper (Non-POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0
Enter 'b' to stop at boot monitor: 0
bootdelay: ===
Booting image from Primary
NAND read: device 0 offset 0x0, size 0x2000000
................................ 33554432 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
Code:
Brocade Bootloader: 10.1.06T215 (Dec 14 2015 - 13:10:57)
Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK
ICX7250-24 Copper (Non-POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0
Enter 'b' to stop at boot monitor: 0
ICX7250-Boot> setenv ipaddr 192.168.1.50
ICX7250-Boot> setenv netmask 255.255.255.0
ICX7250-Boot> setenv serverip 192.168.1.8
ICX7250-Boot> setenv image_name ICX7xxx/SPR08090mc.bin
ICX7250-Boot> setenv uboot ICX7xxx/spz10118.bin
ICX7250-Boot> update_uboot
Loading image to Uboot Partition 2
Using bcmiproc_eth-0 device
TFTP from server 192.168.1.8; our IP address is 192.168.1.50
Filename 'ICX7xxx/spz10118.bin'.
Load address: 0x61007dc0
Loading: ######################################################
done
Bytes transferred = 786944 (c0200 hex)
sf erase 0x0 0x100000
copying uboot image to flash, it will take sometime...
sf write 0x61007fc0 0x0 0xc0000
TFTP to Flash Done.
ICX7250-Boot> saveenv
ICX7250-Boot> reset
resetting ...
Ruckus Networks Bootloader: 10.1.18T215 (Jul 13 2020 - 01:53:05)
Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK
ICX7250-24 Copper (Non-POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0
Enter 'b' to stop at boot monitor: 0
ICX7250-Boot> factory set-default
Execution of "factory set-default" will remove all user data like config, keys etc.
Do you want to continue? (Y/N) Y
Command executed successfully
ICX7250-Boot> update_primary
Updating PRIMARY image over TFTP
Using bcmiproc_eth-0 device
TFTP from server 192.168.1.8; our IP address is 192.168.1.50
Filename 'ICX7xxx/SPR08090mc.bin'.
Load address: 0x1000000
Loading: #################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
#################################################################
###########
done
Bytes transferred = 33554432 (2000000 hex)
img_sz=0x2000000 block_count = 0x20
NAND erase.spread: device 0 offset 0x0, size 0x2000000
Erasing at 0x1f00000 -- 100% complete.
OK
NAND write: device 0 offset 0x0, size 0x2000000
................................ 33554432 bytes written: OK
TFTP to Flash Done.
ICX7250-Boot> setenv ipaddr
ICX7250-Boot> saveenv
ICX7250-Boot> boot_primary
resetting ...
Brocade Bootloader: 10.1.06T215 (Dec 14 2015 - 13:10:57)
Validate Shmoo parameters stored in flash ..... OK
Restoring Shmoo parameters from flash .....
Running simple memory test ..... OK
ICX7250-24 Copper (Non-POE), PVT1
SYS CPLD VER: 0x6, Released Ver: 0
Enter 'b' to stop at boot monitor: 0
bootdelay: ===
Booting image from Primary
NAND read: device 0 offset 0x0, size 0x2000000
................................ 33554432 bytes read: OK
Wrong Image Format for bootm command
ERROR: can't get kernel image!
could not boot from primary, no valid image; trying to boot from secondary
BOOTING image from Secondary
NAND read: device 0 offset 0x4000000, size 0x2000000
................................................................ 33554432 bytes read: OK
## Booting kernel from Legacy Image at 70000200 ...
Image Name: Image
ARM Linux Kernel Image (uncompressed)
Data Size: 31678084 Bytes = Load Address: 61008000
Entry Point: 61008000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK
Starting kernel ...
Config partition mounted
FIPS reset is enable.
Removing startup config & backup
Removing stacking.boot
Removing SSL_CERT_FILE
Removing SSH_RSA_HOSTKEY_FILE
Removing SSH_DSA_HOSTKEY_FILE
Creating TUN device
Starting the FastIron.
FIPS Disabled:PORT NOT DISABLED
platform type 81
OS>Applying factory defaults..
INFO: startup config data is not available, try to read from backup
INFO: startup config data in the backup area is not available
set_board_level: gi_board_type = 108SOC unit 0 attached to PCI device BCM56344_A0
Initializing the parallel detect
Pre Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also
Parsing Config Data ...
INFO: empty config data in the primary area, try to read from backup
INFO: empty config data in the backup area also
Hotplugger Daemon Initialized ..
Successfuly set IngressClassPort on CPU port to 2
System initialization completed...console going online.
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
UNIT 1: compiled on May 19 2016 at 01:44:58 labeled as SPR08030h
(31678660 bytes) from Secondary SPR08030h.bin
SW: Version 08.0.30hT213
Compressed Boot-Monitor Image size = 786944, Version:10.1.06T215 (spz10118)
HW: Stackable ICX7250-24
==========================================================================
UNIT 1: SL 1: ICX7250-24 24-port Management Module
Serial #:DUH3817N019
License: ICX7250_L3_SOFT_PACKAGE (LID: fwjINGMpFGO)
License Compliance: ICX7250-PREM-LIC-SW is Compliant
P-ASIC 0: type B344, rev 01 Chip BCM56344_A0
==========================================================================
UNIT 1: SL 2: ICX7250-SFP-Plus 8-port 80G Module
==========================================================================
1000 MHz ARM processor ARMv7 88 MHz bus
8192 KB boot flash memory
2048 MB code flash memory
2048 MB DRAM
STACKID 1 system uptime is 16 second(s)
The system : started=cold start
My stack unit ID = 1, bootup role = alone
ICX7250-24 Router>
ICX7250-24 Router>
PS 1, Internal Power supply detected.
PS 1, Internal Power supply is up.
Stack unit 1 Power supply 1 is up
TFTP session timed out
TFTP session timed out
TFTP session timed out
TFTP session timed out
TFTP session timed out
TFTP session timed out
TFTP session timed out
TFTP session timed out
TFTP session timed outAny thoughts? Thanks in advance.
Have you tried pinging the gateway IP of the other vlan from the device. ex. Device on vlan16 ping switch ip on vlan 32. If that works then you should have vlan ip routing working. Another would be to check your netmask settings on each device.@kapone I'll try anything at this point!
I've got a VLAN 16 and a VLAN 32 now. Same setup -- x.x.[32/16].10 IP's for two different devices, one on each subnet. Neither can ping the other at this moment.
Anything else I can check? Is there a way I could have inadvertently disabled inter-VLAN routing?
You could also run a "show ip route" and check your routing table to see if there are any strange routes.
Another thing is in the vlan settings, do you have a "router-interface" line.
example
vlan 16 by port
router-interface ve 16
where interface "ve 16" is your interface configured with the gateway IP of your vlan.
So, I recently switched from my Brocade switch to a Cisco one. I was having problems mostly with multicast, mDNS/Bonjour stuff specifically. I would have Google Homes or Chromecasts not showing up in lists to cast to, light switches which wouldn't turn on and off, Airprint printers which would show up sometimes and then not others, and other weird behavior I otherwise couldn't explain. It was not all the time, and sometimes it worked fine.
Initially, I thought this was a result of how Brocade had implemented PIM, as I had phones and PCs in a separate L2 segment. For science, I merged all of my devices into the same vlan and the problem persisted. I tried turning PIM off and just used IGMP. I tried using Ubiquiti, Aruba and finally Cisco APs as I thought maybe one brand or another's WiFi would solve the problem with their various multicast features. The same weird behavior still existed no matter the vendor. Aruba and Cisco worked better and was more stable than Ubiquiti, but that is probably a result of tunneling/CAPWAPing the traffic to a controller, so the MAC addresses of all of the various devices were literally on the same port on the switch, However, if I ran them in Flexconnect or Bridging mode, the multicast behavior was just as unreliable as the Ubiquiti deployment. I even thought my Android phone's IP stack was just hokey (I was having other problems, too. Bluetooth and Wifi calling never worked reliably, as an example), so I switched to an iPhone. Problems persisted.
Finally, I kicked the Brocade 7450 switch (which had replaced an HP E3800 switch because it had better 10GbE density) for a Cisco 3850 switch. I turned PIM on the Cisco and enabled the mdns-sd gateway. The problem is more-or-less solved now. Things populate properly for casting all of the time, the AirPrint printers always show up, I can be in a L3 hop away and things still work, and I can turn my wifi lights on and off reliably. The only problem I have is that Apple Home occasionally does not like when I roam between APs and thinks things are disconnected on my iPhone, but then my iPad is working fine and It is connected to the same AP. I may just work towards CAPWAPing my traffic again and calling this a success.
In addition, the local network is performing measurably better. Take a look at this graph, as an example. This is a SmokePing chart monitoring my core switch. Can you see where I transitioned from the Brocade to the Cisco?
And just so you don't think something is wrong with my SmokePing instance, PRTG tells a similar story, and is running on different metal:
I've kept my ICX 7150 in production as an access switch (L2 only) in my Garage and it seems to do that job fairly well. I have an AP running on it and it seems to be behaving fine for that purpose. It's latency graphs looks better than the 7450 I had in production with L3 services (OSPF, PIM, etc) running.
But the latency is so much better on my Cisco switch that I even took my VDX out of production. This is an enterprise access/aggregation switch performing 40-50% better than a Datacenter switch in terms of latency, if Ping times are to be believed. The VDX always did this weird ping spike thing every day or so that I could never explain...pinging THROUGH it didn't correlate to an increase in latency so I never really gave it much mind but it was an annoying anomaly nonetheless.
As much as I like the value of what Brocade/Ruckus has to offer, clearly it can be worth paying more money to get better stability and performance...
Figured I would share, maybe my experience will help someone make a purchasing decision later.
If anyone wants a Brocade ICX 7450, 6450 or a VDX 6740 (SFP+, not copper, lower power consumption version) I have one of each sitting in a pile right now I would be willing to get rid of for a reasonable price (shipping costs are the issue really).
Initially, I thought this was a result of how Brocade had implemented PIM, as I had phones and PCs in a separate L2 segment. For science, I merged all of my devices into the same vlan and the problem persisted. I tried turning PIM off and just used IGMP. I tried using Ubiquiti, Aruba and finally Cisco APs as I thought maybe one brand or another's WiFi would solve the problem with their various multicast features. The same weird behavior still existed no matter the vendor. Aruba and Cisco worked better and was more stable than Ubiquiti, but that is probably a result of tunneling/CAPWAPing the traffic to a controller, so the MAC addresses of all of the various devices were literally on the same port on the switch, However, if I ran them in Flexconnect or Bridging mode, the multicast behavior was just as unreliable as the Ubiquiti deployment. I even thought my Android phone's IP stack was just hokey (I was having other problems, too. Bluetooth and Wifi calling never worked reliably, as an example), so I switched to an iPhone. Problems persisted.
Finally, I kicked the Brocade 7450 switch (which had replaced an HP E3800 switch because it had better 10GbE density) for a Cisco 3850 switch. I turned PIM on the Cisco and enabled the mdns-sd gateway. The problem is more-or-less solved now. Things populate properly for casting all of the time, the AirPrint printers always show up, I can be in a L3 hop away and things still work, and I can turn my wifi lights on and off reliably. The only problem I have is that Apple Home occasionally does not like when I roam between APs and thinks things are disconnected on my iPhone, but then my iPad is working fine and It is connected to the same AP. I may just work towards CAPWAPing my traffic again and calling this a success.
In addition, the local network is performing measurably better. Take a look at this graph, as an example. This is a SmokePing chart monitoring my core switch. Can you see where I transitioned from the Brocade to the Cisco?
And just so you don't think something is wrong with my SmokePing instance, PRTG tells a similar story, and is running on different metal:
I've kept my ICX 7150 in production as an access switch (L2 only) in my Garage and it seems to do that job fairly well. I have an AP running on it and it seems to be behaving fine for that purpose. It's latency graphs looks better than the 7450 I had in production with L3 services (OSPF, PIM, etc) running.
But the latency is so much better on my Cisco switch that I even took my VDX out of production. This is an enterprise access/aggregation switch performing 40-50% better than a Datacenter switch in terms of latency, if Ping times are to be believed. The VDX always did this weird ping spike thing every day or so that I could never explain...pinging THROUGH it didn't correlate to an increase in latency so I never really gave it much mind but it was an annoying anomaly nonetheless.
As much as I like the value of what Brocade/Ruckus has to offer, clearly it can be worth paying more money to get better stability and performance...
Figured I would share, maybe my experience will help someone make a purchasing decision later.
If anyone wants a Brocade ICX 7450, 6450 or a VDX 6740 (SFP+, not copper, lower power consumption version) I have one of each sitting in a pile right now I would be willing to get rid of for a reasonable price (shipping costs are the issue really).
Attachments
Last edited:
I know you've switched equipment, but when you were troubleshooting everything on the same vlan, did you have an designated IGMP querier? When you were playing with PIM across vlans, did you set up an RP?So, I recently switched from my Brocade switch to a Cisco one. I was having problems mostly with multicast, mDNS/Bonjour stuff specifically. I would have Google Homes or Chromecasts not showing up in lists to cast to, light switches which wouldn't turn on and off, Airprint printers which would show up sometimes and then not others, and other weird behavior I otherwise couldn't explain. It was not all the time, and sometimes it worked fine.
Initially, I thought this was a result of how Brocade had implemented PIM, as I had phones and PCs in a separate L2 segment. For science, I merged all of my devices into the same vlan and the problem persisted. I tried turning PIM off and just used IGMP. I tried using Ubiquiti, Aruba and finally Cisco APs as I thought maybe one brand or another's WiFi would solve the problem with their various multicast features. The same weird behavior still existed no matter the vendor. Aruba and Cisco worked better and was more stable than Ubiquiti, but that is probably a result of tunneling/CAPWAPing the traffic to a controller, so the MAC addresses of all of the various devices were literally on the same port on the switch, However, if I ran them in Flexconnect or Bridging mode, the multicast behavior was just as unreliable as the Ubiquiti deployment. I even thought my Android phone's IP stack was just hokey (I was having other problems, too. Bluetooth and Wifi calling never worked reliably, as an example), so I switched to an iPhone. Problems persisted.
Finally, I kicked the Brocade 7450 switch (which had replaced an HP E3800 switch because it had better 10GbE density) for a Cisco 3850 switch. I turned PIM on the Cisco and enabled the mdns-sd gateway. The problem is more-or-less solved now. Things populate properly for casting all of the time, the AirPrint printers always show up, I can be in a L3 hop away and things still work, and I can turn my wifi lights on and off reliably. The only problem I have is that Apple Home occasionally does not like when I roam between APs and thinks things are disconnected on my iPhone, but then my iPad is working fine and It is connected to the same AP. I may just work towards CAPWAPing my traffic again and calling this a success.
In addition, the local network is performing measurably better. Take a look at this graph, as an example. This is a SmokePing chart monitoring my core switch. Can you see where I transitioned from the Brocade to the Cisco?
View attachment 21491
And just so you don't think something is wrong with my SmokePing instance, PRTG tells a similar story, and is running on different metal:
View attachment 21492
As much as I like the value of what Brocade/Ruckus has to offer, clearly it can be worth paying more money to get better stability and performance...
Figured I would share, maybe my experience will help someone make a purchasing decision later.
If anyone wants a Brocade ICX 7450, 6450 or a VDX 6740 I have one of each sitting in a pile right now I would be willing to get rid of for a reasonable price (shipping costs are the issue really).
The short answer is yes. The longer answer is I had tried so many different variables and versions of this that I'm not sure there would be value in recanting every detail, but I can further elaborate if you want.
I want to also reiterate, it's not that it didn't work, it just didn't work reliably.
Hey there,
I ran into the same problem on my 7450, to run update_uboot immediately before running update_primary otherwise I kept getting the old bootloader after reset. I've quoted my own post below
I ran into the same problem on my 7450, to run update_uboot immediately before running update_primary otherwise I kept getting the old bootloader after reset. I've quoted my own post below
I'm kind of wondering if the Brocade needed more commands to configure correctly & you missed something & the cisco needed less & was easier to make work. I'm going to try this with a brocade (switching away from a 3945e) and I'll post how it goes. My brocades are a stack of 6610s however, not the 64xx series.
Hey Wolfcastle. So I had actually tried this before (then using the update_primary command after removing the temp IP) and no dice--same error. This time, I tried it again and did "reset" instead and now it's working. Bizarre. But thanks for the message--got me to go back and try this again!
And now for a second question as I get further along in the process here with my ICX 7250-24. Is the temp reported in show chassis accurate? This seems way hotter than it should be with nothing connected and no PoE:
It's worth noting that I did swap out the stock fan with a Sunon 40mm x 20mm KDE1204PKVX (8200rpm, 10.2 cfm). The stock fan had a fairly annoying coil whine to it and this thing sits fairly close to my desk. I assumed it wouldn't be that big of a hit to thermals on a 24 port switch without PoE. Thoughts?
I'm going to open it up and look at cleaning / reapplying thermal paste (assuming that's a thing you can do).
Code:
The stack unit 1 chassis info:
Power supply 1 (AC - Regular) present, status ok
Power supply 2 not present
Fan ok, speed (auto): 1<->[[2]]
Fan controlled temperature: 86.8 deg-C
Fan speed switching temperature thresholds:
Speed 1: NM<----->98 deg-C
Speed 2: 67<----->105 deg-C (shutdown)
Fan 1 Air Flow Direction: Front to Back
Slot 1 Current Temperature: 87.4 deg-C (Sensor 1)
Slot 2 Current Temperature: NA
Warning level.......: 100.0 deg-C
Shutdown level......: 105.0 deg-CI'm going to open it up and look at cleaning / reapplying thermal paste (assuming that's a thing you can do).
Last edited:
When I had only one fan in my 6610, my temps where quite high (60c+). When I got a 2nd chassis fans, my tem wnet down 15-20c. I'd look into that.
I had 2x rev A in my 6610 and 1x Rev C in my 7450. There is very little difference between the Rev A and C to my ears. In both cases the 23k delta fan is whiny and loud even when running at low speed. The only real solution is to make the switch 2 RU as described in the fan mod thread with 2x larger fans + quiet 40mm fans in the PSU.
Since the switch is in your office, if you’ve got the vertical space for it, I would highly recommend adding a large fan above the asic. These switches aren’t hot because they put out a ton of heat, they’re hot because they’re 1RU and they’re supposed to go in a datacenter where noise doesn’t matter.
I'm going to test temps again today with the stock fan to see if that makes a big difference. As far as a fan on the ASIC itself, I had seen someone suggest this: https://www.mouser.com/ProductDetail/Sunon/MF60101V3-1000U-A99?qs=EU6FO9ffTwekSIQ43CVQVg==
I'm guessing you'd point it up, but given the space constraints, I'm wondering how much of a difference this would make. I assume I could run it off a y-splitter on the one fan header?
You mentioned vertical space--I guess that would mean running it without the cover? I'd definitely prefer to keep it in the enclosure if possible, but obviously that introduces cooling challenges. Or were you thinking just on the exterior? The bottom gets way hotter than the top, so maybe propping it up off the metal shelf it's sitting on or putting some airflow underneath it might help (update: propping it up didn't make any difference).
I know these things run hot, but it seems like idling at 85-90 is too hot and could cause issues.
Edit: I think the best option is to add a fan to the ASIC along with the quieter exhaust fan. I've seen folks recommend the Sunon MF60101V1-1000U-G99, but I can't find it in stock anywhere... Any other good 60x10mm fans you guys would recommend? I assume I need the 3-wire here and a y-splitter to run both this and the exhaust off the one fan header?
Last edited: