Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
Appreciate the offer and the help you've provided to the community, but wanted to try to do this myself as a learning experience. Fortunately, I did not learn how to brick a switch, but rather how to manually load the bootloader code, which will be a valuable skill as I manage a fleet of these.

I first stumbled across the 6430/6450 steps, but instead followed the 7150-C12P recover that was documented in the end of Nov, in which you used:

##RECOVER
##this example uses the mnz10114 bootloader from the 8080 codetrain/zip
tftp 0x80000000 mnz10114.bin
sf probe 0:0
sf erase 0 +11F3D8
sf write 0x80000200 0 0x11F3D8

##VERIFY
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000

The version of uboot on this switch is exactly the same as that 7150-C12P, the only difference being that this is it's bigger brother (7150-48zp). So I'm leaning toward that being the more correct approach in my case. I had gotten as far as loading the mnz10114.bin file from TFTP/USB into memory, and validating that it wrote successfully. I also saw both sflash chips:

u-boot> sf probe 0:0
Access set to SECONDARY FL..
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB

u-boot> sf probe 0:1
Flash set to PRIMARY FL
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
u-boot>

Following your example of the C12P, you erased the first MB (+11F3D8) on secondary flash (0:0) starting at the beginning (0) and just copied over the ram contents that were populated from the bootloader file loaded from TFTP. That all seems fairly straightforward. I took a leap of faith and performed the exact same steps to write the new bootloader onto the start of secondary flash, crossed my fingers, and it booted into the Brocade 08080 bootloader ca. 2018. From there, re-applying the official boot image and finally full Brocade OS was trivial.

I am still getting the device key or cert file not available errors, even after issuing sz disable from conf t and reloading. Otherwise, life is good, and I will continue to upgrade this to the most recent UFI codebase and maybe even open up telnet to the Linux environment as you've described.

Thanks again for all you've done for the community, and add 7150-48zp to the list of hardware that these steps are known to work for.
d'oh - thought you were on a 7450 for some reason. Indeed if you have an icx7150 variant, they all have the same boot process/loader/flash locations. glad you got it figured out!

the key thing to note is the brocade-shipped bootloader file has a bunch of metadata at the beginning, and if you write that to flash, it's not gunna boot. You'll note in my instructions, you copy in the whole bootloader file to 0x80000000 in RAM, but then when writing to SPI flash, the RAM address you tell it to pull from is 0x80000200 - skipping the first 0x200 bytes of the copied-in brocade bootloader, so it starts writing from the actual bootloader start. One detail is this offset/metadata size could change at anytime (totally up to ruckus) which is one of many reasons I'm weary to give out instructions instead of doing it myself, I don't want people to brick their stuff if the offset changes and they don't know how to verify it. It's pretty obvious when opening the bootloader in a hex editor like HxD, it's the metadata, empty space and then the actual bootloader start, which for these ARM devices typically always has "EA" at the end of the first row of bytes:



so indeed in this example the offset is still 0x200 bytes

the other big thing is verifying where in SPI the actual bootloader needs to be written to. this is almost always at the very beginning, but as usual it's subject to change on different platforms so it needs to be verified first. typically to verify you would read the very first couple hundred bytes of SPI flash, and see if it looks like the very beginning of a bootloader. If it doesn't, that's not where you should write it. it should look similar to the starting bytes of our example bootloader above:

Code:
##to read boot flash for sanity check
(0:0 is primary boot flash, 0:1 is secondary)
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000
you should see something very similar to the following, the DEADBEEF on the fourth line is a dead giveaway you're in the right place:



if you get something close to that, you know the bootloader is supposed to be written right at the start of said SPI and you're good to go
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
Hm that is actually possible, since in the end my Workstation VM has access to all the VLANs... so there is some path at least;)
Would have expected to see stp shutdowns on the other switches then , but maybe this is different then a regular loop.

Thanks, will have a look at that, it sounds like a reasonable explanation.
it's not exactly a loop as the management port has no access or connection to the regular ports, so it won't "complete the loop". It does have the same MAC though, and of course two "devices" having the same MAC is going to confuse the shit out of clients when they try to access it
 
  • Like
Reactions: Rand__

Zer0Day

New Member
Dec 29, 2020
6
6
3
d'oh - thought you were on a 7450 for some reason. Indeed if you have an icx7150 variant, they all have the same boot process/loader/flash locations. glad you got it figured out!

the key thing to note is the brocade-shipped bootloader file has a bunch of metadata at the beginning, and if you write that to flash, it's not gunna boot. You'll note in my instructions, you copy in the whole bootloader file to 0x80000000 in RAM, but then when writing to SPI flash, the RAM address you tell it to pull from is 0x80000200 - skipping the first 0x200 bytes of the copied-in brocade bootloader, so it starts writing from the actual bootloader start. One detail is this offset/metadata size could change at anytime (totally up to ruckus) which is one of many reasons I'm weary to give out instructions instead of doing it myself, I don't want people to brick their stuff if the offset changes and they don't know how to verify it. It's pretty obvious when opening the bootloader in a hex editor like HxD, it's the metadata, empty space and then the actual bootloader start, which for these ARM devices typically always has "EA" at the end of the first row of bytes:



so indeed in this example the offset is still 0x200 bytes

the other big thing is verifying where in SPI the actual bootloader needs to be written to. this is almost always at the very beginning, but as usual it's subject to change on different platforms so it needs to be verified first. typically to verify you would read the very first couple hundred bytes of SPI flash, and see if it looks like the very beginning of a bootloader. If it doesn't, that's not where you should write it. it should look similar to the starting bytes of our example bootloader above:

Code:
##to read boot flash for sanity check
(0:0 is primary boot flash, 0:1 is secondary)
sf probe 0:0
sf read 0x80000000 0 100
md 0x80000000
you should see something very similar to the following, the DEADBEEF on the fourth line is a dead giveaway you're in the right place:



if you get something close to that, you know the bootloader is supposed to be written right at the start of said SPI and you're good to go
Excellent insight and yes as always your mileage will vary, so I'd certainly recommend people be careful when performing this type of surgery. I did use the 200 byte offset and also verified the hex values were what I expected and where I expected them before copying from ram onto SPI, especially the consecutive string of 14F09FE5.
 
  • Like
Reactions: fohdeesha

Zer0Day

New Member
Dec 29, 2020
6
6
3
it's not exactly a loop as the management port has no access or connection to the regular ports, so it won't "complete the loop". It does have the same MAC though, and of course two "devices" having the same MAC is going to confuse the shit out of clients when they try to access it
Right, this isn't a spanning tree issue as the mgmt port isn't participating in STP or switching traffic, it's an issue with the CAM and ARP tables on other switches binding the MGMT IP/MAC to a specific port.
 
Last edited:
  • Like
Reactions: Rand__

ClintE

Member
Feb 22, 2019
30
7
8
I noticed my second 6610 (48P) is showing stack ID 2 instead of 1, like the 24F. I don't want to stack the switches, so I would like to configure the 48P to be ID 1 also. No matter what I try, I can't seem to get this to take. When attempting some stack commands, it returns an error message that the switch is not the active controller. I think I'm missing something simple. Any suggestions?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,645
2,847
113
32
fohdeesha.com
I noticed my second 6610 (48P) is showing stack ID 2 instead of 1, like the 24F. I don't want to stack the switches, so I would like to configure the 48P to be ID 1 also. No matter what I try, I can't seem to get this to take. When attempting some stack commands, it returns an error message that the switch is not the active controller. I think I'm missing something simple. Any suggestions?

did you follow my guide on both of them to reset them to factory defaults? sounds like the second one is still configured to be a stack member. theres a blurb in the update guide about removing stack configs
 

ClintE

Member
Feb 22, 2019
30
7
8
did you follow my guide on both of them to reset them to factory defaults? sounds like the second one is still configured to be a stack member. theres a blurb in the update guide about removing stack configs
Yes, but when I enter the command "stack unconfigure clean", I get the message "This command is not available on standalone or Active Controller".

When entering "show stack" it returns:

***** Warning! stack is not enabled. *****

T=18m55.0: alone: standalone, D: dynamic cfg, S: static
ID Type Role Mac Address Pri State Comment
2 S ICX6610-48P alone xxxx.xxxx.xxxx 0 local None:0


+---+
2/1| 2 |2/6
+---+
Current stack management MAC is xxxx.xxxx.xxxx
 

JacobRhea

New Member
Feb 1, 2019
16
1
3
I'm having issues getting internet access to my VLANs. I have a PFSense router and a Brocade ICX6610. The VLANs can all see each other, and the switch can ping out to google, but I can't get anything on any VLAN to see outside of my LAN.

My PFSense router is 10.0.0.1 and I've added a gateway of 10.0.0.2 for the switch, I've then added static routes for all of the VLANs on the switch (10.0.xx.0/24). The router connects to port 47 on the switch.

I'm currently using the DHCP server on the brocade and was relying on PFSense for DNS.

Code:
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
!
!
!
!
vlan 1 name TRASNIT by port
 untagged ethe 1/1/47
 router-interface ve 1
!
vlan 3 name DEFAULT-VLAN by port
!
vlan 10 name Servers by port
 untagged ethe 1/1/25 to 1/1/32
 router-interface ve 10
!
vlan 20 name Storage by port
 untagged ethe 1/1/35 to 1/1/36
 router-interface ve 20
!
vlan 30 name Media by port
 untagged ethe 1/1/33 to 1/1/34
 router-interface ve 30
!
vlan 99 name Management by port
 untagged ethe 1/1/13 to 1/1/24
 router-interface ve 99
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
default-vlan-id 3
enable telnet authentication
enable aaa console
hostname switch
ip dhcp-client disable
ip dhcp-server enable
!
ip dhcp-server pool servers
 dhcp-default-router 10.0.10.1
 dns-server 10.0.0.1
 excluded-address 10.0.10.1
 lease 1 0 0
 network 10.0.10.0 255.255.255.0
 deploy
!
!
ip dhcp-server pool storage
 dhcp-default-router 10.0.20.1
 dns-server 10.0.0.1
 excluded-address 10.0.20.1
 lease 1 0 0
 network 10.0.20.0 255.255.255.0
 deploy
!
!
ip dhcp-server pool media
 dhcp-default-router 10.0.30.1
 dns-server 10.0.0.1
 excluded-address 10.0.30.1
 lease 1 0 0
 network 10.0.30.0 255.255.255.0
 deploy
!
!
ip dhcp-server pool management
 dhcp-default-router 10.0.99.1
 dns-server 10.0.0.1
 excluded-address 10.0.99.1
 lease 1 0 0
 network 10.0.99.0 255.255.255.0
 deploy
!
ip dns server-address 10.0.0.1
ip route 0.0.0.0/0 10.0.0.1
!
no telnet server
username admin password .....
snmp-server community ..... ro
!
!
clock summer-time
clock timezone gmt GMT-06

!
ntp
 disable serve
 server 216.239.35.0
 server 216.239.35.4
!
!
web-management https
!
!
!
!
!
!
!
interface management 1
 disable
!
interface ethernet 1/3/1
 speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
 speed-duplex 10G-full
!
interface ethernet 1/3/4
 speed-duplex 10G-full
!
interface ethernet 1/3/5
 speed-duplex 10G-full
!
interface ethernet 1/3/6
 speed-duplex 10G-full
!
interface ethernet 1/3/7
 speed-duplex 10G-full
!
interface ethernet 1/3/8
 speed-duplex 10G-full
!
interface ve 1
 ip address 10.0.0.2 255.255.255.0
!
interface ve 10
 ip address 10.0.10.1 255.255.255.0
 ip helper-address 1 10.0.0.1
!
interface ve 20
 ip address 10.0.20.1 255.255.255.0
 ip helper-address 1 10.0.0.1
!
interface ve 30
 ip address 10.0.30.1 255.255.255.0
 ip helper-address 1 10.0.0.1
!
interface ve 99
 ip address 10.0.99.1 255.255.255.0
 ip helper-address 1 10.0.0.1
!
!
!
!
!
!
!
ip ssh  password-authentication no
!
end
 
  • Like
Reactions: tommybackeast

ArmedAviator

Member
May 16, 2020
91
55
18
Kansas
@JacobRhea ,

Is your DHCP working right? To me it looks like it shouldn't for two reasons.....

1) Don't use the switch's DHCP server - it is non-authorative on the ICX6xxx series and this will not work with many devices. ISC's dhcpd server + BIND named for DNS is easy to configure together,, lightweight, and reliable.

2) You have your IP helper address as your pfSense. If it's using the default DHCP server, it will not work for any VLANs not configured and the DHCP server running on those VLANs - it will not work with a transport setup like you have.

Bonus tips:
3) Though it's certainly not necessary, it is common to use /30s for point-to-point links.

4) You can skip the VLAN to connect to your pfSense box and simply assign the port an IP address.
Code:
int eth 1/1/47
ip address 10.0.0.2/30
route-only
Regarding your specific issue......
5) Can you show the pfSense routing table?

6) Perhaps your DNS server is resolving AAAA (IPv6) but your devices have no global IPv6 addresses (the VEs and pfSense LAN/VLAN would need to have IPv6 addresses as well)[/code]
 

JacobRhea

New Member
Feb 1, 2019
16
1
3
@ArmedAviator

Here is the routing table for PFSense:
Code:
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            c-73-183-224-1.hsd UGS        igb0
10.0.0.0/24        link#1             U           em0
router             link#1             UHS         lo0
10.0.10.0/24       10.0.0.2           UGS         em0
10.0.20.0/24       10.0.0.2           UGS         em0
10.0.30.0/24       10.0.0.2           UGS         em0
10.0.99.0/24       10.0.0.2           UGS         em0
10.0.100.0/24      10.0.0.2           UGS         em0
10.10.10.1         link#4             UH          lo0
73.183.224.0/22    link#2             U          igb0
c-73-183-224-186.h link#2             UHS         lo0
localhost          link#4             UH          lo0

Internet6:
Destination        Gateway            Flags     Netif Expire
default            fe80::256:2bff:fe7 UG         igb0
localhost          link#4             UH          lo0
fe80::256:2bff:fe7 fe80::256:2bff:fe7 UGHS       igb0
fe80::%em0/64      link#1             U           em0
fe80::ec4:7aff:fe0 link#1             UHS         lo0
fe80::%igb0/64     link#2             U          igb0
fe80::ec4:7aff:fe0 link#2             UHS         lo0
fe80::%lo0/64      link#4             U           lo0
fe80::1%lo0        link#4             UHS         lo0
Currently, all devices connected to the switch aren't having issues getting an ip address. I know that many people have issues with IoT devices but I don't have any of those on this network currently and just figured I'd stick to using the switches DHCP until I have to setup a separate server.

Are the IP Helper addresses not necessary for this setup?
 

ArmedAviator

Member
May 16, 2020
91
55
18
Kansas
The IP helper address is used when the DHCP server is not on the same L2 network. In your case, the DHCP server is on each L2 network so no need, besides you have yours pointing to pfSense which should not also be offering DHCP leases on those VLANs since you're using the switch's DHCP server.

Your routing table looks good. You can combine them all into one if you want for simplicity and easier adding of subnets, e.g. 10.0.0.0/16.

Thinking about it, I ran into this issue before and know exactly what it is. You must add the subnets to pfSense's NAT Outbound settings. By default, it only translates the networks configured on pfSense through NAT. In this case, you need to add the subnets that is not configured on pfSense. Again, probably easiest to just do a single netmask of 10.0.0.0/16.
 

JacobRhea

New Member
Feb 1, 2019
16
1
3
My NAT settings in pfSense are set to "Automatic" and it appears that the routes have already been added under "Source".
 

JacobRhea

New Member
Feb 1, 2019
16
1
3
I tried that and still had issues, I realized I could actually ping out to google from one of the systems I was having trouble with so I went into pfSense DNS Resolver and added that vlan subnet to the access list and everything seems to be working. I'm not sure if this is the correct fix, but it works.
 

csementuh

Member
Oct 7, 2019
36
10
8
Pittsburgh, PA
Not offhand, sorry. I tried to reproduce your issue here locally with a wireshark dump, but I see no DNS requests (or anything, really) after running those sz disable commands. that was on an icx7250 running 8092d

have you rebooted the switch since running all the SZ disable stuff? Only other thing I can think to try. you SURE you're seeing requests from the switch itself?
Thanks for checking!

I'm running FW 8080e like in your guide. Is the 8092d a newer version I should try? I didn't see any reference to it.

My switch had been up for 180 days so I did reboot it. No change sadly. I really don't see how it's not the switch, considering I don't own anything else Brocade/Ruckus. I haven't done a WireShark or anything so perhaps I'll have to try it and see.

EDIT: I changed my switch DNS to only use the pi-hole and I can see the direct hits from my main switch to the ruckus domains. It is 100% doing the DNS hits directly even though all of the sz stuff is disabled.

Rebooting the switch did clear all the junk from the sz logs. I have this now with current data only, but it doesn't look very exciting.

Code:
>show sz logs
Start i/max/iter 0/5/1
Jan  7 23:48:52:ca_stop_timer>Cancelled Timer id 0, rc=0
Jan  7 23:48:52:ca_start_timer:95912>Started timer with l/p 30000/30000 for u17, rc 0, id 167172236
Jan  7 23:48:53:ca_stop_timer>Cancelled Timer id 0, rc=41
Jan  7 23:48:53:sz_execute_state_machine>Entering with state/event: INIT/0, DISABLE/2005
Jan  7 23:48:53:sz_execute_state_machine>Exit with state/event: DISABLED/1, NONE/2000 RC: 1
End i/max/iter 5/5/0
 
Last edited: