Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
here's a list of ~800 hidden commands in fastiron. Due to the way the ghidra script ripped them out there may be some duplicates but I don't think there's more than a couple. These were pulled from an 8091 firmware image, so some may not work in switches running 8030, but most seem to. Some of these can and probably will mess your switch up and delete your config/licenses/hopes and dreams. I haven't had time to go through too many so if you find anything interesting report back
https://fohdeesha.com/data/other/brocade/FastIron-Hidden.txt
https://fohdeesha.com/data/other/brocade/FastIron-Hidden.txt
I tried the "dd temp-fanspeed" command with "?" and tab to see if FastIron had anything. I wasn't expecting anything, only curious. It didn't and instead I had to go physically reset the switch. The smart thing would have been to try on a switch I wasn't actively using, oh well.
I saw earlier in the thread there may have been an issue with passive DAC cables on the breakout ports. Was this confirmed?
really? what switch / fw version? That's one of the ones I did play with and had no issue, at least on 8030:
Code:
telnet@FCX2#dd temp-fanspeed
DECIMAL fan speed 1 or 2
DECIMAL gap between two readings (sec)
auto fan speed in auto modeupdated the config guide, added the list of ~800 hidden commands to the "hidden dev stuff" section, changed all links to https, added a fan configuration / noise tip section to the 6610 guide: FCX / ICX6610 - Fohdeesha Docs
also changed the guide color from orange to blue as it's no longer halloween
also changed the guide color from orange to blue as it's no longer halloween
Added the cheap FCX model to the main post as an option to learn BGP/VRFs/etc in a lab with - https://forums.servethehome.com/ind...s-cheap-powerful-10gbe-40gbe-switching.21107/
They predate the ICX line (came right before the ICX6610, in fact the ICX6610 and the FCX run the exact same bootloader and firmware images). They have no built in 10gbE, so you can get them cheap - around $30 for a BGP / VRF / tunnels / OSPF/ IPv6 / etc capable switch. The downside is no stock 10gbE as mentioned, they're a little louder, and draw a little more power, but not bad (around 40 watts, ICX6610 sound levels for the non PoE models, rocket ship sound levels for the PoE models). Overall I only recommend these if you're on a tight budget, don't really care about a lot of 10gbE, but want something capable of VRFs, BGP, OSPF, tunnels, etc to learn with
Advanced Routing License Note:
The FCXs required an EEPROM license to unlock the BGP and VRF capabilities. To do so, just run the following command from the bootloader (make sure to flash the latest bootloader from the guide first!):
Notes on adding 10gbE to the FCX:
The FCX has no 10gbE by default. It comes with a card slot to add a 10gbE module. There's a 2 port 10gbE module, and a 4 port 10gbE module. There's two FCX switch types: one only takes the 2-port 10gbE card, and the other only takes the 4-port 10gbE card. To decide which you want/need:
First there's the "stacking" FCX models: These are the FCX624S and the FCX648S (the models with "S" at the end). These are by far the most popular and easiest to find for pennies. These have two special CX4 16gbps ports on the rear to stack with each other. These "stacking" models only take the 2-port 10gbE XFP card. Search ebay for "ES4625M" to find these cards around $40. This is the easiest option if you want a good lab switch with 2x 10gbE ports for around $70 total.
Then there's the more rare "datacenter" models, the FCX624-I, FCX624-E, FCX648-I and FCX648-E (24 or 48 port, intake or exhaust airflow direction). These do not have the special CX4 stacking ports on the back, and only take the 4-port SFP+ 10gbE card. The downside is, if you want to stack these models (they still support stacking), there's no dedicated CX4 ports in the back so you have to use one of the SFP+ ports for the stack connection. These models are harder to find, and typically more expensive. The SFP+ card for them is also more expensive. Search "FCX-4XG" on ebay to find the 4 port SFP+ cards for around $60. This means you can get an FCX datacenter model with 4 SFP+ ports for around $120 total, so it's not recommended over the quieter and more power efficient ICX6450 (unless you really need the BGP / VRFs / ETC and don't want to spring for the ICX6610 to get them)
Overall they're a little louder than the ICX6450 series, and predate the ICX line, but are incredibly powerful software wise and will do everything, so at $30 they can make great lab switches. For a permanent home network switch, or if you care about 10gbE, I would look at the newer models listed in the original post. I will say I've had a lot of these stuck in wiring closets and in use at LAN parties and they are rock solid models that will run forever, just missing out on a lot of 10gbE options
They predate the ICX line (came right before the ICX6610, in fact the ICX6610 and the FCX run the exact same bootloader and firmware images). They have no built in 10gbE, so you can get them cheap - around $30 for a BGP / VRF / tunnels / OSPF/ IPv6 / etc capable switch. The downside is no stock 10gbE as mentioned, they're a little louder, and draw a little more power, but not bad (around 40 watts, ICX6610 sound levels for the non PoE models, rocket ship sound levels for the PoE models). Overall I only recommend these if you're on a tight budget, don't really care about a lot of 10gbE, but want something capable of VRFs, BGP, OSPF, tunnels, etc to learn with
Advanced Routing License Note:
The FCXs required an EEPROM license to unlock the BGP and VRF capabilities. To do so, just run the following command from the bootloader (make sure to flash the latest bootloader from the guide first!):
Code:
i2c write a 0 feedface00000200ffffffffffffffff 1
reset
#will now boot and be permanently licensed for all advanced featuresNotes on adding 10gbE to the FCX:
The FCX has no 10gbE by default. It comes with a card slot to add a 10gbE module. There's a 2 port 10gbE module, and a 4 port 10gbE module. There's two FCX switch types: one only takes the 2-port 10gbE card, and the other only takes the 4-port 10gbE card. To decide which you want/need:
First there's the "stacking" FCX models: These are the FCX624S and the FCX648S (the models with "S" at the end). These are by far the most popular and easiest to find for pennies. These have two special CX4 16gbps ports on the rear to stack with each other. These "stacking" models only take the 2-port 10gbE XFP card. Search ebay for "ES4625M" to find these cards around $40. This is the easiest option if you want a good lab switch with 2x 10gbE ports for around $70 total.
Then there's the more rare "datacenter" models, the FCX624-I, FCX624-E, FCX648-I and FCX648-E (24 or 48 port, intake or exhaust airflow direction). These do not have the special CX4 stacking ports on the back, and only take the 4-port SFP+ 10gbE card. The downside is, if you want to stack these models (they still support stacking), there's no dedicated CX4 ports in the back so you have to use one of the SFP+ ports for the stack connection. These models are harder to find, and typically more expensive. The SFP+ card for them is also more expensive. Search "FCX-4XG" on ebay to find the 4 port SFP+ cards for around $60. This means you can get an FCX datacenter model with 4 SFP+ ports for around $120 total, so it's not recommended over the quieter and more power efficient ICX6450 (unless you really need the BGP / VRFs / ETC and don't want to spring for the ICX6610 to get them)
Overall they're a little louder than the ICX6450 series, and predate the ICX line, but are incredibly powerful software wise and will do everything, so at $30 they can make great lab switches. For a permanent home network switch, or if you care about 10gbE, I would look at the newer models listed in the original post. I will say I've had a lot of these stuck in wiring closets and in use at LAN parties and they are rock solid models that will run forever, just missing out on a lot of 10gbE options
Last edited:
6450-24p with 8030s. I'll try again with a 48p tonight. Until the second attempt, it's hard for me to confirm exactly what happened. The scrollback looks right, but that only covers echoed characters.
Anyone have a link to the 7250 serial cable? Searched the thread the eBay links are all old and not valid any longer.
My soldering station is down ATM and would rather just buy one.
My soldering station is down ATM and would rather just buy one.
@fohdeesha or anyone else who has a 6610 with 2 Rev B/C PSUs.
Is the noise level lower with 2 PSUs vs 1 PSU? I have a 6610 with 2 fans and 1 Rev B PSU mounted in the far left slot when viewed from the rear and am curious if a second PSU will reduce the noise further. I find the noise lower with 2 fans vs. 1 fan and the temperatures are lower as well.
Thanks!
Is the noise level lower with 2 PSUs vs 1 PSU? I have a 6610 with 2 fans and 1 Rev B PSU mounted in the far left slot when viewed from the rear and am curious if a second PSU will reduce the noise further. I find the noise lower with 2 fans vs. 1 fan and the temperatures are lower as well.
Thanks!
I went a step further with my 6610-24. (As historical data...I gutted my 6610-48, cut open the top of the chassis and put THREE 120mm fans on it. It was certainly low noise, wouldn't call it silent).
If you look at the 6610 fan trays, each tray has 2x fans with a bit of space between them, but they flow in the same direction. Not counter rotating so to speak, and not joined together. The fan connector for the tray has enough wires for the two fans. So...
Took two trays...took off one fan from each tray, wired the second tray to the first one (extend the wires enough to go around the stacking card) and put them back in. It is...subjectively quieter than two fan trays (with 4x fans vs 2x fans in two trays) with slightly less power draw.
I just received this yesterday and confirmed it to work with my ICX-7250-48:
Mini-USB(M) to RJ45(M), 8ft, Brocade Console Cable, + 9pin(F) to RJ45(F) adapter | eBay
Speaking of the 7250 and serial communication...
I just got a 7250-24P off of eBay. I also purchased the DB9 female to USB mini console cable that vangoose linked previously. However, I cannot get any response from the switch (serial or otherwise). The power LED and ID #1 LEDs turn on, but nothing else. The fans spin high for a minute or so before slowing down, so it seems to be booting in some manner.
I've tried multiple serial ports at the other end, and I even double-checked the continuity on the cable pins. The only potential issue I can see from the cable is that it connects pins 1 and 4 on the mini USB end to the RTS/CTS flow control pins on the DB9 side, and everyone seems to agree that there is no hardware flow control on the 7250/7450 mini USB connectors. I wouldn't think that would prevent communication, though.
If I hook up a network cable to the administrative ethernet port, it doesn't connect. If I hook it up to one of the other ports, I get a connection, but nothing useful (within my limited experience troubleshooting switch connections). It doesn't pull an IP address from DHCP, and the Mikrotik switch it's attached to identifies it as an "edge" connection with no MAC addresses showing up in that connection.
Could this be a semi-dead switch, or is the cable the likely issue? Anyone had similar problems?
Edit: To eliminate the cable a bit more, I manually bridged only pins 2, 3, and 5 from the DB9 side of the cable to a serial port, bypassing the RTS/CTS pins. Still no communication.
I just got a 7250-24P off of eBay. I also purchased the DB9 female to USB mini console cable that vangoose linked previously. However, I cannot get any response from the switch (serial or otherwise). The power LED and ID #1 LEDs turn on, but nothing else. The fans spin high for a minute or so before slowing down, so it seems to be booting in some manner.
I've tried multiple serial ports at the other end, and I even double-checked the continuity on the cable pins. The only potential issue I can see from the cable is that it connects pins 1 and 4 on the mini USB end to the RTS/CTS flow control pins on the DB9 side, and everyone seems to agree that there is no hardware flow control on the 7250/7450 mini USB connectors. I wouldn't think that would prevent communication, though.
If I hook up a network cable to the administrative ethernet port, it doesn't connect. If I hook it up to one of the other ports, I get a connection, but nothing useful (within my limited experience troubleshooting switch connections). It doesn't pull an IP address from DHCP, and the Mikrotik switch it's attached to identifies it as an "edge" connection with no MAC addresses showing up in that connection.
Could this be a semi-dead switch, or is the cable the likely issue? Anyone had similar problems?
Edit: To eliminate the cable a bit more, I manually bridged only pins 2, 3, and 5 from the DB9 side of the cable to a serial port, bypassing the RTS/CTS pins. Still no communication.
Last edited:
I faced a similar issue with a wiped clean switch ... it's likely that the RJ45 Ethernet port has not been set up.
Plug in two devices to different ports and give them static IP addresses on the same subnet and see if they can talk to one another; if they can, the switch is probably fine.
I'm rather new to this switch, but for me, here's what I did ...
I purchased the cable I linked above and got a Tripplite/Keyspan serial to USB device and bought the "Serial" app from the App Store (this was the easiest way to get it working on my version of OSX as the install CD didn't install the right driver).
The first time I connected via Serial, it came right up, but subsequent attempts required a bunch of finagling and unplugging/replugging of both ends of the cable (to the switch/to the laptop's usb port) and sometimes the console wouldn't display the console without typing in a command first (e.g., it'd be a blank screen, but if you typed '?' it would display the help message). This may have been due to the computer falling asleep with the serial connection active, but I'm not certain.
Anyone know why the brocade switches (icx-6430-C12 ) would have issues displaying the web gui ? It's enable and have setup the user & aaa authentication login default local aaa authentication enable default local aaa authentication web default local commands but it only shows the port display. I can't get to any configuration menu's.
Bought a pair of them for Christmas, I have it up and running just having some little issues with vlans so i wanted to log into the GUI and see what i might be missing.
TIA !
Bought a pair of them for Christmas, I have it up and running just having some little issues with vlans so i wanted to log into the GUI and see what i might be missing.
TIA !
Attachments
Had a tickle of suspicion, so I tried reversing the send and receive pins (2 and 3) on my manual DB9 bridge and booted the switch again. I started getting text instantly!
So, as a heads up: do not buy the DB9 to Mini USB cable linked to by vangoose above (current listing here, sold by "tidunkin2012"). I'm going to work with the seller and I'll report back what they say.
Need help understanding routing issue. I have bought ICX6610 off of eBay to replace Catalyst 3750G switch. I have configured ICX to be a layer 3 switch with OSPF routing between it the the router. Everything works as expected, except the fact that ICX itself cannot access the internet. I can ping internal clients as well as VPN subnets without issues but not the internet hosts. All other local clients have no issues accessing anything at all they are configured to access.
interface ethernet 1/3/1
port-name Uplink to EdgeRoute6 on port eth5
ip address 192.168.29.1 255.255.255.0
ip ospf area 0
SSH@ICX6610-48P#sh ip route
Total number of IP routes: 17
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 192.168.29.254 e 1/3/1 1/1 S 2d23h
2 10.0.0.0/8 DIRECT ve 100 0/0 D 3d2h
3 172.16.31.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
4 172.16.32.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
5 172.16.33.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
6 172.16.34.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
7 172.16.35.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
8 192.168.0.0/24 DIRECT ve 2 0/0 D 2d23h
9 192.168.1.0/24 192.168.29.254 e 1/3/1 110/21 O 2d23h
10 192.168.3.0/24 DIRECT ve 3 0/0 D 3d2h
11 192.168.7.0/24 192.168.29.254 e 1/3/1 110/12 O 2d23h
12 192.168.11.0/24 DIRECT ve 11 0/0 D 3d2h
13 192.168.12.0/24 DIRECT ve 12 0/0 D 3d2h
14 192.168.23.0/24 192.168.29.254 e 1/3/1 110/21 O 2d23h
15 192.168.29.0/24 DIRECT e 1/3/1 0/0 D 2d23h
16 192.168.35.0/24 DIRECT ve 35 0/0 D 3d2h
17 192.168.254.0/24 DIRECT ve 254 0/0 D 3d2h
SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#traceroute 8.8.8.8
Type Control-c to abort
Tracing the route to IP node 8.8.8.8(8.8.8.8) from 1 to 30 hops
1 <1 ms <1 ms <1 ms er-6p [192.168.29.254]
2 * * * ?
3 * * * ?
4 * * * ?
5 * * * ?
6 * ^C
Trace Route aborted!
interface ethernet 1/3/1
port-name Uplink to EdgeRoute6 on port eth5
ip address 192.168.29.1 255.255.255.0
ip ospf area 0
SSH@ICX6610-48P#sh ip route
Total number of IP routes: 17
Type Codes - B:BGP D:Connected O:OSPF R:RIP S:Static; Cost - Dist/Metric
BGP Codes - i:iBGP e:eBGP
OSPF Codes - i:Inter Area 1:External Type 1 2:External Type 2
Destination Gateway Port Cost Type Uptime
1 0.0.0.0/0 192.168.29.254 e 1/3/1 1/1 S 2d23h
2 10.0.0.0/8 DIRECT ve 100 0/0 D 3d2h
3 172.16.31.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
4 172.16.32.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
5 172.16.33.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
6 172.16.34.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
7 172.16.35.0/30 192.168.29.254 e 1/3/1 110/11 O 2d23h
8 192.168.0.0/24 DIRECT ve 2 0/0 D 2d23h
9 192.168.1.0/24 192.168.29.254 e 1/3/1 110/21 O 2d23h
10 192.168.3.0/24 DIRECT ve 3 0/0 D 3d2h
11 192.168.7.0/24 192.168.29.254 e 1/3/1 110/12 O 2d23h
12 192.168.11.0/24 DIRECT ve 11 0/0 D 3d2h
13 192.168.12.0/24 DIRECT ve 12 0/0 D 3d2h
14 192.168.23.0/24 192.168.29.254 e 1/3/1 110/21 O 2d23h
15 192.168.29.0/24 DIRECT e 1/3/1 0/0 D 2d23h
16 192.168.35.0/24 DIRECT ve 35 0/0 D 3d2h
17 192.168.254.0/24 DIRECT ve 254 0/0 D 3d2h
SSH@ICX6610-48P#ping 8.8.8.8
Sending 1, 16-byte ICMP Echo to 8.8.8.8, timeout 5000 msec, TTL 64
Type Control-c to abort
Request timed out.
No reply from remote host.
SSH@ICX6610-48P#traceroute 8.8.8.8
Type Control-c to abort
Tracing the route to IP node 8.8.8.8(8.8.8.8) from 1 to 30 hops
1 <1 ms <1 ms <1 ms er-6p [192.168.29.254]
2 * * * ?
3 * * * ?
4 * * * ?
5 * * * ?
6 * ^C
Trace Route aborted!
Last edited:
Looking at the switch side, it has the correct routing table and on a traceroute the first hop is your edgerouter... Are you sure that the configuration on the ER is all right? Firewall rules, NAT/Masquerading,...?
EDIT: Also, keep in mind that the switch sends the packets with (if I remember correctly) the address of the lowest numbered loopback interface or (in case there are no loopbacks), the lowest VE; in the ER, you need to allow and properly route/masquerade the traffic from that source address...
Last edited:
Thank you for the response. I am not using ve on the Brocade side. I have issued enable command on the eth 1/3/1, assigned in an IP address and enabled OSPF on it. I figured that it is easier than messing with vlans and virtual interfaces. On the Edgerouter side I have not done any changes what so ever. It looks like the Edge router does not have the route back to Brocade for traffic destined for the switch. It is a point to point connection. So I am at a loss here. Not that I really need the switch to access the internet directly, but having the time sync would be good.
P.S. I have not configured loop back interface on the Brocade. Lowest VE interface would be ve 2. I'm not using vlan 1 at all. There is OSFP running on ve 2 interface. So it should be routing correctly...
There are all the VEs I have configured:
interface ve 2
ip address 192.168.0.11 255.255.255.0
ip ospf area 0
!
interface ve 3
ip address 192.168.3.1 255.255.255.0
ip helper-address 1 192.168.35.4
ip ospf area 0
!
interface ve 11
ip address 192.168.11.1 255.255.255.0
!
interface ve 12
ip address 192.168.12.1 255.255.255.0
ip helper-address 1 192.168.35.4
ip ospf area 0
!
interface ve 35
ip address 192.168.35.1 255.255.255.0
ip ospf area 0
!
interface ve 100
ip address 10.0.0.1 255.0.0.0
!
interface ve 254
ip address 192.168.254.1 255.255.255.0
Last edited: