Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

theobassaw

New Member
Sep 30, 2021
6
1
3
I have managed to fix the serial problem. Yet, the licenses are still invalid. Any other suggestions on how to resolve this?

Code:
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Feb 13 2019 at 17:44:29 labeled as ICX64R08030t
                (9868556 bytes) from Primary ICX64R08030t.bin
        SW: Version 08.0.30tT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-24
==========================================================================
UNIT 1: SL 1: ICX6450-24 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6450_BASE_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 10 minute(s) 33 second(s)
The system started at 19:00:30 Eastern Wed Dec 31 1969

 The system : started=warm start         reloaded=by "reload"

Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6450-PREM-LIC-SW    H4CKTH3PLN8    Normal      Invalid    Unlimited         1
2        Node Lock       ICX6450-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         2
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,292
2,383
113
30
fohdeesha.com
I have managed to fix the serial problem. Yet, the licenses are still invalid. Any other suggestions on how to resolve this?

Code:
Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Feb 13 2019 at 17:44:29 labeled as ICX64R08030t
                (9868556 bytes) from Primary ICX64R08030t.bin
        SW: Version 08.0.30tT313
  Boot-Monitor Image size = 786944, Version:10.1.05T310 (kxz10105)
  HW: Stackable ICX6450-24
==========================================================================
UNIT 1: SL 1: ICX6450-24 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6450_BASE_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type DEF0, rev 01
==========================================================================
UNIT 1: SL 2: ICX6450-SFP-Plus 4port 40G Module
==========================================================================
  800 MHz ARM processor ARMv5TE, 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 10 minute(s) 33 second(s)
The system started at 19:00:30 Eastern Wed Dec 31 1969

The system : started=warm start         reloaded=by "reload"

Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
1        Node Lock       ICX6450-PREM-LIC-SW    H4CKTH3PLN8    Normal      Invalid    Unlimited         1
2        Node Lock       ICX6450-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         2
delete all the lics, reboot the switch, then re-import the licenses

if that still doesn't work you might still have a weird space or invisible char in your serial string
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,292
2,383
113
30
fohdeesha.com
That resolved the issue. :oops: I guess the main thing is knowing when to reboot.
under "normal circumstances" you should only have to reboot once (after you change the serial), and that should be it. I think since you had already loaded licenses, and then had a broken serial/LID, the safenet SDK cached the license state of "nope"
 
  • Like
Reactions: tommybackeast

cap

New Member
Sep 20, 2021
7
10
3
I thought I'd introduce myself. I got intoxicated by this thread, and after reading the first 50 pages of it, I drunk-purchased a new sealed ICX 7250-24 through ebay. I cannot express how much I do not need this switch! Nevertheless it is super cool and I could use some advice, request below.

It came with software 08.0.30eT211 and bootloader 10.1.05T215 (Mar 19 2015 - 16:39:20). There is a post a few pages back in which @fohdeesha explains that this bootloader is too old to flash the 8090 firmware recommended in the guide. That post contains instructions to flash a newer bootloader first. I believe there is a small error in those commands. It should be setenv uboot instead of setenv image_name, so in total:
Code:
setenv ipaddr 192.168.1.50
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 192.168.1.8
setenv uboot ICX7xxx/spz10118.bin
update_uboot
reset
#smash b to get back into new bootloader
That got me going and I am now running 08.0.95dT213.

Between ordering and receiving the switch, I have read (but not memorized) the entire thread. After my item shipped (of course), I learned that it lacks two capabilities that I imagined it would have, and that made the idea of owning this switch so appealing to me:
  • I dreamt that this could be my sole piece of networking equipment aside from my cable modem and wireless access points. But I learned that it does not do NAT. I now understand that this is absent from most Layer-3 switches. My bad.
  • I believed I would be able to interface to my ISP's dreadful router on the router's 2.5gbe port, via the Mikrotik S+RJ10 SFP+ module. That question has been thoroughly explored in the thread and now I see that the whole idea is a no-go. (I purchased a couple of S+RJ10 modules and I can't get them to work at all on the ICX 7250. Through show media the vendor is shown, but they are recognized as SFP rather than SFP+. I could not get a link even when connected to a 1GB device on the other end, and having configured speed-duplex on the interface. Oh well, something for the parts bin. I might use them somewhere someday.)
This leaves me needing some method to connect to my ISP's router at 2.5gbe. I am open to using a suitable router for that task, but at the risk of having to hand over my nerd card, I confess that at the moment I am letting the ISP's router handle NAT (which it seems good at) and even DHCP and DNS (at which it is very poor in the feature and UI departments). I had been using a small Protectli device running pfSense until my WAN speed outstripped the ability of my pfSense box to keep up.

I basically need nothing more than a media converter to go between 10gbe SFP+ and 2.5gbe RJ45. This is where I would like your collective opinions. Some choices:
  • Netgear MS510TX switch for US$270. It has a long track record and good user reports for this exact purpose, passing traffic between 10gbe and 2.5gbe without getting all knotted up. It is rack-mountable, has an internal power supply, and is quiet enough. But it pains me to spend new money on a device that is four years old.
  • Qnap QSW-2104-2S-US switch for US$140. This is a consumer desktop switch with 2 SFP+ ports and 4 2.5G RJ45 ports. It is so new that I have not seen a single substantial review of it. No telling if it can reliably do the job I would ask of it. Other disadvantages for me are that it is not rack mountable, and that it is passively cooled. My equipment is in my garage, and it does get extremely hot in there in summer.
  • Mikrotik RB5009UG+S+IN for about US$200. Another brand new product. This is a router so I could use it in ways more flexible than the above switches, so it's not apples–apples comparison. Based on reading threads like this I'm also wary of introducing Mikrotik into my critical path.
I wonder what y'all think about interfacing 10gbe to 2.5gbe.
 
Last edited:

NateS

Active Member
Apr 19, 2021
124
64
28
Sacramento, CA, US
This leaves me needing some method to connect to my ISP's router at 2.5gbe. I am open to using a suitable router for that task, but at the risk of having to hand over my nerd card, I confess that at the moment I am letting the ISP's router handle NAT (which it seems good at) and even DHCP and DNS (at which it is very poor in the feature and UI departments). I had been using a small Protectli device running pfSense until my WAN speed outstripped the ability of my pfSense box to keep up.
Another option would be to get a newer pfSense (or opnSense) box, and put in two network cards, a 10g SFP+ and a multigig NbaseT. This would likely cost about the same or less than the switch options you listed, but it would also be a much more capable router/NAT/DNS/DHCP/etc. box than your ISP's one.
 
  • Like
Reactions: fohdeesha

cap

New Member
Sep 20, 2021
7
10
3
My internet connection peaks at just over 1400Mbps download speed with a direct connection between my desktop and the ISP router's 2.5gbe port, through a USB 2.5gbe dongle. Through my old pfSense box, I was seeing peaks barely over 800Mbps. So it was a 75% improvement I was leaving on the table if I changed nothing.

I'm the first to admit that I need not do anything. I have some regrets about choosing now as the time to upgrade my network infrastructure, when the available improvement isn't going to change my life in any quantifiable way. I guess I was bored and wanted to play with some new (to me) hardware. My core switch has been a Dell 2816 that I bought new 15 years ago.
 
  • Like
Reactions: noduck

fohdeesha

Kaini Industries
Nov 20, 2016
2,292
2,383
113
30
fohdeesha.com
I thought I'd introduce myself. I got intoxicated by this thread, and after reading the first 50 pages of it, I drunk-purchased a new sealed ICX 7250-24 through ebay. I cannot express how much I do not need this switch! Nevertheless it is super cool and I could use some advice, request below.

It came with software 08.0.30eT211 and bootloader 10.1.05T215 (Mar 19 2015 - 16:39:20). There is a post a few pages back in which @fohdeesha explains that this bootloader is too old to flash the 8090 firmware recommended in the guide. That post contains instructions to flash a newer bootloader first. I believe there is a small error in those commands. It should be setenv uboot instead of setenv image_name, so in total:
Code:
setenv ipaddr 192.168.1.50
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 192.168.1.8
setenv uboot ICX7xxx/spz10118.bin
update_uboot
reset
#smash b to get back into new bootloader
That got me going and I am now running 08.0.95dT213.

Between ordering and receiving the switch, I have read (but not memorized) the entire thread. After my item shipped (of course), I learned that it lacks two capabilities that I imagined it would have, and that made the idea of owning this switch so appealing to me:
  • I dreamt that this could be my sole piece of networking equipment aside from my cable modem and wireless access points. But I learned that it does not do NAT. I now understand that this is absent from most Layer-3 switches. My bad.
  • I believed I would be able to interface to my ISP's dreadful router on the router's 2.5gbe port, via the Mikrotik S+RJ10 SFP+ module. That question has been thoroughly explored in the thread and now I see that the whole idea is a no-go. (I purchased a couple of S+RJ10 modules and I can't get them to work at all on the ICX 7250. Through show media the vendor is shown, but they are recognized as SFP rather than SFP+. I could not get a link even when connected to a 1GB device on the other end, and having configured speed-duplex on the interface. Oh well, something for the parts bin. I might use them somewhere someday.)
This leaves me needing some method to connect to my ISP's router at 2.5gbe. I am open to using a suitable router for that task, but at the risk of having to hand over my nerd card, I confess that at the moment I am letting the ISP's router handle NAT (which it seems good at) and even DHCP and DNS (at which it is very poor in the feature and UI departments). I had been using a small Protectli device running pfSense until my WAN speed outstripped the ability of my pfSense box to keep up.

I basically need nothing more than a media converter to go between 10gbe SFP+ and 2.5gbe RJ45. This is where I would like your collective opinions. Some choices:
  • Netgear MS510TX switch for US$270. It has a long track record and good user reports for this exact purpose, passing traffic between 10gbe and 2.5gbe without getting all knotted up. It is rack-mountable, has an internal power supply, and is quiet enough. But it pains me to spend new money on a device that is four years old.
  • Qnap QSW-2104-2S-US switch for US$140. This is a consumer desktop switch with 2 SFP+ ports and 4 2.5G RJ45 ports. It is so new that I have not seen a single substantial review of it. No telling if it can reliably do the job I would ask of it. Other disadvantages for me are that it is not rack mountable, and that it is passively cooled. My equipment is in my garage, and it does get extremely hot in there in summer.
  • Mikrotik RB5009UG+S+IN for about US$200. Another brand new product. This is a router so I could use it in ways more flexible than the above switches, so it's not apples–apples comparison. Based on reading threads like this I'm also wary of introducing Mikrotik into my critical path.
I wonder what y'all think about interfacing 10gbe to 2.5gbe.

woops! good catch, must have been half asleep when I wrote that. edited the post (and the instructions will be on the actual guide site soon)

as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now https://www.planet.com.tw/en/product/xt-705a

they go for about $170. of course you'll still need a firewall as L3 witches don't do anything like NAT as you've found out
 

cap

New Member
Sep 20, 2021
7
10
3
as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now https://www.planet.com.tw/en/product/xt-705a

they go for about $170.
Nice! I looked around a bit and discovered that Optcore has a version too. The photo shows the 1G version but the description describes it as 10G. It's $109 here. 10GBASE-T Copper to 10G SFP+ Fiber Media Converter | Optcore.net
 

Sean Jones

New Member
Feb 16, 2018
7
0
1
39
woops! good catch, must have been half asleep when I wrote that. edited the post (and the instructions will be on the actual guide site soon)

as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now XT-705A - Standard 10G Media Converter - PLANET Technology

they go for about $170. of course you'll still need a firewall as L3 witches don't do anything like NAT as you've found out
Has anyone tried the AQS-107 based SFP+ modules like Supermicro (Aquantia) 10G SFP+ to RJ45 10GBASE-T Optical Transceivers (AOM-AQS-107-B0C2-CX) ? My understanding is that the AQS-107 chip provides the retiming and buffering capabilities so that it connects to the switch at 10Gbps and to the ethernet at the other rates.
 

Dave Corder

Active Member
Dec 21, 2015
160
79
28
39
Has anyone tried the AQS-107 based SFP+ modules like Supermicro (Aquantia) 10G SFP+ to RJ45 10GBASE-T Optical Transceivers (AOM-AQS-107-B0C2-CX) ? My understanding is that the AQS-107 chip provides the retiming and buffering capabilities so that it connects to the switch at 10Gbps and to the ethernet at the other rates.
Can't speak for using that module in an ICX switch, but I'm using one in my pfSense firewall in an older Chelsio dual SFP+ 1Gbps/10Gbps "only" NIC to my Xfinity XB7's 2.5 Gbps port and it's working fine.
 

ZFSZealot

New Member
Aug 16, 2021
20
3
3
Newb question. Snip from the output of show interfaces. What do the egress queue numbers on the bottom mean, especially the "Dropped Packets"? I'm noticing hiccups here and there after switching my lab over to a 6610.

Code:
  10GigabitEthernet 1/2/7 is up, line protocol is up
  Port up for 2 day(s) 1 hour(s) 20 minute(s) 4 second(s)
  Hardware is   10GigabitEthernet , address is 748e.f8e9.a6d0 (bia 748e.f8e9.a707)
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of 10 L2 VLANs, port is tagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to OFF, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Member of active trunk ports 1/2/7,1/2/8, primary port is 1/2/7
  Member of configured trunk ports 1/2/7,1/2/8, primary port is 1/2/7
  Port name is Trunk vSphere Umbra 10G
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 156920128 bits/sec, 14018 packets/sec, 1.58% utilization
  300 second output rate: 86124288 bits/sec, 8181 packets/sec, 0.86% utilization
  508950627 packets input, 649415570695 bytes, 0 no buffer
  Received 8433 broadcasts, 45939 multicasts, 508896255 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  268988240 packets output, 250275068794 bytes, 0 underruns       
  Transmitted 269561 broadcasts, 416566 multicasts, 268302113 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0           268283682                  39
    1              525200                   0
    2                2937                   0
    3                   0                   0
    4                   0                   0
    5              150574                   0
    6               25848                   0
    7                   0                   0
 

richtj99

Member
Jul 8, 2017
56
0
6
48
This might be a silly comment but is this a response to me for seeing the network data on the switch? I started to look into it & before I spend a day messing around I wanted to be sure.

I have the 30gb lag & want to make sure I am near capacity (or not). Is the SNMP what I need to see whats happening?

Configure SNMP, get interface statistics that way. Use a package like LibreNMS to graph it or parse the SNMP data into Influx (using Telegraf maybe?) and display with Grafana or other equivalent tool. The on-switch statistics are either instantaneous or a short rolling average, I haven't really dug into them. So an actual monitoring solution so you can look at the averages is the best way to get what you need.
 

LodeRunner

Active Member
Apr 27, 2019
218
88
28
This might be a silly comment but is this a response to me for seeing the network data on the switch? I started to look into it & before I spend a day messing around I wanted to be sure.

I have the 30gb lag & want to make sure I am near capacity (or not). Is the SNMP what I need to see whats happening?
If you want any sort of metrics over time, yes. The switch command you're using shows what is happening at that instant. You could also possibly fuss about with a NetFlow collector. But SNMP is pretty straightforward and lots of tools using it Just Work.
 

Zombielinux

Member
Jun 14, 2019
30
6
8
the PoE board is damaged, seen this twice before. I ended up just removing the POE daughter board and using them as regular switches
So if you can pluck the PoE daughter board out and use them as regular switches, can you install the PoE daughter board into a non-PoE switch and gain PoE capability? Speaking specifically to an ICX6610.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,292
2,383
113
30
fohdeesha.com
So if you can pluck the PoE daughter board out and use them as regular switches, can you install the PoE daughter board into a non-PoE switch and gain PoE capability? Speaking specifically to an ICX6610.
theoretically, although I can't remember if the non-poe models have the PoE daughterboard header populated with a socket or not. you'd need PoE power supplies too obviously. in the current market it's probably way cheaper to just buy a complete poe model