Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

cap

New Member
Sep 20, 2021
12
14
3
California
I thought I'd introduce myself. I got intoxicated by this thread, and after reading the first 50 pages of it, I drunk-purchased a new sealed ICX 7250-24 through ebay. I cannot express how much I do not need this switch! Nevertheless it is super cool and I could use some advice, request below.

It came with software 08.0.30eT211 and bootloader 10.1.05T215 (Mar 19 2015 - 16:39:20). There is a post a few pages back in which @fohdeesha explains that this bootloader is too old to flash the 8090 firmware recommended in the guide. That post contains instructions to flash a newer bootloader first. I believe there is a small error in those commands. It should be setenv uboot instead of setenv image_name, so in total:
Code:
setenv ipaddr 192.168.1.50
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 192.168.1.8
setenv uboot ICX7xxx/spz10118.bin
update_uboot
reset
#smash b to get back into new bootloader
That got me going and I am now running 08.0.95dT213.

Between ordering and receiving the switch, I have read (but not memorized) the entire thread. After my item shipped (of course), I learned that it lacks two capabilities that I imagined it would have, and that made the idea of owning this switch so appealing to me:
  • I dreamt that this could be my sole piece of networking equipment aside from my cable modem and wireless access points. But I learned that it does not do NAT. I now understand that this is absent from most Layer-3 switches. My bad.
  • I believed I would be able to interface to my ISP's dreadful router on the router's 2.5gbe port, via the Mikrotik S+RJ10 SFP+ module. That question has been thoroughly explored in the thread and now I see that the whole idea is a no-go. (I purchased a couple of S+RJ10 modules and I can't get them to work at all on the ICX 7250. Through show media the vendor is shown, but they are recognized as SFP rather than SFP+. I could not get a link even when connected to a 1GB device on the other end, and having configured speed-duplex on the interface. Oh well, something for the parts bin. I might use them somewhere someday.)
This leaves me needing some method to connect to my ISP's router at 2.5gbe. I am open to using a suitable router for that task, but at the risk of having to hand over my nerd card, I confess that at the moment I am letting the ISP's router handle NAT (which it seems good at) and even DHCP and DNS (at which it is very poor in the feature and UI departments). I had been using a small Protectli device running pfSense until my WAN speed outstripped the ability of my pfSense box to keep up.

I basically need nothing more than a media converter to go between 10gbe SFP+ and 2.5gbe RJ45. This is where I would like your collective opinions. Some choices:
  • Netgear MS510TX switch for US$270. It has a long track record and good user reports for this exact purpose, passing traffic between 10gbe and 2.5gbe without getting all knotted up. It is rack-mountable, has an internal power supply, and is quiet enough. But it pains me to spend new money on a device that is four years old.
  • Qnap QSW-2104-2S-US switch for US$140. This is a consumer desktop switch with 2 SFP+ ports and 4 2.5G RJ45 ports. It is so new that I have not seen a single substantial review of it. No telling if it can reliably do the job I would ask of it. Other disadvantages for me are that it is not rack mountable, and that it is passively cooled. My equipment is in my garage, and it does get extremely hot in there in summer.
  • Mikrotik RB5009UG+S+IN for about US$200. Another brand new product. This is a router so I could use it in ways more flexible than the above switches, so it's not apples–apples comparison. Based on reading threads like this I'm also wary of introducing Mikrotik into my critical path.
I wonder what y'all think about interfacing 10gbe to 2.5gbe.
 
Last edited:

NateS

Active Member
Apr 19, 2021
159
91
28
Sacramento, CA, US
This leaves me needing some method to connect to my ISP's router at 2.5gbe. I am open to using a suitable router for that task, but at the risk of having to hand over my nerd card, I confess that at the moment I am letting the ISP's router handle NAT (which it seems good at) and even DHCP and DNS (at which it is very poor in the feature and UI departments). I had been using a small Protectli device running pfSense until my WAN speed outstripped the ability of my pfSense box to keep up.
Another option would be to get a newer pfSense (or opnSense) box, and put in two network cards, a 10g SFP+ and a multigig NbaseT. This would likely cost about the same or less than the switch options you listed, but it would also be a much more capable router/NAT/DNS/DHCP/etc. box than your ISP's one.
 
  • Like
Reactions: fohdeesha

cap

New Member
Sep 20, 2021
12
14
3
California
My internet connection peaks at just over 1400Mbps download speed with a direct connection between my desktop and the ISP router's 2.5gbe port, through a USB 2.5gbe dongle. Through my old pfSense box, I was seeing peaks barely over 800Mbps. So it was a 75% improvement I was leaving on the table if I changed nothing.

I'm the first to admit that I need not do anything. I have some regrets about choosing now as the time to upgrade my network infrastructure, when the available improvement isn't going to change my life in any quantifiable way. I guess I was bored and wanted to play with some new (to me) hardware. My core switch has been a Dell 2816 that I bought new 15 years ago.
 
  • Like
Reactions: noduck

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,082
113
33
fohdeesha.com
I thought I'd introduce myself. I got intoxicated by this thread, and after reading the first 50 pages of it, I drunk-purchased a new sealed ICX 7250-24 through ebay. I cannot express how much I do not need this switch! Nevertheless it is super cool and I could use some advice, request below.

It came with software 08.0.30eT211 and bootloader 10.1.05T215 (Mar 19 2015 - 16:39:20). There is a post a few pages back in which @fohdeesha explains that this bootloader is too old to flash the 8090 firmware recommended in the guide. That post contains instructions to flash a newer bootloader first. I believe there is a small error in those commands. It should be setenv uboot instead of setenv image_name, so in total:
Code:
setenv ipaddr 192.168.1.50
setenv netmask 255.255.255.0

#tell the switch the IP of your tftp server:
setenv serverip 192.168.1.8
setenv uboot ICX7xxx/spz10118.bin
update_uboot
reset
#smash b to get back into new bootloader
That got me going and I am now running 08.0.95dT213.

Between ordering and receiving the switch, I have read (but not memorized) the entire thread. After my item shipped (of course), I learned that it lacks two capabilities that I imagined it would have, and that made the idea of owning this switch so appealing to me:
  • I dreamt that this could be my sole piece of networking equipment aside from my cable modem and wireless access points. But I learned that it does not do NAT. I now understand that this is absent from most Layer-3 switches. My bad.
  • I believed I would be able to interface to my ISP's dreadful router on the router's 2.5gbe port, via the Mikrotik S+RJ10 SFP+ module. That question has been thoroughly explored in the thread and now I see that the whole idea is a no-go. (I purchased a couple of S+RJ10 modules and I can't get them to work at all on the ICX 7250. Through show media the vendor is shown, but they are recognized as SFP rather than SFP+. I could not get a link even when connected to a 1GB device on the other end, and having configured speed-duplex on the interface. Oh well, something for the parts bin. I might use them somewhere someday.)
This leaves me needing some method to connect to my ISP's router at 2.5gbe. I am open to using a suitable router for that task, but at the risk of having to hand over my nerd card, I confess that at the moment I am letting the ISP's router handle NAT (which it seems good at) and even DHCP and DNS (at which it is very poor in the feature and UI departments). I had been using a small Protectli device running pfSense until my WAN speed outstripped the ability of my pfSense box to keep up.

I basically need nothing more than a media converter to go between 10gbe SFP+ and 2.5gbe RJ45. This is where I would like your collective opinions. Some choices:
  • Netgear MS510TX switch for US$270. It has a long track record and good user reports for this exact purpose, passing traffic between 10gbe and 2.5gbe without getting all knotted up. It is rack-mountable, has an internal power supply, and is quiet enough. But it pains me to spend new money on a device that is four years old.
  • Qnap QSW-2104-2S-US switch for US$140. This is a consumer desktop switch with 2 SFP+ ports and 4 2.5G RJ45 ports. It is so new that I have not seen a single substantial review of it. No telling if it can reliably do the job I would ask of it. Other disadvantages for me are that it is not rack mountable, and that it is passively cooled. My equipment is in my garage, and it does get extremely hot in there in summer.
  • Mikrotik RB5009UG+S+IN for about US$200. Another brand new product. This is a router so I could use it in ways more flexible than the above switches, so it's not apples–apples comparison. Based on reading threads like this I'm also wary of introducing Mikrotik into my critical path.
I wonder what y'all think about interfacing 10gbe to 2.5gbe.

woops! good catch, must have been half asleep when I wrote that. edited the post (and the instructions will be on the actual guide site soon)

as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now https://www.planet.com.tw/en/product/xt-705a

they go for about $170. of course you'll still need a firewall as L3 witches don't do anything like NAT as you've found out
 

cap

New Member
Sep 20, 2021
12
14
3
California
as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now https://www.planet.com.tw/en/product/xt-705a

they go for about $170.
Nice! I looked around a bit and discovered that Optcore has a version too. The photo shows the 1G version but the description describes it as 10G. It's $109 here. 10GBASE-T Copper to 10G SFP+ Fiber Media Converter | Optcore.net
 

Sean Jones

New Member
Feb 16, 2018
8
0
1
41
woops! good catch, must have been half asleep when I wrote that. edited the post (and the instructions will be on the actual guide site soon)

as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now XT-705A - Standard 10G Media Converter - PLANET Technology

they go for about $170. of course you'll still need a firewall as L3 witches don't do anything like NAT as you've found out
Has anyone tried the AQS-107 based SFP+ modules like Supermicro (Aquantia) 10G SFP+ to RJ45 10GBASE-T Optical Transceivers (AOM-AQS-107-B0C2-CX) ? My understanding is that the AQS-107 chip provides the retiming and buffering capabilities so that it connects to the switch at 10Gbps and to the ethernet at the other rates.
 

Dave Corder

Active Member
Dec 21, 2015
291
186
43
41
Has anyone tried the AQS-107 based SFP+ modules like Supermicro (Aquantia) 10G SFP+ to RJ45 10GBASE-T Optical Transceivers (AOM-AQS-107-B0C2-CX) ? My understanding is that the AQS-107 chip provides the retiming and buffering capabilities so that it connects to the switch at 10Gbps and to the ethernet at the other rates.
Can't speak for using that module in an ICX switch, but I'm using one in my pfSense firewall in an older Chelsio dual SFP+ 1Gbps/10Gbps "only" NIC to my Xfinity XB7's 2.5 Gbps port and it's working fine.
 

ZFSZealot

New Member
Aug 16, 2021
26
6
3
Newb question. Snip from the output of show interfaces. What do the egress queue numbers on the bottom mean, especially the "Dropped Packets"? I'm noticing hiccups here and there after switching my lab over to a 6610.

Code:
  10GigabitEthernet 1/2/7 is up, line protocol is up
  Port up for 2 day(s) 1 hour(s) 20 minute(s) 4 second(s)
  Hardware is   10GigabitEthernet , address is 748e.f8e9.a6d0 (bia 748e.f8e9.a707)
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of 10 L2 VLANs, port is tagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to OFF, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Member of active trunk ports 1/2/7,1/2/8, primary port is 1/2/7
  Member of configured trunk ports 1/2/7,1/2/8, primary port is 1/2/7
  Port name is Trunk vSphere Umbra 10G
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 156920128 bits/sec, 14018 packets/sec, 1.58% utilization
  300 second output rate: 86124288 bits/sec, 8181 packets/sec, 0.86% utilization
  508950627 packets input, 649415570695 bytes, 0 no buffer
  Received 8433 broadcasts, 45939 multicasts, 508896255 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  268988240 packets output, 250275068794 bytes, 0 underruns       
  Transmitted 269561 broadcasts, 416566 multicasts, 268302113 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0           268283682                  39
    1              525200                   0
    2                2937                   0
    3                   0                   0
    4                   0                   0
    5              150574                   0
    6               25848                   0
    7                   0                   0
 

richtj99

Member
Jul 8, 2017
70
1
8
50
This might be a silly comment but is this a response to me for seeing the network data on the switch? I started to look into it & before I spend a day messing around I wanted to be sure.

I have the 30gb lag & want to make sure I am near capacity (or not). Is the SNMP what I need to see whats happening?

Configure SNMP, get interface statistics that way. Use a package like LibreNMS to graph it or parse the SNMP data into Influx (using Telegraf maybe?) and display with Grafana or other equivalent tool. The on-switch statistics are either instantaneous or a short rolling average, I haven't really dug into them. So an actual monitoring solution so you can look at the averages is the best way to get what you need.
 

LodeRunner

Active Member
Apr 27, 2019
540
227
43
This might be a silly comment but is this a response to me for seeing the network data on the switch? I started to look into it & before I spend a day messing around I wanted to be sure.

I have the 30gb lag & want to make sure I am near capacity (or not). Is the SNMP what I need to see whats happening?
If you want any sort of metrics over time, yes. The switch command you're using shows what is happening at that instant. You could also possibly fuss about with a NetFlow collector. But SNMP is pretty straightforward and lots of tools using it Just Work.
 

Zombielinux

Member
Jun 14, 2019
71
21
8
the PoE board is damaged, seen this twice before. I ended up just removing the POE daughter board and using them as regular switches
So if you can pluck the PoE daughter board out and use them as regular switches, can you install the PoE daughter board into a non-PoE switch and gain PoE capability? Speaking specifically to an ICX6610.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,082
113
33
fohdeesha.com
So if you can pluck the PoE daughter board out and use them as regular switches, can you install the PoE daughter board into a non-PoE switch and gain PoE capability? Speaking specifically to an ICX6610.
theoretically, although I can't remember if the non-poe models have the PoE daughterboard header populated with a socket or not. you'd need PoE power supplies too obviously. in the current market it's probably way cheaper to just buy a complete poe model
 

Zombielinux

Member
Jun 14, 2019
71
21
8
theoretically, although I can't remember if the non-poe models have the PoE daughterboard header populated with a socket or not. you'd need PoE power supplies too obviously. in the current market it's probably way cheaper to just buy a complete poe model
I only ask because I already have the power supplies, fans, and Poe board from a dead icx6610. A PoE model is $70 more at the moment. From my memory, they’re .1” headers too, so a cheapish digikey item.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,729
3,082
113
33
fohdeesha.com
I only ask because I already have the power supplies, fans, and Poe board from a dead icx6610. A PoE model is $70 more at the moment. From my memory, they’re .1” headers too, so a cheapish digikey item.
in that case go for it. they probably do have the headers, I believe they all share the same mobo between models (24 port boards and 48 port boards). if the oe board is from a 24 port and you have a 48 port, or vice versa, obviously that won't work
 

OKGolombRuler

New Member
Mar 13, 2020
21
6
3
I have an ICX7150-24, with a 10g brocade optic 1/3/1 connected to an ICX7150-c12p via another 10g brocade optic and some glass. Stacking immediately worked a treat.

I'd now like to extend the stack to another C12P in another part of the house by way of a couple copper SFPs and a 2.5gbps MOCA point-to-point link. (1/3/3 > Brocade 1G copper SFP > 2.5gbps MOCA bridge isolated (point to point, using it like a media converter > Brocade 1G copper SFP > 1/3/1). Can't get stack interactive to pick up the second c12p. Copper SFP will happily move traffic when moved to 1/3/4 on the -24 but won't 'talk stack' on 1/3/3 over the same rest of the path. Ideas? Here's what the Stupid Idea Fairy has brought me so far:

  • Bad 1/3/3 on my -24. Sucks to be me. Wondering if I can flip this from stacking to regular mode to test the SFP slot.
  • Something in fastiron says, can only stack if link speed 10gbps full duplex - sucks to be me.
  • Something in FastIron says, can stack if SFPs identify at 10gbps regardless of whether the link itself will sustain that speed (would never run in prod this way but it could work fine for the house, this is probably best-case for me).
    • What SFP would you recommend for this application?
    • Would a pair of Mikrotik S+RJ10 6-speed RJ-45 module for up to 10 Gbpss.....
      • a) 'link' at 10gbps full duplex even though they would effectively be rate-limited at 2.5gbps, or
      • b) autonegotiate to 2.5gbps?
      • In the latter case will the 7150 agree to stack with this port? I like accumulating toys but don't need $280 of unusable SFPs at the moment.

Ideally I'd bracket every MOCA bridge PTP link with SRJ10s, the 7150-24 on one end and a C12P on the other, stack everything into one logical switch and be informed, fat, and happy. Experienced Brocade Brigade-- how far am I from the Dream? :)

I guess, failing all that, anybody got any good tips for pulling fiber through lots of EDIT: OLD construction? Anything rated to cheat through HVAC duct?
 
Last edited:

jei

Active Member
Aug 8, 2021
150
77
28
Finland
I have an ICX7150-24, with a 10g brocade optic 1/3/1 connected to an ICX7150-c12p via another 10g brocade optic and some glass. Stacking immediately worked a treat.

I'd now like to extend the stack to another C12P in another part of the house by way of a couple copper SFPs and a 2.5gbps MOCA point-to-point link. (1/3/3 > Brocade 1G copper SFP > 2.5gbps MOCA bridge isolated (point to point, using it like a media converter > Brocade 1G copper SFP > 1/3/1). Can't get stack interactive to pick up the second c12p. Copper SFP will happily move traffic when moved to 1/3/4 on the -24 but won't 'talk stack' on 1/3/3 over the same rest of the path. Ideas? Here's what the Stupid Idea Fairy has brought me so far:

  • Bad 1/3/3 on my -24. Sucks to be me. Wondering if I can flip this from stacking to regular mode to test the SFP slot.
  • Something in fastiron says, can only stack if link speed 10gbps full duplex - sucks to be me.
  • Something in FastIron says, can stack if SFPs identify at 10gbps regardless of whether the link itself will sustain that speed (would never run in prod this way but it could work fine for the house, this is probably best-case for me).
    • What SFP would you recommend for this application?
    • Would a pair of Mikrotik S+RJ10 6-speed RJ-45 module for up to 10 Gbpss.....
      • a) 'link' at 10gbps full duplex even though they would effectively be rate-limited at 2.5gbps, or
      • b) autonegotiate to 2.5gbps?
      • In the latter case will the 7150 agree to stack with this port? I like accumulating toys but don't need $280 of unusable SFPs at the moment.

Ideally I'd bracket every MOCA bridge PTP link with SRJ10s, the 7150-24 on one end and a C12P on the other, stack everything into one logical switch and be informed, fat, and happy. Experienced Brocade Brigade-- how far am I from the Dream? :)

I guess, failing all that, anybody got any good tips for pulling fiber through lots of new construction? Anything rated to cheat through HVAC duct?
Incidentally I was just figuring 6450 -> 7150 connection today and surface OM3 cable was the easiest. It takes a beating and is dirt cheap. Pics:om3_1.jpgom3_2.jpg
 
  • Like
Reactions: OKGolombRuler