Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[Sean Jones]

woops! good catch, must have been half asleep when I wrote that. edited the post (and the instructions will be on the actual guide site soon)

as for a media converter, I had planned on using something like this to adapt from multigig to 10gbase so I can pass it into one of my switches on a vlan, then pass that WAN vlan off to my opnsense router like I am now XT-705A - Standard 10G Media Converter - PLANET Technology

they go for about $170. of course you'll still need a firewall as L3 witches don't do anything like NAT as you've found out

Has anyone tried the AQS-107 based SFP+ modules like Supermicro (Aquantia) 10G SFP+ to RJ45 10GBASE-T Optical Transceivers (AOM-AQS-107-B0C2-CX) ? My understanding is that the AQS-107 chip provides the retiming and buffering capabilities so that it connects to the switch at 10Gbps and to the ethernet at the other rates.

 

[Dave Corder]

Has anyone tried the AQS-107 based SFP+ modules like Supermicro (Aquantia) 10G SFP+ to RJ45 10GBASE-T Optical Transceivers (AOM-AQS-107-B0C2-CX) ? My understanding is that the AQS-107 chip provides the retiming and buffering capabilities so that it connects to the switch at 10Gbps and to the ethernet at the other rates.

Can't speak for using that module in an ICX switch, but I'm using one in my pfSense firewall in an older Chelsio dual SFP+ 1Gbps/10Gbps "only" NIC to my Xfinity XB7's 2.5 Gbps port and it's working fine.

 

[ZFSZealot]

Newb question. Snip from the output of show interfaces. What do the egress queue numbers on the bottom mean, especially the "Dropped Packets"? I'm noticing hiccups here and there after switching my lab over to a 6610.

  10GigabitEthernet 1/2/7 is up, line protocol is up
  Port up for 2 day(s) 1 hour(s) 20 minute(s) 4 second(s)
  Hardware is   10GigabitEthernet , address is 748e.f8e9.a6d0 (bia 748e.f8e9.a707)
  Configured speed 10Gbit, actual 10Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual none
  Member of 10 L2 VLANs, port is tagged, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to OFF, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Member of active trunk ports 1/2/7,1/2/8, primary port is 1/2/7
  Member of configured trunk ports 1/2/7,1/2/8, primary port is 1/2/7
  Port name is Trunk vSphere Umbra 10G
  MTU 1500 bytes, encapsulation ethernet
  300 second input rate: 156920128 bits/sec, 14018 packets/sec, 1.58% utilization
  300 second output rate: 86124288 bits/sec, 8181 packets/sec, 0.86% utilization
  508950627 packets input, 649415570695 bytes, 0 no buffer
  Received 8433 broadcasts, 45939 multicasts, 508896255 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  268988240 packets output, 250275068794 bytes, 0 underruns       
  Transmitted 269561 broadcasts, 416566 multicasts, 268302113 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0           268283682                  39
    1              525200                   0
    2                2937                   0
    3                   0                   0
    4                   0                   0
    5              150574                   0
    6               25848                   0
    7                   0                   0

 

[klui]

Flow Control is config enabled, oper enabled, negotiation disabled

It's related to flow control, 802.1p. IEEE P802.1p - Wikipedia

 

[richtj99]

This might be a silly comment but is this a response to me for seeing the network data on the switch? I started to look into it & before I spend a day messing around I wanted to be sure.

I have the 30gb lag & want to make sure I am near capacity (or not). Is the SNMP what I need to see whats happening?

Configure SNMP, get interface statistics that way. Use a package like LibreNMS to graph it or parse the SNMP data into Influx (using Telegraf maybe?) and display with Grafana or other equivalent tool. The on-switch statistics are either instantaneous or a short rolling average, I haven't really dug into them. So an actual monitoring solution so you can look at the averages is the best way to get what you need.

 

[LodeRunner]

This might be a silly comment but is this a response to me for seeing the network data on the switch? I started to look into it & before I spend a day messing around I wanted to be sure.

I have the 30gb lag & want to make sure I am near capacity (or not). Is the SNMP what I need to see whats happening?

If you want any sort of metrics over time, yes. The switch command you're using shows what is happening at that instant. You could also possibly fuss about with a NetFlow collector. But SNMP is pretty straightforward and lots of tools using it Just Work.

 

[Zombielinux]

the PoE board is damaged, seen this twice before. I ended up just removing the POE daughter board and using them as regular switches

So if you can pluck the PoE daughter board out and use them as regular switches, can you install the PoE daughter board into a non-PoE switch and gain PoE capability? Speaking specifically to an ICX6610.

 

[fohdeesha]

So if you can pluck the PoE daughter board out and use them as regular switches, can you install the PoE daughter board into a non-PoE switch and gain PoE capability? Speaking specifically to an ICX6610.

theoretically, although I can't remember if the non-poe models have the PoE daughterboard header populated with a socket or not. you'd need PoE power supplies too obviously. in the current market it's probably way cheaper to just buy a complete poe model

 

[Zombielinux]

theoretically, although I can't remember if the non-poe models have the PoE daughterboard header populated with a socket or not. you'd need PoE power supplies too obviously. in the current market it's probably way cheaper to just buy a complete poe model

I only ask because I already have the power supplies, fans, and Poe board from a dead icx6610. A PoE model is $70 more at the moment. From my memory, they’re .1” headers too, so a cheapish digikey item.

 

[fohdeesha]

I only ask because I already have the power supplies, fans, and Poe board from a dead icx6610. A PoE model is $70 more at the moment. From my memory, they’re .1” headers too, so a cheapish digikey item.

in that case go for it. they probably do have the headers, I believe they all share the same mobo between models (24 port boards and 48 port boards). if the oe board is from a 24 port and you have a 48 port, or vice versa, obviously that won't work

 

[OKGolombRuler]

I have an ICX7150-24, with a 10g brocade optic 1/3/1 connected to an ICX7150-c12p via another 10g brocade optic and some glass. Stacking immediately worked a treat.

I'd now like to extend the stack to another C12P in another part of the house by way of a couple copper SFPs and a 2.5gbps MOCA point-to-point link. (1/3/3 > Brocade 1G copper SFP > 2.5gbps MOCA bridge isolated (point to point, using it like a media converter > Brocade 1G copper SFP > 1/3/1). Can't get stack interactive to pick up the second c12p. Copper SFP will happily move traffic when moved to 1/3/4 on the -24 but won't 'talk stack' on 1/3/3 over the same rest of the path. Ideas? Here's what the Stupid Idea Fairy has brought me so far:

• Bad 1/3/3 on my -24. Sucks to be me. Wondering if I can flip this from stacking to regular mode to test the SFP slot.
• Something in fastiron says, can only stack if link speed 10gbps full duplex - sucks to be me.
• Something in FastIron says, can stack if SFPs identify at 10gbps regardless of whether the link itself will sustain that speed (would never run in prod this way but it could work fine for the house, this is probably best-case for me).
  • What SFP would you recommend for this application?
  • Would a pair of Mikrotik S+RJ10 6-speed RJ-45 module for up to 10 Gbpss.....
    • a) 'link' at 10gbps full duplex even though they would effectively be rate-limited at 2.5gbps, or
    • b) autonegotiate to 2.5gbps?
    • In the latter case will the 7150 agree to stack with this port? I like accumulating toys but don't need $280 of unusable SFPs at the moment.

Ideally I'd bracket every MOCA bridge PTP link with SRJ10s, the 7150-24 on one end and a C12P on the other, stack everything into one logical switch and be informed, fat, and happy. Experienced Brocade Brigade-- how far am I from the Dream?

I guess, failing all that, anybody got any good tips for pulling fiber through lots of EDIT: OLD construction? Anything rated to cheat through HVAC duct?

 

[jei]

I have an ICX7150-24, with a 10g brocade optic 1/3/1 connected to an ICX7150-c12p via another 10g brocade optic and some glass. Stacking immediately worked a treat.

I'd now like to extend the stack to another C12P in another part of the house by way of a couple copper SFPs and a 2.5gbps MOCA point-to-point link. (1/3/3 > Brocade 1G copper SFP > 2.5gbps MOCA bridge isolated (point to point, using it like a media converter > Brocade 1G copper SFP > 1/3/1). Can't get stack interactive to pick up the second c12p. Copper SFP will happily move traffic when moved to 1/3/4 on the -24 but won't 'talk stack' on 1/3/3 over the same rest of the path. Ideas? Here's what the Stupid Idea Fairy has brought me so far:

• Bad 1/3/3 on my -24. Sucks to be me. Wondering if I can flip this from stacking to regular mode to test the SFP slot.
• Something in fastiron says, can only stack if link speed 10gbps full duplex - sucks to be me.
• Something in FastIron says, can stack if SFPs identify at 10gbps regardless of whether the link itself will sustain that speed (would never run in prod this way but it could work fine for the house, this is probably best-case for me).
  • What SFP would you recommend for this application?
  • Would a pair of Mikrotik S+RJ10 6-speed RJ-45 module for up to 10 Gbpss.....
    • a) 'link' at 10gbps full duplex even though they would effectively be rate-limited at 2.5gbps, or
    • b) autonegotiate to 2.5gbps?
    • In the latter case will the 7150 agree to stack with this port? I like accumulating toys but don't need $280 of unusable SFPs at the moment.

Ideally I'd bracket every MOCA bridge PTP link with SRJ10s, the 7150-24 on one end and a C12P on the other, stack everything into one logical switch and be informed, fat, and happy. Experienced Brocade Brigade-- how far am I from the Dream?

I guess, failing all that, anybody got any good tips for pulling fiber through lots of new construction? Anything rated to cheat through HVAC duct?

Incidentally I was just figuring 6450 -> 7150 connection today and surface OM3 cable was the easiest. It takes a beating and is dirt cheap. Pics:

 

[nickf1227]

ehhhhhhh
I don't think the fiber will last like that....

 

[jei]

You might be right! But the one on the wooden floor has been there for over a year, been stepped on every day, no problems yet. New cable costs ~10€ so it's by far the cheapeast option for me. If there's ever any work done with the walls, got to drop some fiber there at the same time.

 

[EngChiSTH]

Anyone knows on the power consumption difference between 7250-24 vs 7250-48 ? I am now needing >4 SFP+ plugins and overgrown my 6450, not sure if I need 48 gig ports and wondering about power consumption. thank you

 

[Freebsd1976]

my 7250-24 power consumption on idle 46-50W , don't know 48 port

 

[pir2]

I have an ICX7150-24, with a 10g brocade optic 1/3/1 connected to an ICX7150-c12p via another 10g brocade optic and some glass. Stacking immediately worked a treat.

I'd now like to extend the stack to another C12P in another part of the house by way of a couple copper SFPs and a 2.5gbps MOCA point-to-point link. (1/3/3 > Brocade 1G copper SFP > 2.5gbps MOCA bridge isolated (point to point, using it like a media converter > Brocade 1G copper SFP > 1/3/1). Can't get stack interactive to pick up the second c12p. Copper SFP will happily move traffic when moved to 1/3/4 on the -24 but won't 'talk stack' on 1/3/3 over the same rest of the path. Ideas? Here's what the Stupid Idea Fairy has brought me so far:

• Bad 1/3/3 on my -24. Sucks to be me. Wondering if I can flip this from stacking to regular mode to test the SFP slot.
• Something in fastiron says, can only stack if link speed 10gbps full duplex - sucks to be me.
• Something in FastIron says, can stack if SFPs identify at 10gbps regardless of whether the link itself will sustain that speed (would never run in prod this way but it could work fine for the house, this is probably best-case for me).
  • What SFP would you recommend for this application?
  • Would a pair of Mikrotik S+RJ10 6-speed RJ-45 module for up to 10 Gbpss.....
    • a) 'link' at 10gbps full duplex even though they would effectively be rate-limited at 2.5gbps, or
    • b) autonegotiate to 2.5gbps?
    • In the latter case will the 7150 agree to stack with this port? I like accumulating toys but don't need $280 of unusable SFPs at the moment.

Ideally I'd bracket every MOCA bridge PTP link with SRJ10s, the 7150-24 on one end and a C12P on the other, stack everything into one logical switch and be informed, fat, and happy. Experienced Brocade Brigade-- how far am I from the Dream?

I guess, failing all that, anybody got any good tips for pulling fiber through lots of new construction? Anything rated to cheat through HVAC duct?

how long are your copper cables? there's a distance limitation when using copper.

Commscope Technical Content Portal

docs.commscope.com

 

[dragonian]

Anyone knows on the power consumption difference between 7250-24 vs 7250-48 ? I am now needing >4 SFP+ plugins and overgrown my 6450, not sure if I need 48 gig ports and wondering about power consumption. thank you

Since I was moving some stuff around in the closet today, I did a little power measurement.
Using killawatt:
7250-48 - 31W idle (only console)
7250-24p - 45W idle (only console)

Before the poe was initialized i saw 35W on the 24p.


Edit: I realize that the -48 was previously a stack member, and not talking to the master.. maybe that 31W is not realistic?
I plugged in 2 SFP+ DACs to rebuild the stack, and the it's currently drawing 45W. Not sure how much of that addtional draw is the 2 DACs vs the full stack participation.

 

[Thorzeen]

Hey everyone,
Is anyone here running "Oxidized" on the ICX 6610 to backup the config ? I am have issues with getting it run on this switch and so I thought I would ask here on the thread where I have learned so much about this device.

I have "Oxidized" running on Librenms.

Thanks.

 

[EngChiSTH]

Any DAC (direct attach copper) cable recommendations for the Brocade switch or works with anything? I went with AOC earlier however same parts are now more expensive (2 x $20 for each end of the transceiver + cable itself). I was hoping that DAC for short distance (<5 feet) works and looking for any recommendations. thank you

what I already have connected is Brocade 10G-SFPP-SR Compatible 10GBASE-SR SFP+ 850nm 300m DOM LC MMF Transceiver Module and ' LC UPC to LC UPC Duplex OM3 Multimode PVC ' for fiber cable. this works without any issues..