Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

LodeRunner · Mar 18, 2022

pinkypie said:
Hello LR, trying to understand something. I put the static route in my laptop and can now ping devices on VLAN 2, VLAN 3 and those VLANs can ping VLAN 1. Also, VLAN 2 and VLAN 3 can ping between themselves, however, they could do this before the static route was added.

What I am trying to figure out is why the static route would go on the source device and not the device doing the routing, i.e. Nighthawk or the ICX 6450?

The static route on your laptop is to tell it how to get to the other subnets. Since those subnets are not connected via your Nighthawk, and your Nighthawk is your default gateway, you have to tell your client (source) how it can reach those networks. For devices in VLAN 2 and 3, they're using the ICX as the gateway. The ICX knows what networks are directly connected and can route them; then anything that does not match a direct connected network is sent to the ICX's default gateway, the Nighthawk.

In your specific case, this is required because the Nighthawk and the ICX are in the same subnet.

If your Nighthawk had the ability to do static routes, the more enterprise correct way of setting this up would be

each VLAN has its own subnet (you have this)
each VLAN has a VE with IP
a dedicated VLAN between the ICX and firewall; this VLAN has its own subnet
ICX uses firewall VLAN IP as its upstream gateway
The firewall has reverse routes for each subnet using the ICX IP as its gateway

Theoretically, you could try setting the gateway on your laptop to use the ICX IP and the ICX would send any non-local traffic to the Nighthawk, but the Nighthawk can directly see your client, so return traffic would come direct instead of going back through the ICX leading to asymmetric routing which generally causes issues.

Serhan · Mar 18, 2022

kpfleming said:
That should be fine, OM4 can handle 40GbE. QSFP+ transceivers like this could be used, and they can be purchased with Brocade coding.

Thank you.

Chow · Mar 18, 2022

LodeRunner said:
As I understand the output you provided, the switch natively comes with 24 of the ports licensed for 10G operation and the remaining 24 require POD licensing. So 24 port base license, plus 16 port POD license means you have 40 ports that can be brought online in at 10G, plus the 4x40 G port license. It looks like as ports are brought up, the licenses are first assigned from the base pool; once that is consumed, it will start issuing licenses from the POD pool.

To get all 48 ports active at 10G you'd need to find an additional 8 port POD license.

There was a dedicated thread: Brocade VDX 6740 | ServeTheHome Forums Posters in that thread might have additional help/guidance for you in terms of firmware and licensing issues and how they might be resolved.

Thank´s a lot LodeRunner ;-) I will write there ;-)

Ah ok that´s very cool. I thought, that i have only 16 Ports @ 10 Gbit and 4 Ports on 40 Gbit :-D

But you think a VDX6740-24 Model comes with normaly 24 x 10 Gbit Ports in Standard, yes? And the Licenses goes up on top, right?

hibby50 · Mar 19, 2022

Hey Guys, I bought a 6610 based on this thread and I could use some help. I'm having an issue configuring extended acls. I have a port tagged with multiple vlans (10,20,40,50) going to an access point, and a machine with an untagged LAG on vlan 50. The problem is on my inbound rule (mgmt-in) for vlan 50 I have `permit tcp any any established` And when that rule is in the inbound acl, the other vlans can ssh into the device on the LAG. If I remove it they can't but then it also cant get out

I also tried enable acl-per-port-per-vlan despite not knowing what it means, but that did nothing

Edit: I dumbed it down and put one machine on an untagged vlan 20 port and the other on the LAG untagged vlan 50 port with only 1 acl on the vlan 50 ve: 'permit tcp any any established' and I was able to initiate an ssh session from the vlan 20 machine. Totally stumped.

Code:

SSH@ICX6610-48P-Router(config)#show lag

=== LAG "hibsrv-LAG" ID 2028 (dynamic Deployed) ===

LAG Configuration:
Ports: e 1/3/1 to 1/3/2
Port Count: 2
Primary Port: 1/3/1
Trunk Type: hash-based
LACP Key: 22028
LACP Timeout: long
Deployment: HW Trunk ID 2
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/3/1 Up Forward Full 10G 2028 No 50 0 748e.f8ff.fc4e
1/3/2 Up Forward Full 10G 2028 No 50 0 748e.f8ff.fc4e

Port [Sys P] [Port P] [ Key ] [Act][Tio][Agg][Syn][Col][Dis][Def][Exp][Ope]
1/3/1 1 1 22028 Yes L Agg Syn Col Dis No No Ope
1/3/2 1 1 22028 Yes L Agg Syn Col Dis No No Ope


Partner Info and PDU Statistics
Port Partner Partner LACP LACP
System ID Key Rx Count Tx Count
1/3/1 65535-246e.9600.5080 15 11 9
1/3/2 65535-246e.9600.5080 15 8 8

Code:

SSH@ICX6610-48P-Router(config)#sh run int e 1/1/13

interface ethernet 1/1/13
inline power
!

Code:

SSH@ICX6610-48P-Router(config)#sh vlan br     

PORT-VLAN 10, Name guest, Priority level0, Spanning tree Off

Untagged Ports: None
Tagged Ports: (U1/M1) 13 14 15
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

PORT-VLAN 20, Name iot, Priority level0, Spanning tree Off
Untagged Ports: (U1/M1) 39
Tagged Ports: (U1/M1) 13 14 15
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

!

PORT-VLAN 50, Name mgmt, Priority level0, Spanning tree Off

Untagged Ports: (U1/M1)  37  38
Untagged Ports: (U1/M2) 1 2 3 4 5 6 7 8 9 10
Untagged Ports: (U1/M3) 1 2 3 4 5 6 7 8
Tagged Ports: (U1/M1) 13 14
Uplink Ports: None
DualMode Ports: None
Mac-Vlan Ports: None
Monitoring: Disabled

Code:

Extended IP access list  mgmt-in : 10 entry

ACL Remark:  ALLOW DHCP
permit udp any any eq bootps
permit udp any any eq bootpc
ACL Remark: ALLOW DNS
permit tcp 192.168.50.0 0.0.0.255 host 10.0.0.2 eq dns
permit udp 192.168.50.0 0.0.0.255 host 10.0.0.2 eq dns
ACL Remark: ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
ACL Remark: ALLOW mDNS
permit udp any host 224.0.0.251 eq 5353
ACL Remark: ALLOW HERE TO ANYWHERE
permit ip 192.168.50.0 0.0.0.255 any
ACL Remark: DENY INTER-VLAN TRAFFIC
deny ip 192.168.0.0 0.0.255.255 any
deny ip 10.0.0.0 0.255.255.255 any
deny ip 172.16.0.0 0.15.255.255 any

hibby50 · Mar 19, 2022

I just solved it. I realized I had a fundamental misunderstanding of inbound and outbound acl. It is inbound and outbound to the PORT, not inbound outbound to the vlan/subnet

LodeRunner · Mar 19, 2022

Chow said:
Thank´s a lot LodeRunner ;-) I will write there ;-)

Ah ok that´s very cool. I thought, that i have only 16 Ports @ 10 Gbit and 4 Ports on 40 Gbit :-D

But you think a VDX6740-24 Model comes with normaly 24 x 10 Gbit Ports in Standard, yes? And the Licenses goes up on top, right?

I'm taking what the output you provided says at face value:

Code:

40 10G port assignments are provisioned for use in this switch:
        24 10G port assignments are provisioned by the base switch license
        16 10G port assignments are provisioned by the 10G Port Upgrade license
14 10G ports are assigned to installed licenses:
        14 10G ports are assigned to the base switch license
         0 10G ports are assigned to the 10G Port Upgrade license

Chow · Mar 20, 2022

Thanks for your answer LodeRunner ;-)

Hmm ok, i thought that the "base switch license" is only 1 Gbit, not 10 Gbit?!

fohdeesha · Mar 21, 2022

Serhan said:
I have 5 runs of OM4 LC/LC fiber between these two locations. Those were for 5 servers connected to one of the 7450s which was the previous set-up. I had. I wish there could be a way to repurpose those pre-terminated lc/lc cables to stack the 7450s

the 7450's happily stack over 10gbe as well, if you have the 4x 10gbe port modules for your 7450s. if you only have 40gb qsfp modules, there's the much cheaper Kaiam LR4 lite modules that would allow you to use your existing lc fiber if it were singlemode, but you say it's multimode - there's multimode SR4 BiDi transceivers like @kpfleming linked, but I don't know if the ICX7450 QSFP+ ports can supply enough power for SR4 BiDi (haven't seen anyone try, although the module he links claims only 1.5W of draw)

RoachedCoach · Mar 21, 2022

Ruchira said:
I bought an ICX 7250-24P too I unplugged one fan out of two but It is significantly louder than my dell R630 at 20% fan load. I'm going to replace the fans and I'm torn between these 2 options

109P0412B3013 Sanyo Denki America Inc. | Fans, Thermal Management | DigiKey

Order today, ships today. 109P0412B3013 – Fan Tubeaxial 12VDC Square - 40mm L x 40mm H Ball 13.4 CFM (0.375m³/min) 3 Wire Leads from Sanyo Denki America Inc.. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com.au

MR4020X12B1-RSR Mechatronics Fan Group | Fans, Thermal Management | DigiKey

Order today, ships today. MR4020X12B1-RSR – Fan Tubeaxial 12VDC Square - 40mm L x 40mm H Ball 17.3 CFM (0.484m³/min) 3 Wire Leads from Mechatronics Fan Group. Pricing and Availability on millions of electronic components from Digi-Key Electronics.

www.digikey.com.au

Can anyone help to select the best fit between these?

Thanks

I replaced mine with Mechatronix, they've been perfect for me.

Best bet is sticking a fan on the ASIC as well, makes all the difference.

kache · Mar 21, 2022

Hi guys,

Finally installed the icx7250 for my secondary network, replacing the messy server-to-server connection routed by an pfsense VM, but I'm wondering concerning the power consumption.
This is the icx7250-24, so only 24 ports and no POE, yet the power consumption of the whole rack went up from ~480w up to ~540w, an increase of about 60w, and this is with 4 ports used (3x 10G, 1x 1G)

Does that seem correct? The ICX6450 48P with 2 POE devices connected takes around ~100W, so 60w for the icx7250-24 without POE seems a bit excessive.

fohdeesha · Mar 21, 2022

kache said:
Hi guys,

Finally installed the icx7250 for my secondary network, replacing the messy server-to-server connection routed by an pfsense VM, but I'm wondering concerning the power consumption.
This is the icx7250-24, so only 24 ports and no POE, yet the power consumption of the whole rack went up from ~480w up to ~540w, an increase of about 60w, and this is with 4 ports used (3x 10G, 1x 1G)

Does that seem correct? The ICX6450 48P with 2 POE devices connected takes around ~100W, so 60w for the icx7250-24 without POE seems a bit excessive.

what power monitoring device are you using? it sounds grossly inaccurate if it measured any 6450 variant at a hundred watts

kache · Mar 22, 2022

fohdeesha said:
what power monitoring device are you using? it sounds grossly inaccurate if it measured any 6450 variant at a hundred watts

Hi,

Not a bad point, currently I'm taking the total power consumption at the UPS, minus the two servers (data from IDRAC), minus an estimated 100w for the rest of the infra (ISP modem, 2x lenovo tiny, 2x laptops, 1x shuttle small PC, multiple rack fans).

I might need to get my hands on some IOT devices to analyze power consumption for each device individually to have a clearer idea of the power consumption.

zunder1990 · Mar 22, 2022

I am having problems getting the license to apply, I got a stack of two 6610. I use the switches for the Linux fest that I run.

Code:

  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
    UNIT 2: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: BASE_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
UNIT 2: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: BXK3845L00A
         License: BASE_SOFT_PACKAGE   (LID: dzmINJKnFFc)
         P-ENGINE  0: type E023, rev 01
         P-ENGINE  1: type E023, rev 01
==========================================================================
UNIT 2: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 2: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 30 minute(s) 54 second(s)
STACKID 2  system uptime is 31 minute(s) 3 second(s)
The system started at 06:00:23 GMT+00 Thu Feb 07 2036

 The system : started=warm start         reloaded=by "reload"
My stack unit ID = 1, bootup role = active

Code:

nocsw.self.lan#show stack
T=33m23.4: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-24F   active  cc4e.24c2.3614   0 local   Ready
2  S ICX6610-48P   standby cc4e.24f8.83b4   0 remote  Ready

    active       standby
     +---+        +---+
 =2/6| 1 |2/1==2/6| 2 |2/1=
 |   +---+        +---+   |
 |                        |
 |------------------------|
Standby u2 - protocols ready, can failover or manually switch over
Current stack management MAC is cc4e.24c2.3614

Code:

nocsw.self.lan#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
4        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Active     Unlimited         8
5        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Active     Unlimited         1
6        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Active     Unlimited         1
Stack unit 2:
1        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         8
2        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Invalid    Unlimited         1
3        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Invalid    Unlimited         1
nocsw.self.lan#

Code:

nocsw.self.lan#copy tftp license 10.1.0.1 1-6610-ports.xml unit 2
nocsw.self.lan#Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.

Copy software license to stack unit 2 success
Download request from active unit 1 mac = cc4e.24c2.3614
Downloading - $$$license
Done.
Can't add the license string - 93 (DUPLICATE_LICENSE)

fohdeesha · Mar 23, 2022

zunder1990 said:

I am having problems getting the license to apply, I got a stack of two 6610. I use the switches for the Linux fest that I run.

Code:

  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
    UNIT 2: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: BASE_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
UNIT 2: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: BXK3845L00A
         License: BASE_SOFT_PACKAGE   (LID: dzmINJKnFFc)
         P-ENGINE  0: type E023, rev 01
         P-ENGINE  1: type E023, rev 01
==========================================================================
UNIT 2: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 2: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 30 minute(s) 54 second(s)
STACKID 2  system uptime is 31 minute(s) 3 second(s)
The system started at 06:00:23 GMT+00 Thu Feb 07 2036

The system : started=warm start         reloaded=by "reload"
My stack unit ID = 1, bootup role = active

Code:

nocsw.self.lan#show stack
T=33m23.4: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-24F   active  cc4e.24c2.3614   0 local   Ready
2  S ICX6610-48P   standby cc4e.24f8.83b4   0 remote  Ready

    active       standby
     +---+        +---+
=2/6| 1 |2/1==2/6| 2 |2/1=
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover or manually switch over
Current stack management MAC is cc4e.24c2.3614

Code:

nocsw.self.lan#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
4        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Active     Unlimited         8
5        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Active     Unlimited         1
6        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Active     Unlimited         1
Stack unit 2:
1        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         8
2        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Invalid    Unlimited         1
3        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Invalid    Unlimited         1
nocsw.self.lan#

Code:

nocsw.self.lan#copy tftp license 10.1.0.1 1-6610-ports.xml unit 2
nocsw.self.lan#Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.

Copy software license to stack unit 2 success
Download request from active unit 1 mac = cc4e.24c2.3614
Downloading - $$$license
Done.
Can't add the license string - 93 (DUPLICATE_LICENSE)

you have already succesfully copied the correct licenses to the second switch, which is why it's getting a duplicate license error when you try to copy them again. they're showing as invalid on the second switch because you didn't change the serial/LID to match the licenses like the guide says on the second switch, only the first. You need to remove the second switch from the stack so it has a regular console again, then run the hw pid-prom commands on it to change its serial like you did on the first switch

fohdeesha · Mar 23, 2022

moving a hypervisor to a new /24 which means re-addressing everything, tldr this includes the VM the brocade site/resources are on so there will be about an hour or so of downtime tonight around 4 or 5am EST (or however long your DNS takes to update)

edit: done

Serhan · Mar 25, 2022

fohdeesha said:
the 7450's happily stack over 10gbe as well, if you have the 4x 10gbe port modules for your 7450s. if you only have 40gb qsfp modules, there's the much cheaper Kaiam LR4 lite modules that would allow you to use your existing lc fiber if it were singlemode, but you say it's multimode - there's multimode SR4 BiDi transceivers like @kpfleming linked, but I don't know if the ICX7450 QSFP+ ports can supply enough power for SR4 BiDi (haven't seen anyone try, although the module he links claims only 1.5W of draw)

Thank you @fohdeesha. When I use a 4x10 gb module as stacking port at the back, can I just have one sfp port for stacking, or would it need to utilise all 4 sfp ports?

kpfleming · Mar 25, 2022

It would not be required to use all four ports. In fact I don't think it's even possible to use more than two ports per switch for stacking.

Serhan · Mar 25, 2022

kpfleming said:
It would not be required to use all four ports. In fact I don't think it's even possible to use more than two ports per switch for stacking.

Sorry for my incorrect use of the terminology which is causing the confusion. I was wondering if I can use just 2 Kaiam Lite transceivers, or would I need to use 8 to stack two 7450s. These transceivers are cheaper than Bidi modules, and if I can do this with just 2, this will be the ultimate solution.

kpfleming · Mar 25, 2022

We're saying the same thing

A 4x10Gb module makes four 10GbE ports available to the switch, and they are independent of each other (other than the physical dependence of being in a single module which could fail). If you configure one port on each switch to be a stacking port, with a pair of transceivers installed linking them, that should work just fine, but of course your stack bandwidth will be limited to 10Gb.

As far as I know your only options are to use one port on each switch, or two ports on each switch (a ring configuration). It likely isn't possible to use three, four, or any more ports to stack just two switches together. It is not necessary to provide a 40Gb link to stack the switches, you can have a 10Gb link or 2x10Gb links.

Serhan · Mar 25, 2022

kpfleming said:
We're saying the same thing A 4x10Gb module makes four 10GbE ports available to the switch, and they are independent of each other (other than the physical dependence of being in a single module which could fail). If you configure one port on each switch to be a stacking port, with a pair of transceivers installed linking them, that should work just fine, but of course your stack bandwidth will be limited to 10Gb.

As far as I know your only options are to use one port on each switch, or two ports on each switch (a ring configuration). It likely isn't possible to use three, four, or any more ports to stack just two switches together. It is not necessary to provide a 40Gb link to stack the switches, you can have a 10Gb link or 2x10Gb links.

Thank you for the detailed explanation, all clear now.

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Active Member

Active Member

New Member

New Member

New Member

Active Member

New Member

Kaini Industries

Member

New Member

Kaini Industries

New Member

Active Member

Kaini Industries

Kaini Industries

Active Member

Active Member

Active Member

Active Member

Active Member