Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

kache

New Member
Jun 27, 2020
15
2
3
Hi guys,

Finally installed the icx7250 for my secondary network, replacing the messy server-to-server connection routed by an pfsense VM, but I'm wondering concerning the power consumption.
This is the icx7250-24, so only 24 ports and no POE, yet the power consumption of the whole rack went up from ~480w up to ~540w, an increase of about 60w, and this is with 4 ports used (3x 10G, 1x 1G)

Does that seem correct? The ICX6450 48P with 2 POE devices connected takes around ~100W, so 60w for the icx7250-24 without POE seems a bit excessive.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,562
2,753
113
31
fohdeesha.com
Hi guys,

Finally installed the icx7250 for my secondary network, replacing the messy server-to-server connection routed by an pfsense VM, but I'm wondering concerning the power consumption.
This is the icx7250-24, so only 24 ports and no POE, yet the power consumption of the whole rack went up from ~480w up to ~540w, an increase of about 60w, and this is with 4 ports used (3x 10G, 1x 1G)

Does that seem correct? The ICX6450 48P with 2 POE devices connected takes around ~100W, so 60w for the icx7250-24 without POE seems a bit excessive.
what power monitoring device are you using? it sounds grossly inaccurate if it measured any 6450 variant at a hundred watts
 

kache

New Member
Jun 27, 2020
15
2
3
what power monitoring device are you using? it sounds grossly inaccurate if it measured any 6450 variant at a hundred watts
Hi,

Not a bad point, currently I'm taking the total power consumption at the UPS, minus the two servers (data from IDRAC), minus an estimated 100w for the rest of the infra (ISP modem, 2x lenovo tiny, 2x laptops, 1x shuttle small PC, multiple rack fans).

I might need to get my hands on some IOT devices to analyze power consumption for each device individually to have a clearer idea of the power consumption.
 

zunder1990

Active Member
Nov 15, 2012
130
30
28
I am having problems getting the license to apply, I got a stack of two 6610. I use the switches for the Linux fest that I run.

Code:
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
    UNIT 2: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: BASE_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
UNIT 2: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: BXK3845L00A
         License: BASE_SOFT_PACKAGE   (LID: dzmINJKnFFc)
         P-ENGINE  0: type E023, rev 01
         P-ENGINE  1: type E023, rev 01
==========================================================================
UNIT 2: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 2: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 30 minute(s) 54 second(s)
STACKID 2  system uptime is 31 minute(s) 3 second(s)
The system started at 06:00:23 GMT+00 Thu Feb 07 2036

 The system : started=warm start         reloaded=by "reload"
My stack unit ID = 1, bootup role = active
Code:
nocsw.self.lan#show stack
T=33m23.4: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-24F   active  cc4e.24c2.3614   0 local   Ready
2  S ICX6610-48P   standby cc4e.24f8.83b4   0 remote  Ready

    active       standby
     +---+        +---+
 =2/6| 1 |2/1==2/6| 2 |2/1=
 |   +---+        +---+   |
 |                        |
 |------------------------|
Standby u2 - protocols ready, can failover or manually switch over
Current stack management MAC is cc4e.24c2.3614
Code:
nocsw.self.lan#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
4        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Active     Unlimited         8
5        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Active     Unlimited         1
6        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Active     Unlimited         1
Stack unit 2:
1        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         8
2        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Invalid    Unlimited         1
3        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Invalid    Unlimited         1
nocsw.self.lan#
Code:
nocsw.self.lan#copy tftp license 10.1.0.1 1-6610-ports.xml unit 2
nocsw.self.lan#Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.

Copy software license to stack unit 2 success
Download request from active unit 1 mac = cc4e.24c2.3614
Downloading - $$$license
Done.
Can't add the license string - 93 (DUPLICATE_LICENSE)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,562
2,753
113
31
fohdeesha.com
I am having problems getting the license to apply, I got a stack of two 6610. I use the switches for the Linux fest that I run.

Code:
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
    UNIT 2: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: BASE_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
UNIT 2: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: BXK3845L00A
         License: BASE_SOFT_PACKAGE   (LID: dzmINJKnFFc)
         P-ENGINE  0: type E023, rev 01
         P-ENGINE  1: type E023, rev 01
==========================================================================
UNIT 2: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 2: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 30 minute(s) 54 second(s)
STACKID 2  system uptime is 31 minute(s) 3 second(s)
The system started at 06:00:23 GMT+00 Thu Feb 07 2036

The system : started=warm start         reloaded=by "reload"
My stack unit ID = 1, bootup role = active
Code:
nocsw.self.lan#show stack
T=33m23.4: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-24F   active  cc4e.24c2.3614   0 local   Ready
2  S ICX6610-48P   standby cc4e.24f8.83b4   0 remote  Ready

    active       standby
     +---+        +---+
=2/6| 1 |2/1==2/6| 2 |2/1=
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover or manually switch over
Current stack management MAC is cc4e.24c2.3614
Code:
nocsw.self.lan#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
4        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Active     Unlimited         8
5        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Active     Unlimited         1
6        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Active     Unlimited         1
Stack unit 2:
1        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         8
2        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Invalid    Unlimited         1
3        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Invalid    Unlimited         1
nocsw.self.lan#
Code:
nocsw.self.lan#copy tftp license 10.1.0.1 1-6610-ports.xml unit 2
nocsw.self.lan#Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.

Copy software license to stack unit 2 success
Download request from active unit 1 mac = cc4e.24c2.3614
Downloading - $$$license
Done.
Can't add the license string - 93 (DUPLICATE_LICENSE)
you have already succesfully copied the correct licenses to the second switch, which is why it's getting a duplicate license error when you try to copy them again. they're showing as invalid on the second switch because you didn't change the serial/LID to match the licenses like the guide says on the second switch, only the first. You need to remove the second switch from the stack so it has a regular console again, then run the hw pid-prom commands on it to change its serial like you did on the first switch
 
  • Like
Reactions: zunder1990

fohdeesha

Kaini Industries
Nov 20, 2016
2,562
2,753
113
31
fohdeesha.com
moving a hypervisor to a new /24 which means re-addressing everything, tldr this includes the VM the brocade site/resources are on so there will be about an hour or so of downtime tonight around 4 or 5am EST (or however long your DNS takes to update)

edit: done
 
Last edited:

Serhan

Member
Sep 22, 2017
71
15
8
the 7450's happily stack over 10gbe as well, if you have the 4x 10gbe port modules for your 7450s. if you only have 40gb qsfp modules, there's the much cheaper Kaiam LR4 lite modules that would allow you to use your existing lc fiber if it were singlemode, but you say it's multimode - there's multimode SR4 BiDi transceivers like @kpfleming linked, but I don't know if the ICX7450 QSFP+ ports can supply enough power for SR4 BiDi (haven't seen anyone try, although the module he links claims only 1.5W of draw)
Thank you @fohdeesha. When I use a 4x10 gb module as stacking port at the back, can I just have one sfp port for stacking, or would it need to utilise all 4 sfp ports?
 

Serhan

Member
Sep 22, 2017
71
15
8
It would not be required to use all four ports. In fact I don't think it's even possible to use more than two ports per switch for stacking.
Sorry for my incorrect use of the terminology which is causing the confusion. I was wondering if I can use just 2 Kaiam Lite transceivers, or would I need to use 8 to stack two 7450s. These transceivers are cheaper than Bidi modules, and if I can do this with just 2, this will be the ultimate solution.
 
Last edited:

kpfleming

Active Member
Dec 28, 2021
199
78
28
Pelham NY USA
We're saying the same thing :) A 4x10Gb module makes four 10GbE ports available to the switch, and they are independent of each other (other than the physical dependence of being in a single module which could fail). If you configure one port on each switch to be a stacking port, with a pair of transceivers installed linking them, that should work just fine, but of course your stack bandwidth will be limited to 10Gb.

As far as I know your only options are to use one port on each switch, or two ports on each switch (a ring configuration). It likely isn't possible to use three, four, or any more ports to stack just two switches together. It is not necessary to provide a 40Gb link to stack the switches, you can have a 10Gb link or 2x10Gb links.
 
  • Like
Reactions: Serhan

Serhan

Member
Sep 22, 2017
71
15
8
We're saying the same thing :) A 4x10Gb module makes four 10GbE ports available to the switch, and they are independent of each other (other than the physical dependence of being in a single module which could fail). If you configure one port on each switch to be a stacking port, with a pair of transceivers installed linking them, that should work just fine, but of course your stack bandwidth will be limited to 10Gb.

As far as I know your only options are to use one port on each switch, or two ports on each switch (a ring configuration). It likely isn't possible to use three, four, or any more ports to stack just two switches together. It is not necessary to provide a 40Gb link to stack the switches, you can have a 10Gb link or 2x10Gb links.
Thank you for the detailed explanation, all clear now.
 

kpfleming

Active Member
Dec 28, 2021
199
78
28
Pelham NY USA
It would be worth your time to read the 'Switch Stacking Guide' for the version of the firmware you are using; it has very detailed descriptions of the possible configurations, including diagrams showing how the ports can be connected.

One thing to note: according to the guide for 09.x firmware, the 4x10GF module can only be used for stacking if it is in the module 2 (front) slot. If it is installed in either of the rear slots it can only be used for data ports. The reverse is true for the 1x40GF modules; they can only be used for stacking in the rear slots, not the front slot.

It is also possible to use all four 10Gb ports to connect two switches, with 20Gb bandwidth in each direction in a ring topology. This would require four fiber pairs though (unless you use BiDi transceivers), so that's probably more than you want to do.
 

bpye

New Member
Apr 13, 2021
10
0
1
Is there any way to change the fan speed threshold on the ICX6450? I swapped the fans with some Sunon MagLevs and they are quiet to the point that I would happily have them at 100% all the time to keep the temp a bit lower, if possible. If not it's hardly an issue, just wondering if it's something that's configurable.
 

dswartz

Active Member
Jul 14, 2011
589
73
28
It would be worth your time to read the 'Switch Stacking Guide' for the version of the firmware you are using; it has very detailed descriptions of the possible configurations, including diagrams showing how the ports can be connected.
Amen! I'm not that savvy of L2, so this was tres helpful.
 

deff_janiels

New Member
Mar 28, 2021
2
4
3
I'm biased since I wrote a lot of the posts, but for the 7250, I'm all in on the Mechatronics for the rear fans, and the single Sunon for the ASIC fan. Pay close attention to the model numbers in the posts.

And if it's cranking up every so often, probably a room temp/airflow issue, so do whatever you can to get that thing better ventilated to avoid the spinups in the first place. The fan replacements will NOT improve airflow, they'll just run slower/somewhat quieter (though the ASCI fan does help offset a bit). Things to consider.
I want to add a recommendation for this configuration. I just replaced my twin 6450-24Ps with 7250-24Ps as I needed more 10G ports. In the 6450s I had been running Delta EFB0412VHD-F00s which worked quite well. I swapped them into the 7250s and found they would cycle up to high speed every 20 minutes or so, run for about 3 minutes, then go back to low speed.

I just replaced the Deltas with Mechatronics MR4020X12B1-RSRs and added a Mechatronics MB6010M12B-RSR (60x10) to the top of each ASIC heatsink. (All of these are currently available at Digikey). The ASIC fan was secured into the heatsink with coarse thread screws that engage the heatsink fins at two corners. I crimped the two wires of the ASIC fans into the header of one of the rear fans. Here's what it ended up looking like:

1648328339942.jpeg

The setup is now stable, no throttling up to high fan speeds with these kind of temperatures in a closet that's around 30 C:

Code:
SSH@coresw#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 64.5 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 41.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 65.0 deg-C (Sensor 1), 41.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA                                   
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fa9.f328
Management MAC: 609c.9fa9.f328

The stack unit 2 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 41.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C                 
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:Front to Back
Fan 2 Air Flow Direction:Front to Back
Slot 1 Current Temperature: 62.9 deg-C (Sensor 1), 41.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fa9.9da8
 

koifish59

Member
Sep 30, 2020
66
19
8
Has anyone gotten fiber WAN to work with their switches? I must be doing something wrong.

Fiber internet from ISP going into an untagged port vLAN 60 with interface v60, going out to an untagged port for pfsense VM doesn’t work for me. I’ve been through 3 different transceiver modules, including an official Brocade E1MG-LX-OM already (SFP 1310nm SMF). Tested both with ICX 6650 and 6610. Other notes about my setup:

- changed the port from 10g to 1g
- flow control disabled
- interface 60, vLAN 60 untagged

My current bandaid solution currently is plugging the fiber line into THIS media converter, then the ethernet out goes back into THIS (E1MG-TX 57-1000042-01) RJ-45 SFP module and into the switch, which works. It just seems silly to me to have to convert fiber to ethernet back to fiber connection for the switch.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,562
2,753
113
31
fohdeesha.com
Has anyone gotten fiber WAN to work with their switches? I must be doing something wrong.

Fiber internet from ISP going into an untagged port vLAN 60 with interface v60, going out to an untagged port for pfsense VM doesn’t work for me. I’ve been through 3 different transceiver modules, including an official Brocade E1MG-LX-OM already (SFP 1310nm SMF). Tested both with ICX 6650 and 6610. Other notes about my setup:

- changed the port from 10g to 1g
- flow control disabled
- interface 60, vLAN 60 untagged

My current bandaid solution currently is plugging the fiber line into THIS media converter, then the ethernet out goes back into THIS (E1MG-TX 57-1000042-01) RJ-45 SFP module and into the switch, which works. It just seems silly to me to have to convert fiber to ethernet back to fiber connection for the switch.
I'm assuming the optic you've confirmed works in the media converter, you've tried in the switch? also remove the ve interface of the vlan, if you're passing the wan connection to your router over a vlan you don't want the switch having an l3 interface in the network
 

tubs-ffm

Active Member
Sep 1, 2013
122
39
28
Hello,

I would like to isolate ports to avoid clients to talk to each other on my ICX.

VLAN 20 is my guest network. On WLAN access point I can use the function "client isolation" for the guest network. But this does not prevent the communication from WLAN client to a client on LAN in same guest network (VLAN 20).

My configuration on ICX (simplified):

1/1/1 trunk to firewall: VLAN 1, 10, 20, 30
1/1/2 access point: VLAN 1, 20, 30
1/1/3 guest LAN: VLAN 20

Is there a simple way to block communication on ICX between port 1/1/2 and 1/1/3?
Or is PVLAN the only option to achieve this?
 

koifish59

Member
Sep 30, 2020
66
19
8
I'm assuming the optic you've confirmed works in the media converter, you've tried in the switch? also remove the ve interface of the vlan, if you're passing the wan connection to your router over a vlan you don't want the switch having an l3 interface in the network
Yep, confirmed all optics works in the media converter but doesn’t for both switches.

Maybe it is the interfaces. I’ll remove them for the WAN connections. I’ve had a different interface for each vLAN because I didn’t have a full understanding on using them.
So interfaces are only useful for L3 functions? In what situations would a specific interface have two vLANs as opposed to separate interface for each vLAN?