Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

I am having problems getting the license to apply, I got a stack of two 6610. I use the switches for the Linux fest that I run.

  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
    UNIT 2: compiled on Apr 23 2020 at 12:11:06 labeled as FCXS08030u
                (7723621 bytes) from Primary FCXS08030u.bin
        SW: Version 08.0.30uT7f1
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module
         Serial  #: 2ax5o2jk68e
         License: BASE_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
UNIT 2: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: BXK3845L00A
         License: BASE_SOFT_PACKAGE   (LID: dzmINJKnFFc)
         P-ENGINE  0: type E023, rev 01
         P-ENGINE  1: type E023, rev 01
==========================================================================
UNIT 2: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 2: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 30 minute(s) 54 second(s)
STACKID 2  system uptime is 31 minute(s) 3 second(s)
The system started at 06:00:23 GMT+00 Thu Feb 07 2036

The system : started=warm start         reloaded=by "reload"
My stack unit ID = 1, bootup role = active

nocsw.self.lan#show stack
T=33m23.4: alone: standalone, D: dynamic cfg, S: static
ID   Type          Role    Mac Address    Pri State   Comment
1  S ICX6610-24F   active  cc4e.24c2.3614   0 local   Ready
2  S ICX6610-48P   standby cc4e.24f8.83b4   0 remote  Ready

    active       standby
     +---+        +---+
=2/6| 1 |2/1==2/6| 2 |2/1=
|   +---+        +---+   |
|                        |
|------------------------|
Standby u2 - protocols ready, can failover or manually switch over
Current stack management MAC is cc4e.24c2.3614

nocsw.self.lan#show license
Index    Lic Mode        Lic Name               Lid/Serial No  Lic Type    Status     Lic Period    Lic Capacity
Stack unit 1:
4        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Active     Unlimited         8
5        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Active     Unlimited         1
6        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Active     Unlimited         1
Stack unit 2:
1        Node Lock       ICX6610-10G-LIC-POD    H4CKTH3PLN8    Normal      Invalid    Unlimited         8
2        Node Lock       ICX6610-ADV-LIC-SW     H4CKTH3PLN8    Normal      Invalid    Unlimited         1
3        Node Lock       ICX-MACSEC-LIC         H4CKTH3PLN8    Normal      Invalid    Unlimited         1
nocsw.self.lan#

nocsw.self.lan#copy tftp license 10.1.0.1 1-6610-ports.xml unit 2
nocsw.self.lan#Flash Memory Write (8192 bytes per dot) .
Copy Software License from TFTP to Flash Done.

Copy software license to stack unit 2 success
Download request from active unit 1 mac = cc4e.24c2.3614
Downloading - $$$license
Done.
Can't add the license string - 93 (DUPLICATE_LICENSE)

you have already succesfully copied the correct licenses to the second switch, which is why it's getting a duplicate license error when you try to copy them again. they're showing as invalid on the second switch because you didn't change the serial/LID to match the licenses like the guide says on the second switch, only the first. You need to remove the second switch from the stack so it has a regular console again, then run the hw pid-prom commands on it to change its serial like you did on the first switch

 

[fohdeesha]

moving a hypervisor to a new /24 which means re-addressing everything, tldr this includes the VM the brocade site/resources are on so there will be about an hour or so of downtime tonight around 4 or 5am EST (or however long your DNS takes to update)

edit: done

 

[Serhan]

the 7450's happily stack over 10gbe as well, if you have the 4x 10gbe port modules for your 7450s. if you only have 40gb qsfp modules, there's the much cheaper Kaiam LR4 lite modules that would allow you to use your existing lc fiber if it were singlemode, but you say it's multimode - there's multimode SR4 BiDi transceivers like @kpfleming linked, but I don't know if the ICX7450 QSFP+ ports can supply enough power for SR4 BiDi (haven't seen anyone try, although the module he links claims only 1.5W of draw)

Thank you @fohdeesha. When I use a 4x10 gb module as stacking port at the back, can I just have one sfp port for stacking, or would it need to utilise all 4 sfp ports?

 

[kpfleming]

It would not be required to use all four ports. In fact I don't think it's even possible to use more than two ports per switch for stacking.

 

[Serhan]

It would not be required to use all four ports. In fact I don't think it's even possible to use more than two ports per switch for stacking.

Sorry for my incorrect use of the terminology which is causing the confusion. I was wondering if I can use just 2 Kaiam Lite transceivers, or would I need to use 8 to stack two 7450s. These transceivers are cheaper than Bidi modules, and if I can do this with just 2, this will be the ultimate solution.

 

[kpfleming]

We're saying the same thing A 4x10Gb module makes four 10GbE ports available to the switch, and they are independent of each other (other than the physical dependence of being in a single module which could fail). If you configure one port on each switch to be a stacking port, with a pair of transceivers installed linking them, that should work just fine, but of course your stack bandwidth will be limited to 10Gb.

As far as I know your only options are to use one port on each switch, or two ports on each switch (a ring configuration). It likely isn't possible to use three, four, or any more ports to stack just two switches together. It is not necessary to provide a 40Gb link to stack the switches, you can have a 10Gb link or 2x10Gb links.

 

[Serhan]

We're saying the same thing A 4x10Gb module makes four 10GbE ports available to the switch, and they are independent of each other (other than the physical dependence of being in a single module which could fail). If you configure one port on each switch to be a stacking port, with a pair of transceivers installed linking them, that should work just fine, but of course your stack bandwidth will be limited to 10Gb.

As far as I know your only options are to use one port on each switch, or two ports on each switch (a ring configuration). It likely isn't possible to use three, four, or any more ports to stack just two switches together. It is not necessary to provide a 40Gb link to stack the switches, you can have a 10Gb link or 2x10Gb links.

Thank you for the detailed explanation, all clear now.

 

[kpfleming]

It would be worth your time to read the 'Switch Stacking Guide' for the version of the firmware you are using; it has very detailed descriptions of the possible configurations, including diagrams showing how the ports can be connected.

One thing to note: according to the guide for 09.x firmware, the 4x10GF module can only be used for stacking if it is in the module 2 (front) slot. If it is installed in either of the rear slots it can only be used for data ports. The reverse is true for the 1x40GF modules; they can only be used for stacking in the rear slots, not the front slot.

It is also possible to use all four 10Gb ports to connect two switches, with 20Gb bandwidth in each direction in a ring topology. This would require four fiber pairs though (unless you use BiDi transceivers), so that's probably more than you want to do.

 

[bpye]

Is there any way to change the fan speed threshold on the ICX6450? I swapped the fans with some Sunon MagLevs and they are quiet to the point that I would happily have them at 100% all the time to keep the temp a bit lower, if possible. If not it's hardly an issue, just wondering if it's something that's configurable.

 

[dswartz]

It would be worth your time to read the 'Switch Stacking Guide' for the version of the firmware you are using; it has very detailed descriptions of the possible configurations, including diagrams showing how the ports can be connected.

Amen! I'm not that savvy of L2, so this was tres helpful.

 

[deff_janiels]

I'm biased since I wrote a lot of the posts, but for the 7250, I'm all in on the Mechatronics for the rear fans, and the single Sunon for the ASIC fan. Pay close attention to the model numbers in the posts.

And if it's cranking up every so often, probably a room temp/airflow issue, so do whatever you can to get that thing better ventilated to avoid the spinups in the first place. The fan replacements will NOT improve airflow, they'll just run slower/somewhat quieter (though the ASCI fan does help offset a bit). Things to consider.

I want to add a recommendation for this configuration. I just replaced my twin 6450-24Ps with 7250-24Ps as I needed more 10G ports. In the 6450s I had been running Delta EFB0412VHD-F00s which worked quite well. I swapped them into the 7250s and found they would cycle up to high speed every 20 minutes or so, run for about 3 minutes, then go back to low speed.

I just replaced the Deltas with Mechatronics MR4020X12B1-RSRs and added a Mechatronics MB6010M12B-RSR (60x10) to the top of each ASIC heatsink. (All of these are currently available at Digikey). The ASIC fan was secured into the heatsink with coarse thread screws that engage the heatsink fins at two corners. I crimped the two wires of the ASIC fans into the header of one of the rear fans. Here's what it ended up looking like:



The setup is now stable, no throttling up to high fan speeds with these kind of temperatures in a closet that's around 30 C:

SSH@coresw#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 64.5 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 41.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 65.0 deg-C (Sensor 1), 41.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA                                   
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fa9.f328
Management MAC: 609c.9fa9.f328

The stack unit 2 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 41.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C                 
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:Front to Back
Fan 2 Air Flow Direction:Front to Back
Slot 1 Current Temperature: 62.9 deg-C (Sensor 1), 41.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fa9.9da8

 

[koifish59]

Has anyone gotten fiber WAN to work with their switches? I must be doing something wrong.

Fiber internet from ISP going into an untagged port vLAN 60 with interface v60, going out to an untagged port for pfsense VM doesn’t work for me. I’ve been through 3 different transceiver modules, including an official Brocade E1MG-LX-OM already (SFP 1310nm SMF). Tested both with ICX 6650 and 6610. Other notes about my setup:

- changed the port from 10g to 1g
- flow control disabled
- interface 60, vLAN 60 untagged

My current bandaid solution currently is plugging the fiber line into THIS media converter, then the ethernet out goes back into THIS (E1MG-TX 57-1000042-01) RJ-45 SFP module and into the switch, which works. It just seems silly to me to have to convert fiber to ethernet back to fiber connection for the switch.

 

[fohdeesha]

Has anyone gotten fiber WAN to work with their switches? I must be doing something wrong.

Fiber internet from ISP going into an untagged port vLAN 60 with interface v60, going out to an untagged port for pfsense VM doesn’t work for me. I’ve been through 3 different transceiver modules, including an official Brocade E1MG-LX-OM already (SFP 1310nm SMF). Tested both with ICX 6650 and 6610. Other notes about my setup:

- changed the port from 10g to 1g
- flow control disabled
- interface 60, vLAN 60 untagged

My current bandaid solution currently is plugging the fiber line into THIS media converter, then the ethernet out goes back into THIS (E1MG-TX 57-1000042-01) RJ-45 SFP module and into the switch, which works. It just seems silly to me to have to convert fiber to ethernet back to fiber connection for the switch.

I'm assuming the optic you've confirmed works in the media converter, you've tried in the switch? also remove the ve interface of the vlan, if you're passing the wan connection to your router over a vlan you don't want the switch having an l3 interface in the network

 

[tubs-ffm]

Hello,

I would like to isolate ports to avoid clients to talk to each other on my ICX.

VLAN 20 is my guest network. On WLAN access point I can use the function "client isolation" for the guest network. But this does not prevent the communication from WLAN client to a client on LAN in same guest network (VLAN 20).

My configuration on ICX (simplified):

1/1/1 trunk to firewall: VLAN 1, 10, 20, 30
1/1/2 access point: VLAN 1, 20, 30
1/1/3 guest LAN: VLAN 20

Is there a simple way to block communication on ICX between port 1/1/2 and 1/1/3?
Or is PVLAN the only option to achieve this?

 

[Serhan]

Would a ICX7650-4X10GF Expansion Module work on a ICX7450?

 

[koifish59]

I'm assuming the optic you've confirmed works in the media converter, you've tried in the switch? also remove the ve interface of the vlan, if you're passing the wan connection to your router over a vlan you don't want the switch having an l3 interface in the network

Yep, confirmed all optics works in the media converter but doesn’t for both switches.

Maybe it is the interfaces. I’ll remove them for the WAN connections. I’ve had a different interface for each vLAN because I didn’t have a full understanding on using them.
So interfaces are only useful for L3 functions? In what situations would a specific interface have two vLANs as opposed to separate interface for each vLAN?

 

[kpfleming]

Is there a simple way to block communication on ICX between port 1/1/2 and 1/1/3?
Or is PVLAN the only option to achieve this?

Access lists can be used to do this; if the access-list doesn't permit traffic to other addresses on the same LAN (VLAN), the traffic will be blocked. If your guest network is 192.168.10.0/24, for example, you can put an explicit 'drop' rule in the access-list for that destination address, and IP traffic between the ports will be blocked. Non-IP traffic won't be blocked, but that's very rare.

You'll need a rule *ahead* of the drop rule which permits traffic to the gateway (your firewall) for that LAN, as well as a catch-all rule which allows traffic to all other addresses.

 

[tubs-ffm]

Access lists can be used to do this; if the access-list doesn't permit traffic to other addresses on the same LAN (VLAN), the traffic will be blocked.

Thank you.
I will have a look in it.

 

[Balteck]

Hello everyone, I'm building a new house and I try to decide for ICX6610-48P or ICX7450-48P or two 24 port switches in stack (PoE and not-PoE)

I know that my question is already present al lot of time, but I didn't find the right answer.

My needs are:

- POE+ ports for APs, VOIP Adapters, IP Cameras, maybe POE Lighting system with sensors, Video Entryphone and any other device that I can connect (also with PoE splitter) by wire instead of WIFI
- Three 10GB and 12 1Gb connections to my little cluster (2 esxi box, 1 nas) and my workstation (with 10base-t transceiver)
- 16 1GB devices (PCs, RPIs, AV Receiver, Video Projector, TV, SAT->IP...)
- Dual PSU protected by UPS to prevent any possible downtime for critical devices (Alarm, Camera IP, Door Bell, Video Entryphone and so on)
- when I will go to live in, a big media center (now I'm using Emby on my Workstation, but it is full of HDDs) like this one: 4U 24 Bay SAS3 Vmware Storage Server X10QBi Includes CPU/Memory
- power drain is not a a problem
- noise is not a problem (I will dedicate a room in basement)


My concern is about the pro and cons between the two units:
ICX 6xxx is EOL and ICX7xxx is supported, so it means that the first one had the last firmware, while the latter will have new firmware (now 9.0).
But is the new firmware a worth update? Which new features does it have?
ICX7xxx seems to have less port 10GB or 40 GB ports
ICX7xxx have also PoH support (90W), but I didn't find a device that use it. Also the last Wifi 6 Ap uses the PoE+ standard. Anyone could give me a list of these devices?

After all, I ask everyone to convince me which one is the best or alternative solutions.

I'm not in hurry, because the house will be ready and the end on the year

Thank you very much

 

[adman_c]

So I'm in the process of rethinking/redoing my network so that inter-vlan routing can be done on my switch rather than my firewall (pfsense). I'm curious if it's possible for some but not all inter-vlan routing to be handled on the switch? For example, I have 2 VLANs (MGMT/HOME) that can access everything and can each access each other. But I have other VLANs that I want to remain segregated and have access only to the internet and nothing local (IOT/GUEST). Is it possible to just have VEs for MGMT and HOME, and have those routed on the switch with no ACLs, but keep forcing my other VLANs to go out to the firewall?

Thanks!