Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[kpfleming]

It would be worth your time to read the 'Switch Stacking Guide' for the version of the firmware you are using; it has very detailed descriptions of the possible configurations, including diagrams showing how the ports can be connected.

One thing to note: according to the guide for 09.x firmware, the 4x10GF module can only be used for stacking if it is in the module 2 (front) slot. If it is installed in either of the rear slots it can only be used for data ports. The reverse is true for the 1x40GF modules; they can only be used for stacking in the rear slots, not the front slot.

It is also possible to use all four 10Gb ports to connect two switches, with 20Gb bandwidth in each direction in a ring topology. This would require four fiber pairs though (unless you use BiDi transceivers), so that's probably more than you want to do.

 

[bpye]

Is there any way to change the fan speed threshold on the ICX6450? I swapped the fans with some Sunon MagLevs and they are quiet to the point that I would happily have them at 100% all the time to keep the temp a bit lower, if possible. If not it's hardly an issue, just wondering if it's something that's configurable.

 

[dswartz]

It would be worth your time to read the 'Switch Stacking Guide' for the version of the firmware you are using; it has very detailed descriptions of the possible configurations, including diagrams showing how the ports can be connected.

Amen! I'm not that savvy of L2, so this was tres helpful.

 

[deff_janiels]

I'm biased since I wrote a lot of the posts, but for the 7250, I'm all in on the Mechatronics for the rear fans, and the single Sunon for the ASIC fan. Pay close attention to the model numbers in the posts.

And if it's cranking up every so often, probably a room temp/airflow issue, so do whatever you can to get that thing better ventilated to avoid the spinups in the first place. The fan replacements will NOT improve airflow, they'll just run slower/somewhat quieter (though the ASCI fan does help offset a bit). Things to consider.

I want to add a recommendation for this configuration. I just replaced my twin 6450-24Ps with 7250-24Ps as I needed more 10G ports. In the 6450s I had been running Delta EFB0412VHD-F00s which worked quite well. I swapped them into the 7250s and found they would cycle up to high speed every 20 minutes or so, run for about 3 minutes, then go back to low speed.

I just replaced the Deltas with Mechatronics MR4020X12B1-RSRs and added a Mechatronics MB6010M12B-RSR (60x10) to the top of each ASIC heatsink. (All of these are currently available at Digikey). The ASIC fan was secured into the heatsink with coarse thread screws that engage the heatsink fins at two corners. I crimped the two wires of the ASIC fans into the header of one of the rear fans. Here's what it ended up looking like:



The setup is now stable, no throttling up to high fan speeds with these kind of temperatures in a closet that's around 30 C:

SSH@coresw#show chassis
The stack unit 1 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 64.5 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 41.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:  Front to Back
Fan 2 Air Flow Direction:  Front to Back
Slot 1 Current Temperature: 65.0 deg-C (Sensor 1), 41.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA                                   
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fa9.f328
Management MAC: 609c.9fa9.f328

The stack unit 2 chassis info:

Power supply 1 (AC - PoE) present, status ok
Power supply 2 not present

Fan 1 ok, speed (auto): [[1]]<->2
Fan 2 ok, speed (auto): [[1]]<->2

Fan controlled temperature:
        Rule 1/2 (MGMT THERMAL PLANE): 62.4 deg-C
        Rule 2/2 (AIR OUTLET NEAR PSU): 41.0 deg-C

Fan speed switching temperature thresholds:
        Rule 1/2 (MGMT THERMAL PLANE):
                Speed 1: NM<-----> 93       deg-C
                Speed 2:        82<----->105 deg-C (shutdown)
        Rule 2/2 (AIR OUTLET NEAR PSU):
                Speed 1: NM<-----> 58       deg-C                 
                Speed 2:        49<----->105 deg-C (shutdown)

Fan 1 Air Flow Direction:Front to Back
Fan 2 Air Flow Direction:Front to Back
Slot 1 Current Temperature: 62.9 deg-C (Sensor 1), 41.0 deg-C (Sensor 2)
Slot 2 Current Temperature: NA
        Warning level.......: 100.0 deg-C
        Shutdown level......: 105.0 deg-C
Boot Prom MAC : 609c.9fa9.9da8

 

[koifish59]

Has anyone gotten fiber WAN to work with their switches? I must be doing something wrong.

Fiber internet from ISP going into an untagged port vLAN 60 with interface v60, going out to an untagged port for pfsense VM doesn’t work for me. I’ve been through 3 different transceiver modules, including an official Brocade E1MG-LX-OM already (SFP 1310nm SMF). Tested both with ICX 6650 and 6610. Other notes about my setup:

- changed the port from 10g to 1g
- flow control disabled
- interface 60, vLAN 60 untagged

My current bandaid solution currently is plugging the fiber line into THIS media converter, then the ethernet out goes back into THIS (E1MG-TX 57-1000042-01) RJ-45 SFP module and into the switch, which works. It just seems silly to me to have to convert fiber to ethernet back to fiber connection for the switch.

 

[fohdeesha]

Has anyone gotten fiber WAN to work with their switches? I must be doing something wrong.

Fiber internet from ISP going into an untagged port vLAN 60 with interface v60, going out to an untagged port for pfsense VM doesn’t work for me. I’ve been through 3 different transceiver modules, including an official Brocade E1MG-LX-OM already (SFP 1310nm SMF). Tested both with ICX 6650 and 6610. Other notes about my setup:

- changed the port from 10g to 1g
- flow control disabled
- interface 60, vLAN 60 untagged

My current bandaid solution currently is plugging the fiber line into THIS media converter, then the ethernet out goes back into THIS (E1MG-TX 57-1000042-01) RJ-45 SFP module and into the switch, which works. It just seems silly to me to have to convert fiber to ethernet back to fiber connection for the switch.

I'm assuming the optic you've confirmed works in the media converter, you've tried in the switch? also remove the ve interface of the vlan, if you're passing the wan connection to your router over a vlan you don't want the switch having an l3 interface in the network

 

[tubs-ffm]

Hello,

I would like to isolate ports to avoid clients to talk to each other on my ICX.

VLAN 20 is my guest network. On WLAN access point I can use the function "client isolation" for the guest network. But this does not prevent the communication from WLAN client to a client on LAN in same guest network (VLAN 20).

My configuration on ICX (simplified):

1/1/1 trunk to firewall: VLAN 1, 10, 20, 30
1/1/2 access point: VLAN 1, 20, 30
1/1/3 guest LAN: VLAN 20

Is there a simple way to block communication on ICX between port 1/1/2 and 1/1/3?
Or is PVLAN the only option to achieve this?

 

[Serhan]

Would a ICX7650-4X10GF Expansion Module work on a ICX7450?

 

[koifish59]

I'm assuming the optic you've confirmed works in the media converter, you've tried in the switch? also remove the ve interface of the vlan, if you're passing the wan connection to your router over a vlan you don't want the switch having an l3 interface in the network

Yep, confirmed all optics works in the media converter but doesn’t for both switches.

Maybe it is the interfaces. I’ll remove them for the WAN connections. I’ve had a different interface for each vLAN because I didn’t have a full understanding on using them.
So interfaces are only useful for L3 functions? In what situations would a specific interface have two vLANs as opposed to separate interface for each vLAN?

 

[kpfleming]

Is there a simple way to block communication on ICX between port 1/1/2 and 1/1/3?
Or is PVLAN the only option to achieve this?

Access lists can be used to do this; if the access-list doesn't permit traffic to other addresses on the same LAN (VLAN), the traffic will be blocked. If your guest network is 192.168.10.0/24, for example, you can put an explicit 'drop' rule in the access-list for that destination address, and IP traffic between the ports will be blocked. Non-IP traffic won't be blocked, but that's very rare.

You'll need a rule *ahead* of the drop rule which permits traffic to the gateway (your firewall) for that LAN, as well as a catch-all rule which allows traffic to all other addresses.

 

[tubs-ffm]

Access lists can be used to do this; if the access-list doesn't permit traffic to other addresses on the same LAN (VLAN), the traffic will be blocked.

Thank you.
I will have a look in it.

 

[Balteck]

Hello everyone, I'm building a new house and I try to decide for ICX6610-48P or ICX7450-48P or two 24 port switches in stack (PoE and not-PoE)

I know that my question is already present al lot of time, but I didn't find the right answer.

My needs are:

- POE+ ports for APs, VOIP Adapters, IP Cameras, maybe POE Lighting system with sensors, Video Entryphone and any other device that I can connect (also with PoE splitter) by wire instead of WIFI
- Three 10GB and 12 1Gb connections to my little cluster (2 esxi box, 1 nas) and my workstation (with 10base-t transceiver)
- 16 1GB devices (PCs, RPIs, AV Receiver, Video Projector, TV, SAT->IP...)
- Dual PSU protected by UPS to prevent any possible downtime for critical devices (Alarm, Camera IP, Door Bell, Video Entryphone and so on)
- when I will go to live in, a big media center (now I'm using Emby on my Workstation, but it is full of HDDs) like this one: 4U 24 Bay SAS3 Vmware Storage Server X10QBi Includes CPU/Memory
- power drain is not a a problem
- noise is not a problem (I will dedicate a room in basement)


My concern is about the pro and cons between the two units:
ICX 6xxx is EOL and ICX7xxx is supported, so it means that the first one had the last firmware, while the latter will have new firmware (now 9.0).
But is the new firmware a worth update? Which new features does it have?
ICX7xxx seems to have less port 10GB or 40 GB ports
ICX7xxx have also PoH support (90W), but I didn't find a device that use it. Also the last Wifi 6 Ap uses the PoE+ standard. Anyone could give me a list of these devices?

After all, I ask everyone to convince me which one is the best or alternative solutions.

I'm not in hurry, because the house will be ready and the end on the year

Thank you very much

 

[adman_c]

So I'm in the process of rethinking/redoing my network so that inter-vlan routing can be done on my switch rather than my firewall (pfsense). I'm curious if it's possible for some but not all inter-vlan routing to be handled on the switch? For example, I have 2 VLANs (MGMT/HOME) that can access everything and can each access each other. But I have other VLANs that I want to remain segregated and have access only to the internet and nothing local (IOT/GUEST). Is it possible to just have VEs for MGMT and HOME, and have those routed on the switch with no ACLs, but keep forcing my other VLANs to go out to the firewall?

Thanks!

 

[kpfleming]

Yes. Any VLAN without a VE will be a layer 2 VLAN.

 

[kpfleming]

Hello everyone, I'm building a new house and I try to decide for ICX6610-48P or ICX7450-48P or two 24 port switches in stack (PoE and not-PoE)

I know that my question is already present al lot of time, but I didn't find the right answer.

My needs are:

- POE+ ports for APs, VOIP Adapters, IP Cameras, maybe POE Lighting system with sensors, Video Entryphone and any other device that I can connect (also with PoE splitter) by wire instead of WIFI
- Three 10GB and 12 1Gb connections to my little cluster (2 esxi box, 1 nas) and my workstation (with 10base-t transceiver)
- 16 1GB devices (PCs, RPIs, AV Receiver, Video Projector, TV, SAT->IP...)
- Dual PSU protected by UPS to prevent any possible downtime for critical devices (Alarm, Camera IP, Door Bell, Video Entryphone and so on)
- when I will go to live in, a big media center (now I'm using Emby on my Workstation, but it is full of HDDs) like this one: 4U 24 Bay SAS3 Vmware Storage Server X10QBi Includes CPU/Memory
- power drain is not a a problem
- noise is not a problem (I will dedicate a room in basement)


My concern is about the pro and cons between the two units:
ICX 6xxx is EOL and ICX7xxx is supported, so it means that the first one had the last firmware, while the latter will have new firmware (now 9.0).
But is the new firmware a worth update? Which new features does it have?
ICX7xxx seems to have less port 10GB or 40 GB ports
ICX7xxx have also PoH support (90W), but I didn't find a device that use it. Also the last Wifi 6 Ap uses the PoE+ standard. Anyone could give me a list of these devices?

After all, I ask everyone to convince me which one is the best or alternative solutions.

I'm not in hurry, because the house will be ready and the end on the year

Thank you very much

Given the size of this thread, it might best if you started a separate thread for this specific discussion... otherwise it will get lost in the mix with everything else here.

 

[tubs-ffm]

Is it possible to just have VEs for MGMT and HOME, and have those routed on the switch with no ACLs, but keep forcing my other VLANs to go out to the firewall?

Yes. Any VLAN without a VE will be a layer 2 VLAN.

Add VE to the VLAN you want to have L3 routing in between.
And without VE it is L2 switching, what means no routing.

I did similar here but was looking for additional control by filters.

https://forums.servethehome.com/index.php?threads/ruckus-brocade-acl-l3-default-behavior.31723/post-294921

 

[adman_c]

Yes. Any VLAN without a VE will be a layer 2 VLAN.

Add VE to the VLAN you want to have L3 routing in between.
And without VE it is L2 switching, what means no routing.

I did similar here but was looking for additional control by filters.

https://forums.servethehome.com/index.php?threads/ruckus-brocade-acl-l3-default-behavior.31723/post-294921

Great. I already have my firewall rules setup the way I want for my isolated VLANs. This way I don't need to mess with ACLs--MGMT and HOME can talk to each other freely as needed, and at line speed.

 

[clcorbin]

This may be a function of my extremely poor search skills, but I only found 3 posts in this thread talking about 2.5 Gb speeds. And none of them actually answer my question. So...

Can the x/3/x ports on the ICX 6610 support NBase-T copper (2.5Gb in my case) with the right SFP+ transceiver? I'm getting a new cable modem that supports that link speed (old modem appears to be overheating and downshifting to 100mbps) and while I don't need >gigabit speeds yet, it is only a matter of time before I do.

Thanks!

 

[Balteck]

This may be a function of my extremely poor search skills, but I only found 3 posts in this thread talking about 2.5 Gb speeds. And none of them actually answer my question. So...

Can the x/3/x ports on the ICX 6610 support NBase-T copper (2.5Gb in my case) with the right SFP+ transceiver? I'm getting a new cable modem that supports that link speed (old modem appears to be overheating and downshifting to 100mbps) and while I don't need >gigabit speeds yet, it is only a matter of time before I do.

Thanks!

This thread answers your question: https://forums.servethehome.com/ind...t-marvell-88x3300-v-s-aquantia-aqs-107.30004/

 

[mintchipmadness]

Hello All,
I am trying to setup separate wlans on my access point (ruckus r710) and wanted to see if my switch (icx 7250) setup is the problem. I currently have 3 wlans setup on the access point (AP). 1 wlan is untagged (main) and the other two are tagged (10 and 15 respectively). In the end I would like the tagged wlans to go on separate networks (192.168.5.0 and 192.168.10.0) with the untagged going to my main lan (192.168.1.0). I setup the same vlans on the switch and tagged both vlans to the port of the AP. Just in case it is the problem, I created a lag from the access point to switch (to learn how) so the port I tagged on the new vlans is lag 1. Overall, when I connect to the tagged wlans I cannot get past the AP. Is my switch setup the issue? All I did was setup the vlans and tag the ports. Should I have setup virtual interfaces for each vlan? I tried that too and I get the same result. I am guessing the issue is the router (OPNsense) but I wanted to make sure it wasn't the switch first. Thank you for your help.