Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

NablaSquaredG · Mar 23, 2024

Matta said:
I was wondering, what kind of CPU Brocade ICX6610 has ? I know it's some sort of PowerPC processor.
And how does it compare to, for example, Mikrotik CCR2116-12G-4S+ processor (Annapurna Labs Alpine AL73400 , 16 cores @ 2000Mhz).
Are L3 capabilities same, lower or higher than this particular Mikrotik ? I'm aware that one is switch and other is fully fledged router but still, one is enterprise gear, other is "affordable" class.

The CPU on the ICX6610 doesn't do a lot. It's purely management plane, and only very specific kinds of traffic are processed on the CPU.

All routing and switching happens inside the ASICs at wirespeed.

Matta · Mar 23, 2024

NablaSquaredG said:
The CPU on the ICX6610 doesn't do a lot. It's purely management plane, and only very specific kinds of traffic are processed on the CPU.

All routing and switching happens inside the ASICs at wirespeed.

Thanks for the clarification.
However, as all layer 3 switches, they are - after all - switches. Proper router is still needed if there's plan to do anything beyond plain simple routing (NAT, firewall, etc.).

i386 · Mar 23, 2024

Matta said:
Proper router is still needed if there's plan to do anything beyond plain simple routing (NAT, firewall, etc.).

bgp, ospf and rip v1 & 2 are "proper" routing protocols and supported by the icx 6610, 6450, 7250 etc....
Nat, firewall etc are functions that you find on devices that cisco calls "isr" (integrated services router)

kpfleming · Mar 23, 2024

ICX devices do not have NAT, but do have ACLs, so you can do limited firewalling on them.

richtj99 · Mar 24, 2024

I had a power question - I am thinking about upgrading a ICX6450 to a ICX7150-C12.

My POE usage on the 6450:

Code:

 Port   Admin   Oper    ---Power(mWatts)---  PD Type  PD Class  Pri  Fault/
        State   State   Consumed  Allocated                          Error
--------------------------------------------------------------------------
  1/1/2 On      On         12900      30000  802.3at  Class 4     3  n/a
  1/1/3 On      On          3800      30000  802.3at  Class 4     3  n/a
1/1/15 On      On          2300      15400  802.3af  n/a         3  n/a
1/1/18 On      On          2900      15400  802.3af  n/a         3  n/a
1/1/19 On      On          4300      15400  802.3af  n/a         3  n/a
1/1/39 On      On          2500      15400  802.3af  n/a         3  n/a
--------------------------------------------------------------------------
Total                     28700     121600

The ICX-7150-C12 has a budget of 124000 mWatts.

The consumed power on the 6450 is 28700 mWatts & allocated is 121600 mWatts which should be under the POE budget & OK

BUT
before i make a switch i wanted to make sure this would work.

ICX7150 - 10 watts idle
ICX6450 - 50 watts idle (48 port)

Savings of 40 watts - idle to idle

Looks like it is a 3 year breakeven on the cost of electric.

Thanks,
Rich

Kirkenjerk · Mar 24, 2024

Really hoping someone can shed some light on what I am doing wrong. I've been trying to figure this out for like 2 weeks now and still can't figure out how to get my setup working. I am trying to improve my networking skills so don't hate me too much if I've done something stupid.

Firewall: Opnsense
L3/Router: ICX 7250

Needs:
ICX 7250 handles all routing for the network, and the Opnsense box only does firewall things.

Problems:
I can get inter-vlan routing working (somewhat), but keep running into issues where the VLANs on the 7250 cannot hit the internet. I am not sure if its a routing issue on the switch or a NAT/Firewall/Routing issue on the Opnsense box. I can hit the Opnsense IP, and WAN IP, but not anything beyond that.

I also think I broke my config so now I can't get anything working at the moment.

If anyone has this same setup and can provide info on their Opnsense settings and their ICX 7250 config that would be amazing!

-------------------------------------------------------------------------------------------------------------------------------------------------------
Opnsense Settings:

Transit VLAN (VLAN tag 1000) for communication with switch (10.0.0.1/30) - Firewall has an IP of 10.0.0.1 on this VLAN, and the Switch has an IP of 10.0.0.2 on this VLAN. This VLAN is assigned to the ix0 port on my opnsense box, and I've tagged this VLAN on port 1/2/8 on the switch (and I've also untagged it to no avail).

Gateway was created on Opnsense for this Transit VLAN, the Gateway IP is 10.0.0.2 (the IP of the transit vlan on the switch).

Static Routes for each VLAN that exists on the switch, that is, 10.1.X.0/24 which use the Transit Gateway (10.0.0.2).

Firewall rules for the Transit VLAN on the opnsense are set to allow anything out and in. No restrictions in place here.

Outbound NAT rules are configured to allow anything outbound for each VLAN on the switch.

ICX 7250 Running Config:

Code:

Current configuration:
!
ver 08.0.95mT213
!
stack unit 1
  module 1 icx7250-48-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
  stack-port 1/2/1
  stack-port 1/2/3
!
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree
!
vlan 10 name TenGig by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1
untagged ethe 1/2/2
router-interface ve 10
spanning-tree
!
vlan 20 name DMZ by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
untagged ethe 1/1/24
router-interface ve 20
spanning-tree
!
vlan 30 name Security by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
untagged ethe 1/1/32
router-interface ve 30
spanning-tree
!
vlan 40 name Server by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
untagged ethe 1/1/40
router-interface ve 40
spanning-tree
!
vlan 50 name IOT by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
router-interface ve 50
spanning-tree
!
vlan 60 name Home by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
untagged ethe 1/1/8
router-interface ve 60
spanning-tree
!
vlan 70 name OOB by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
untagged ethe 1/1/34
router-interface ve 70
spanning-tree
!
vlan 80 name Hosts by port
tagged ethe 1/1/2 ethe 1/2/2
untagged ethe 1/1/22 ethe 1/1/28 ethe 1/1/30
router-interface ve 80
spanning-tree
!
vlan 90 name Desktops by port
tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
untagged ethe 1/1/18
router-interface ve 90
spanning-tree
!
!
vlan 1000 name transit by port
tagged ethe 1/2/8
router-interface ve 1000
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
ip dhcp-client disable
ip default-network 10.0.0.2/32
ip route next-hop-enable-default
ip route 0.0.0.0/0 10.0.0.1
ip router-id 10.0.0.2
!
no telnet server
username super password .....
!
!
!
!
no web-management http
!
manager disable
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ve 1
ip address 10.1.1.254 255.255.255.0
!
interface ve 10
ip address 10.10.10.1 255.255.255.0
!
interface ve 20
ip address 10.1.2.1 255.255.255.0
!
interface ve 30
ip address 10.1.3.1 255.255.255.0
!
interface ve 40
ip address 10.1.4.1 255.255.255.0
!
interface ve 50
ip address 10.1.5.1 255.255.255.0
!
interface ve 60
ip address 10.1.6.1 255.255.255.0
!
interface ve 70
ip address 10.1.7.1 255.255.255.0
!
interface ve 80
ip address 10.1.8.1 255.255.255.0
!
interface ve 90
ip address 10.1.9.1 255.255.255.0
!
interface ve 1000
ip address 10.0.0.2 255.255.255.252
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end

If there is any info I am missing please let me know!

richtj99 · Mar 24, 2024

Kirkenjerk said:

Code:

Current configuration:

optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
ip dhcp-client disable
ip default-network 10.0.0.2/32
ip route next-hop-enable-default
ip route 0.0.0.0/0 10.0.0.1
ip router-id 10.0.0.2

Im not sure but my config has

ip dns server-address 10.0.0.2

Maybe pointing the dns to 10.0.0.2 - your brocade or 10.0.0.1 - your firewall - or even 8.8.8.8 for testing?

Kirkenjerk · Mar 24, 2024

richtj99 said:
Im not sure but my config has

ip dns server-address 10.0.0.2

Maybe pointing the dns to 10.0.0.2 - your brocade or 10.0.0.1 - your firewall - or even 8.8.8.8 for testing?

I should have added, that from VLAN 1 on the switch (IP of 10.1.1.254) I can ping the IP of the Opnsense firewall (which is 10.1.1.1), and the internet without issue. It's only the VLANs on the 7250 that cant ping or reach the internet at all.

I don't have my DNS servers or DHCP servers configured on the 7250 at the moment, everything is done via IP or set statically until I can figure out the routing/firewall issues.

dbvader · Mar 24, 2024

Kirkenjerk said:
everything is done via IP or set statically until I can figure out the routing/firewall issues.

Do the clients connected to the switch have 10.0.0.1 or 10.0.0.2 as their (default) gateway? I believe it should be 10.0.0.2 with your config. If the gateway on the clients is 10.0.0.1 then I don't see how they would be able to reach that IP (without intermediate routing via 10.0.0.2).

opnsense (10.0.0.1) <-> switch (10.0.0.2) <-> client

Kirkenjerk · Mar 24, 2024

dbvader said:
Do the clients connected to the switch have 10.0.0.1 or 10.0.0.2 as their (default) gateway? I believe it should be 10.0.0.2 with your config. If the gateway on the clients is 10.0.0.1 then I don't see how they would be able to reach that IP (without intermediate routing via 10.0.0.2).

opnsense (10.0.0.1) <-> switch (10.0.0.2) <-> client

When setting up the VE for each vlan I set an IP. The IP of the VE is the gateway for each vlan.

I don’t think 10.0.0.2 should be set as the gateway for each client.

dbvader · Mar 24, 2024

Sort of what I meant, albeit for the wrong vlan (1000) which is clearly your opnsense <-> switch vlan.

Your switch's default route seems to be pointing to itself (10.0.0.2) instead of the opnsense IP (10.0.0.1).

Code:

ip default-network 10.0.0.2/32

Kirkenjerk · Mar 24, 2024

dbvader said:
Sort of what I meant, albeit for the wrong vlan (1000) which is clearly your opnsense <-> switch vlan.

Your switch's default route seems to be pointing to itself (10.0.0.2) instead of the opnsense IP (10.0.0.1).

Code:

ip default-network 10.0.0.2/32

You are absolutely right. I’ll try changing that and see what happens.

Kirkenjerk · Mar 24, 2024

dbvader said:
Sort of what I meant, albeit for the wrong vlan (1000) which is clearly your opnsense <-> switch vlan.

Your switch's default route seems to be pointing to itself (10.0.0.2) instead of the opnsense IP (10.0.0.1).

Code:

ip default-network 10.0.0.2/32

After making that change the switch can still ping out to the internet but the VLANs cannot.

EDIT: IT ****ING WORKS NOW. I am just stupid, and your advice pointed me in the right direction. I needed to change that default-network and also realized I never setup the route for the specific VLAN I am testing. I also had to make a change to the outbound NAT rule.

dbvader · Mar 24, 2024

Hope your (presumably vlan unaware) switch clients are not hooked up to tagged ports.

Perhaps run a traceroute from a client to 8.8.8.8 or some other internet IP address.

Kirkenjerk · Mar 24, 2024

dbvader said:
Hope your (presumably vlan unaware) switch clients are not hooked up to tagged ports.

Perhaps run a traceroute from a client to 8.8.8.8 or some other internet IP address.

I got it working!!!! Edited my last comment but i'll add it here too. You pointed me in the right direction.

I needed to change that default-network and also realized I never setup the route for the specific VLAN I am testing (I was testing vlan 10 but then switched to VLAN 90 and never made that static route). I also had to make a change to the outbound NAT rule as the cidr mask wasnt correct.

I knew...just knew it was something stupid that I was doing. Thank you so much.

richtj99 · Mar 24, 2024

Hi - would you mind reposting the working settings for your sh run? I would like to see how that works as my router is doing the routing but might be nice to have the brocade do it.

Kirkenjerk · Mar 24, 2024

richtj99 said:
Hi - would you mind reposting the working settings for your sh run? I would like to see how that works as my router is doing the routing but might be nice to have the brocade do it.

Code:

Current configuration:
!
ver 08.0.95mT213
!
stack unit 1
  module 1 icx7250-48-port-management-module
  module 2 icx7250-sfp-plus-8port-80g-module
  stack-port 1/2/1
  stack-port 1/2/3
!
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
 spanning-tree
!
vlan 10 name TenGig by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1
 untagged ethe 1/2/2
 router-interface ve 10
 spanning-tree
!
vlan 20 name DMZ by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 untagged ethe 1/1/24
 router-interface ve 20
 spanning-tree
!
vlan 30 name Security by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 untagged ethe 1/1/32
 router-interface ve 30
 spanning-tree
!
vlan 40 name Server by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 untagged ethe 1/1/40
 router-interface ve 40
 spanning-tree
!
vlan 50 name IOT by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 router-interface ve 50
 spanning-tree
!
vlan 60 name Home by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 untagged ethe 1/1/8
 router-interface ve 60
 spanning-tree
!
vlan 70 name OOB by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 untagged ethe 1/1/34
 router-interface ve 70
 spanning-tree
!
vlan 80 name Hosts by port
 tagged ethe 1/1/2 ethe 1/2/2
 untagged ethe 1/1/22 ethe 1/1/28 ethe 1/1/30
 router-interface ve 80
 spanning-tree
!
vlan 90 name Desktops by port
 tagged ethe 1/1/2 ethe 1/1/22 ethe 1/1/28 ethe 1/1/30 ethe 1/2/1 to 1/2/2
 untagged ethe 1/1/18
 router-interface ve 90
 spanning-tree
!
!
vlan 1000 name transit by port
 tagged ethe 1/2/8
 router-interface ve 1000
!
!
!
!
!
!
!
!
!
!
!
!
!
optical-monitor
optical-monitor non-ruckus-optic-enable
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
ip dhcp-client disable
ip default-network 10.0.0.0/8
ip default-network 10.0.0.1/32
ip route next-hop-enable-default
ip route 0.0.0.0/0 10.0.0.1
ip router-id 10.0.0.2
!
no telnet server
username super password .....
!
!
!
!
no web-management http
!
manager disable
!
!
manager port-list 987
!
!
!
!
!
!
!
!
!
interface ve 1
 ip address 10.1.1.254 255.255.255.0
!
interface ve 10
 ip address 10.10.10.1 255.255.255.0
!
interface ve 20
 ip address 10.1.2.1 255.255.255.0
!
interface ve 30
 ip address 10.1.3.1 255.255.255.0
!
interface ve 40
 ip address 10.1.4.1 255.255.255.0
!
interface ve 50
 ip address 10.1.5.1 255.255.255.0
!
interface ve 60
 ip address 10.1.6.1 255.255.255.0
!
interface ve 70
 ip address 10.1.7.1 255.255.255.0
!
interface ve 80
 ip address 10.1.8.1 255.255.255.0
!
interface ve 90
 ip address 10.1.9.1 255.255.255.0
!
interface ve 1000
 ip address 10.0.0.2 255.255.255.252
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
end

If you need the Opnsense settings let me know.

TonyArrr · Mar 24, 2024

richtj99 said:
I had a power question - I am thinking about upgrading a ICX6450 to a ICX7150-C12.

My POE usage on the 6450:

Code:

Port Admin Oper ---Power(mWatts)--- PD Type PD Class Pri Fault/ State State Consumed Allocated Error -------------------------------------------------------------------------- 1/1/2 On On 12900 30000 802.3at Class 4 3 n/a 1/1/3 On On 3800 30000 802.3at Class 4 3 n/a 1/1/15 On On 2300 15400 802.3af n/a 3 n/a 1/1/18 On On 2900 15400 802.3af n/a 3 n/a 1/1/19 On On 4300 15400 802.3af n/a 3 n/a 1/1/39 On On 2500 15400 802.3af n/a 3 n/a -------------------------------------------------------------------------- Total 28700 121600

The ICX-7150-C12 has a budget of 124000 mWatts.

The consumed power on the 6450 is 28700 mWatts & allocated is 121600 mWatts which should be under the POE budget & OK

BUT
before i make a switch i wanted to make sure this would work.

ICX7150 - 10 watts idle
ICX6450 - 50 watts idle (48 port)

Savings of 40 watts - idle to idle

Looks like it is a 3 year breakeven on the cost of electric.

Thanks,
Rich

My only thought here is to make sure there is at least a little airflow running by the 7150-C12. Mine has PoE off and it still gets quite toasty. When I did my read through of this thread, there was a patch where people kept having the PSUs of these suffer heat death (very fixable, but obviously annoying).

It doesn’t take much to keep it very cool though, a 80mm fan sitting on top at 900rpm has it as cool as a consumer dumb switch

and it’s totally inaudible still.

None of this is to say they’re bad switches, they just seem to be walking a fine line with the heat by the time they make it to the second hand market. They’re otherwise great, I’m actually on the prowl for a second one

Sealside · Mar 25, 2024

junicast said:
Thank you. I will check, maybe I will try to remove the 40x40 fans completely and put 2 times 80x80 in the top of the case. They will run more silently with even more throughput.

I'm in the same situation. I have 3x 40mm mechatronics in the back and a 20mm on the asic cooler. Still after some time they will spin up. My conclusion is to go for 80mm fans in order to reach acceptable noise and cooling.

Another approach would be to take noctuas 40mm and always run them on max voltage plus faking rpm with a esp8266, but i don't think the will provide enough cooling for the long run.

/S

junicast · Mar 25, 2024

Sealside said:
I'm in the same situation. I have 3x 40mm mechatronics in the back and a 20mm on the asic cooler. Still after some time they will spin up. My conclusion is to go for 80mm fans in order to reach acceptable noise and cooling.

Another approach would be to take noctuas 40mm and always run them on max voltage plus faking rpm with a esp8266, but i don't think the will provide enough cooling for the long run.

/S

I tried just that. Actually I went for 2 x 120mm fans but when the system slow the fans down, the switch is only reachable for like 1 or 2 seconds and then suddenly reboots. The same happens if I connect 2 x 80mm fans.
I do NOT know but my guess is there is a threshold in the firmware that the fans must have a minimum speed and if that's not guaranteed, the system reboots in order to *fix* the *problem*.

Edit:
I tried with one Maglev 40mm fan combined with a 120mm case fan. This does not result in a rebooting device. This is the fan status.

Code:

SSH@7250.example.com#dm fan

Fan 1 Speed at 573 RPM.

Fan 2 Speed at 2760 RPM.

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Bringing 100G switches to homelabs

Member

Well-Known Member

Active Member

Member

New Member

Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

New Member

Member

New Member

Active Member

Active Member

New Member