Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ciree8]

Hello y'all, sorry but here's another vlan post.
So I set up VyOS router as an VM. Eth0 is the WAN. and eth1 is LAN, with eth1.20 as vlan 20. It all works when I used tags on other VM. So I don't think anything wrong with my Vyos side. The issue I'm having is on my switch, which is ICX-6450-24P. Config below.

Port 1/1/13 is connected to WAN
Port 1/1/14 is VyOS WAN port(eth0)
Port 1/1/1 is VyOS LAN port(eth1 and eth1.20)

VLAN 10: 1/1/1(dual-mode) & 1/1/2(dual-mode)
VLAN 20: 1/1/1 & 1/1/3
VLAN 100: 1/1/13 & 1/1/14

When I plug my computer into 1/1/2(vlan10), it works, it can grab the DHCP server(IP:10.0.10.1). It works, I can browse the internet. So the problem I'm having is when I plug my computer to 1/1/3(vlan20), it does not work. I want it to grab vyos DHCP server(IP:10.0.20.1). What am I doing wrong? I tried setting 1/1/3 tagged, untagged, and dual-mode but those do not seem to work.

6450 configs

Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-24p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
stack disable
!
global-stp
!
!
!
vlan 10 name Internal by port
tagged ethe 1/1/1 to 1/1/2
spanning-tree 802-1w
!
vlan 20 name Main by port
tagged ethe 1/1/1 ethe 1/1/3
spanning-tree 802-1w
!
vlan 100 name WAN by port
untagged ethe 1/1/13 to 1/1/14
router-interface ve 100
spanning-tree 802-1w
!
vlan 1095 by port
!
vlan 4095 name DEFAULT-VLAN by port
router-interface ve 1
spanning-tree 802-1w
!
!
!
!
!
aaa authentication web-server default local
aaa authentication login default local
default-vlan-id 4095
enable acl-per-port-per-vlan
hostname ICX6450-24p-01
ip dhcp-client disable
!
no telnet server
username root password .....
!
!
web-management https
web-management frame bottom
web-management page-menu
web-management session-timeout 3600
!
!
router rip
!
!
!
interface ethernet 1/1/1
dual-mode 10
!
interface ethernet 1/1/2
dual-mode 10
!
interface ve 1
ip address 10.0.0.150 255.255.255.0
!
interface ve 100
!
!
!
!
!
!
!
!
!
end

VyOS config

vyos@vyos:~$ show config
firewall {
name LAN-LOCAL {
default-action accept
description "LAN outbound to LOCAL(this device)"
}
name LAN-WAN {
default-action accept
description "LAN outbound to WAN"
}
name LOCAL-LAN {
default-action accept
description "LOCAL(this device) outbound to LAN "
}
name LOCAL-WAN {
default-action accept
description "LOCAL(this device) outbound to WAN "
}
name WAN-LAN {
default-action drop
description "WAN inbound to LAN"
rule 5 {
action accept
description "Allow EST/Related Traffic"
state {
established enable
related enable
}
}
rule 20 {
action accept
protocol icmp
state {
new enable
}
}
}
name WAN-LOCAL {
default-action drop
description "WAN inbound to LOCAL(this device)"
rule 5 {
action accept
description "Allow EST/Related Traffic"
state {
established enable
related enable
}
}
rule 20 {
action accept
protocol icmp
state {
new enable
}
}
}
zone LAN {
default-action drop
from LOCAL {
firewall {
name LOCAL-WAN
}
}
from WAN {
firewall {
name WAN-LAN
}
}
interface eth1
interface eth1.20
}
zone LOCAL {
default-action drop
from LAN {
firewall {
name LAN-LOCAL
}
}
from WAN {
firewall {
name WAN-LOCAL
}
}
local-zone
}
zone WAN {
default-action drop
from LAN {
firewall {
name LAN-WAN
}
}
from LOCAL {
firewall {
name LOCAL-WAN
}
}
interface eth0
}
}
interfaces {
ethernet eth0 {
address dhcp
description WAN(internet)
}
ethernet eth1 {
address 10.0.10.1/24
description "LAN 1"
vif 20 {
address 10.0.20.1/24
description "VLAN20 Main"
}
loopback lo {
}
}
nat {
source {
rule 100 {
outbound-interface eth0
source {
address 10.0.0.0/16
}
translation {
address masquerade
}
}
}
}
service {
dhcp-server {
shared-network-name LAN1 {
subnet 10.0.10.0/24 {
default-router 10.0.10.1
name-server 10.0.10.1
range 0 {
start 10.0.10.100
stop 10.0.10.200
}
}
}
shared-network-name VLAN20 {
subnet 10.0.20.0/24 {
default-router 10.0.20.1
name-server 10.0.20.1
range 0 {
start 10.0.20.100
stop 10.0.20.200
}
}
}
}
dns {
forwarding {
allow-from 10.0.0.0/16
cache-size 0
listen-address 10.0.10.1
listen-address 10.0.20.1
name-server 10.0.1.11
}
}
ntp {
allow-client {
address 0.0.0.0/0
address ::/0
}
server time1.google.com {
}
server time2.google.com {
}
server time3.google.com {
}
}
ssh {
port 22
}
}
system {
config-management {
commit-revisions 100
}
conntrack {
modules {
ftp
h323
nfs
pptp
sip
sqlnet
tftp
}
}
console {
device ttyS0 {
speed 115200
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password ****************
plaintext-password ****************
public-keys ********************* {
key ****************
options "from="10.0.10.0/24""
type ssh-rsa
}
}
}
}
name-server eth0
syslog {
global {
facility all {
level info
}
facility protocols {
level debug
}
}
}
}

 

[Vesalius]

@ciree8 Can the computer you are plugging in set either vlan10 or vlan20 on its port natively? if not make switch port 1/1/3 untagged vlan20.

The posted start-config from the 6450 is not showing any dual-mode ports, but I suspect you are doing that in the running-config only.

From a brief overview seems vyos only defines vlan20, not vlan10. Since 1/1/1 is tagged vlan10 no way that is getting through to vyos.

As you stated, if 1/1/1 and 1/1/2 are basic dual-mode (simply means untagged in the native vlan, while tagged in another vlan), then the vyos (LAN 1 eth1) and your PC are both put into the default vlan4095 on the switch and that is how they communicate. Not sure why setting 1/1/3 dual mode does not allow the same communication though.

 

[binarynightowl]

Hey I have two ICX 6610s and a MikroTik CRS305. I have some Brocade Compatible ProLabs SFP+ optics that show up as BROCADE 57-0000076-01 10GBase LR Optics on both the MikroTik Switch as well as when I do show media 1/3/1 on the ICX6610. I have both the MikroTik switch and the ICX6610 set to 10G, and the MikroTik indicates that it is transmitting at -2.023 dBm and the module does heat up, but it shows no receiving signal. But on the ICX6610, show optic 1/3/1 has no output and the module doesn't feel warm enough to be transmitting. Is there something I need to do on the ICX6610 to get it to power up the optic? I have the 10G licences installed but it does the same when both are set to 1G as well. The single mode fiber optic that I am using is know good as well ,and Brocade 10G LR optics should work so I am kinda pulling my hair out here.

 

[ciree8]

@ciree8 Can the computer you are plugging in set either vlan10 or vlan20 on its port natively? if not make switch port 1/1/3 untagged vlan20.

The posted start-config from the 6450 is not showing any dual-mode ports, but I suspect you are doing that in the running-config only.

From a brief overview seems vyos only defines vlan20, not vlan10. Since 1/1/1 is tagged vlan10 no way that is getting through to vyos.

As you stated, if 1/1/1 and 1/1/2 are basic dual-mode (simply means untagged in the native vlan, while tagged in another vlan), then the vyos (LAN 1 eth1) and your PC are both put into the default vlan4095 on the switch and that is how they communicate. Not sure why setting 1/1/3 dual mode does not allow the same communication though.

I found the issue and fixed it!
If anyone is having the same issue, here how I fixed it: VLAN Trunking in a VM | XCP-ng Documentation
I follow that guide, setting the MTU=1504 on the hypervisor. I restarted the hypervisor. Then I had to set the MTU on the interface in VyOS, I also had to change the offloading to sg as well to get it to work. No sure I had to set ICX switch to jumbo frame, but I did anyways.
set interface ethernet eth1 offload sg
set interface ethernet eth1 mtu 1504

 

[TonyArrr]

So I lucked out

And slipped it a little. Got the Medium Beef, but the serial cable included is the rj45 one. So now I gotta wait for a mini USBcable that’s wired right

 

[pancake_riot]

I picked up an ICX7150-C12P and it's been great, aside from the POE budget being a little tighter than I expected running 3 APs and 3 cameras.

The microbeefer has 2x1G uplinks and 2x10G SFP+ uplinks. A few posters have asked about the uplink ports before, but never specifically about using both at the same time as far as I can tell.

The documentation seems to suggest that these ports behave like combos, but in the CLI they show up as their own modules (1/2/* and 1/3/*). I'm wondering if they can be used independently of one another. Has anyone been able to do that?

 

[Vesalius]

I picked up an ICX7150-C12P and it's been great, aside from the POE budget being a little tighter than I expected running 3 APs and 3 cameras.

The microbeefer has 2x1G uplinks and 2x10G SFP+ uplinks. A few posters have asked about the uplink ports before, but never specifically about using both at the same time as far as I can tell.

The documentation seems to suggest that these ports behave like combos, but in the CLI they show up as their own modules (1/2/* and 1/3/*). I'm wondering if they can be used independently of one another. Has anyone been able to do that?

Don’t have this specific model, but from what I remember, in terms of licensing the 10G capability they are both turned on or off together. In terms of physically using the ports they are independent of each other and both can be used separately.

 

[LodeRunner]

I picked up an ICX7150-C12P and it's been great, aside from the POE budget being a little tighter than I expected running 3 APs and 3 cameras.

The microbeefer has 2x1G uplinks and 2x10G SFP+ uplinks. A few posters have asked about the uplink ports before, but never specifically about using both at the same time as far as I can tell.

The documentation seems to suggest that these ports behave like combos, but in the CLI they show up as their own modules (1/2/* and 1/3/*). I'm wondering if they can be used independently of one another. Has anyone been able to do that?

The 7150-C12P doesn't have combo ports. It has 12 PoE Gbe ports, 2 non-PoE Gbe ports, and the 2 SFP+ 10Gb ports; all usable at the same time. I have used it like that.

 

[dswartz]

Yes, I've been using it since a few days after it was released.

I'm not sure what the story is. I have a POE 7150 next to my desk. Two LAG'ed ports through the drop ceiling to the stacked 7250s in the 'IT corner'. I upgraded the 7150 first. Only the primary flash until I was happy. Took awhile to install packages, and etc. And.... my windows 10 workstation (plugged into that switch) complaining about 'no internet'. After some poking, my work laptop on the same 7150 (via enet usb) and a ruckus unleashed AP all with no internet. Connected already to the 7150 via usb serial, so I try pinging various hosts on the LAN (connected to the 7250 stack), and... it works? But the local hosts don't? Hmmm. Out of desperation, I unplug and re-plug the enet cables for those 3 hosts, and that 'fixed' the issue. Any idea what the **** was going on? Here are the firmware loads:

SSH@switch2>show flash
Stack unit 1:
NAND Type: Micron NAND 2GiB (x 1)
Compressed Pri Code size = 29360128, Version:09.0.10dT211 (SPS09010d.bin)
Compressed Sec Code size = 28287100, Version:08.0.92bT211 (SPS08092b.bin)
Compressed Pri Boot Code size = 786944, Version:10.1.25T225 (mnz10125)
Compressed Sec Boot Code size = 786944, Version:10.1.17T225 (mnz10117)
Code Flash Free Space = 1019936768

this has me a little leery of upgrading the 7250 stack, I must admit...

 

[Vesalius]

@dswartz As you might be aware Ruckus does not recommend this new 09.* firmware for anything that I know of yet. Certainly not the 7150 or the 7250. Had major bugs earlier and still proving itself over time. If you are leery, 08.0.95 is a good choice. The config file format has slightly changed for some things with 09.* and downgrading can be an issue for some.

Have you tried to reload the 7150 and see if the problem recurs or whether it was a one-time post-install bootup issue?

 

[dswartz]

Well, it's working now. I haven't reloaded, since it seems to be 'fixed' ATM. When I get an hour or two with my wife out of the house, I'll reboot and see what happens... This switch isn't as mission critical as the stacked 7250s, so assuming it's stable now, I might let it slide...

 

[NablaSquaredG]

There seems to be a new series of switches, the 8000 series (surprise!)

ICX8200 | RUCKUS ICX 8200 Switches

Entry-Level+ Enterprise-Class Stackable Access Switch

www.commscope.com

ICX 8200-24
· 24× 10/100/1000 Mbps RJ-45 ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports

ICX 8200-24P PoE
· 24× 10/100/1000 Mbps RJ-45 PoE+ ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports
· 370 W PoE budget. PoE+ 802.3at

ICX 8200-24ZP Multigigabit PoE
· 24× 100/1000/2500 Mbps RJ-45 PoE++ 90W ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports
· 740 W PoE budget

ICX 8200-48
· 48× 10/100/1000 Mbps RJ-45 ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports

ICX 8200-48P PoE
· 48× 10/100/1000 Mbps RJ-45 PoE+ ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports
· 370 W PoE budget. PoE+ 802.3at

ICX 8200-48PF PoE
· 48× 10/100/1000 Mbps RJ-45 PoE+ ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports
· 740 W PoE budget. PoE+ 802.3at

Hot swappable:

ICX 8200-48PF2 PoE
· 48× 10/100/1000 Mbps RJ-45 PoE+ ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports
· 1440 W PoE budget with two PSUs (740W with one PSU)
· Dual hot swappable power supplies and fans

ICX 8200-48ZP2 Multigigabit PoE
· 32× 10/100/1000 Mbps RJ-45 PoE+ ports
· 16× 100/1000/2500 Mbps RJ-45 PoE++ 90W ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports
· 1480 W PoE budget with two PSUs (740W with one PSU)
· Dual hot swappable power supplies and fans

Compact:

ICX 8200-C08P PoE
· 8× 10/100/1000 Mbps RJ-45 PoE+ ports
· 2× 1/10GbE uplink/stacking SFP+ ports
· 124 W PoE budget PoE+ 802.3at

ICX 8200-C08ZP Multigigabit PoE
· 4× 100/1000/2500 Mbps RJ-45 PoE++ 90W ports
· 4× 1/2.5/5/10 Gbps RJ-45 PoE++ 90W ports
· 2× 1/10/25 GbE uplink/stacking SFP28 ports
· 240 W PoE budget

Fiber:

ICX 8200-24F Fiber
· 24× 1GbE SFP ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports

ICX 8200-48F Fiber
· 48× 1GbE SFP ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports

ICX 8200-24FX 10G Fiber
· 16× 1/10GbE SFP+ ports
· 8× 1/10/25 GbE uplink/stacking SFP28 ports

Maybe we're going to see more 7000s on the market soon?

 

[kpfleming]

These appear to be the replacements for the EOLed 72xx series.

 

[Revoman]

If anyone is interested, I have 2 ICX 6610's I'm looking to sell.
ICX-6610-48P (rj45 48 port PoE) and an ICX-6610-24F(24 SFP ports)
Message me if interested.

 

[zunder1990]

Ok what is the need for this switch. I fail to see the need for this switch. Fiber switches are often core/mfd switches but why not just link at 10gb then?
ICX 8200-24F Fiber
· 24× 1GbE SFP ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports

 

[bitbckt]

There seems to be a new series of switches, the 8000 series (surprise!)

The beef marches on.

 

[NablaSquaredG]

ICX 8200-24F Fiber
· 24× 1GbE SFP ports
· 4× 1/10/25 GbE uplink/stacking SFP28 ports

I could see a use case if you have FTTD (Fiber to the Desk) or ISP of an AON

 

[zunder1990]

I could see a use case if you have FTTD (Fiber to the Desk) or ISP of an AON

who is doing fiber to desk but not just gpon? such a odd switch for be a brand new switch.

 

[Vesalius]

FYI @kpfleming, @gseeley & @tubs-ffm

RUCKUS ICX FastIron 09.0.10e (GA) Software Release (.zip) and RUCKUS ICX FastIron 09.0.10e (GA) Release Notes Published or Updated on 02/10/23

For anyone using the 09.* firmware, I am dropping this here in case anyone runs into this edge case.

I run a proxmox cluster that host multiple IoT vm's and containers (Homebrige, HA, Scrypted) among other things attached to my 7150-48zp. Ever since booting the fastiron 09. series ipv4 mDNS has been a real problem, but only from the proxmox guest VM, LXC, containers, everything else running baremetal and directly connected worked fine. For the longest time, I thought it was some conflict on the proxmox linux bridge side as I also upgraded to a new proxmox version about that time as well. Resorted to setting up avahi to publish/rebroadcast from everything hosted on proxmox.

By luck, I recently found that the IXC 08.* series firmware by default enables/allows Flooding of Unregistered IPv4 Multicast Frames with any IGMP-Snooping whereas the icx 09.* series firmware default disabled/denied them. As the result the cli commands changed, but the ruckus documentation title for the page was not changed and subsequently was misleading or incorrect.

device# ip multicast flood-unregistered

FastIron 09.0.10e Multicast documentation wrong

The default for Flooding of Unregistered IPv4 Multicast Frames changed from enabled to disabled between 8.0.95 and 9.0.*, and the documentation page title does not accurately reflect that change. 8.0.95 Disabling the Flooding of Unregistered IPv4 Multicast Frames in an IGMP-Snooping-Enabled...

community.ruckuswireless.com

 

[dswartz]

For anyone using the 09.* firmware, I am dropping this here in case anyone runs into this edge case.

I run a proxmox cluster that host multiple IoT vm's and containers (Homebrige, HA, Scrypted) among other things attached to my 7150-48zp. Ever since booting the fastiron 09. series ipv4 mDNS has been a real problem, but only from the proxmox guest VM, LXC, containers, everything else running baremetal and directly connected worked fine. For the longest time, I thought it was some conflict on the proxmox linux bridge side as I also upgraded to a new proxmox version about that time as well. Resorted to setting up avahi to publish/rebroadcast from everything hosted on proxmox.

By luck, I recently found that the IXC 08.* series firmware by default enables/allows Flooding of Unregistered IPv4 Multicast Frames with any IGMP-Snooping whereas the icx 09.* series firmware disabled/denied them. As the result the cli commands changed, but the ruckus documentation title for the page was not changed and subsequently was misleading or incorrect.

device# ip multicast flood-unregistered

FastIron 09.0.10e Multicast documentation wrong

The default for Flooding of Unregistered IPv4 Multicast Frames changed from enabled to disabled between 8.0.95 and 9.0.*, and the documentation page title does not accurately reflect that change. 8.0.95 Disabling the Flooding of Unregistered IPv4 Multicast Frames in an IGMP-Snooping-Enabled...

community.ruckuswireless.com

Huh. Thanks for the tip. I have a 3-node proxmox cluster off the stacked 7250s, so that's good to know!