Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Gerhen

Member
Nov 29, 2023
44
7
8
Hi all,
Trying to understand the difference between tagged and untagged ports and when to you which when dealing with VLANs.

I'm using untagged at the moment, but I'd like to understand why choose one vs the other, can someone provide a couple of real-world examples to help me understand how to apply this in practice?
 

audiobahn

Member
Sep 29, 2021
38
11
8
Hi all,
Trying to understand the difference between tagged and untagged ports and when to you which when dealing with VLANs.

I'm using untagged at the moment, but I'd like to understand why choose one vs the other, can someone provide a couple of real-world examples to help me understand how to apply this in practice?
Best explanation I've had so far was...

Imagine each port has a little bouncer at it's door. When a data packet arrives at the port door it's either tagged (has an access badge for a specific VLAN) or un-tagged (does not have any access badges).

If it comes tagged, as long as that access badge is allowed through the door, the bouncer will let it through, otherwise it won't.
If it's un-tagged, it's given a badge of the un-tagged designation of the port and then let it through.

Each port could have multiple "tagged" assignments but only one "un-tagged" assignment (even if it's just the "default" 1).

Example:
An un-tagged data packet from a PC turns up to Port 1 which has an un-tagged VLAN40 designation.
Since it has no badge it's given a VLAN40 badge and let in the club (switch).
It tries to go out through Port 2 which is un-tagged VLAN1 & tagged VLAN 20 & 30. The bouncer doesn't let it through.
It then tries to go out through Port 3 which is un-tagged VLAN1 & tagged VLAN 30 & 40. The bouncer let's it through since it has a valid VLAN40 badge (given to it from Port 1).

I hope this makes sense. Others might be able to explain it better.
 
Last edited:

ManoftheSea

Member
Apr 18, 2023
41
16
8
Dual mode is no longer a thing. It seems they removed it since FastIron 80.0.80. I think the VLANs setup is correct and the problem lies more with DHCP somehow. I found something about IP helper but that only seems to be available for Ports, not LAGs.
IP Helper is for unicast forwarding the packets from a broadcast in a VLAN. It should not be necessary when the client and server are on the same VLAN.

You're saying it works when you don't have the LAG setup, and just one port, but when you enable the LAG, and you've checked that the firewall sees the LAG, the DHCP broadcast stops flowing?
 

kpfleming

Active Member
Dec 28, 2021
438
223
43
Pelham NY USA
I hope this makes sense. Others might be able to explain it better.
In addition to that, think of VLANs as if they were literally separate physical networks connecting nodes together (a non-virtual LAN). If you had that, you'd have to have separate NICs/ports in every devices for each of the physical networks to connect into.

In order to avoid that, VLANs allow you to simulate having those separate physical networks, while sharing ports/NICs. To make that work, the packets have to have identifiers (badges, as above, or VLAN tags) to indicate which VLAN they belong to, so the devices can make decisions about how to handle them.

So to expand on the 'bouncer' metaphor above, which applies to both switches and hosts... on a switch, once the bouncer has a allowed a packet in, the switch has to decide which other ports (if any) that packet should be sent out on. As it sends the packet out on each of those ports, it either applies or removes a suitable badge so that the device on the other end of that link (which has its own bouncer) will be able to handle the packet when it arrives.

It's also important to understand that *inside the switch* all packets have tags (badges), always. If a packet arrives on a port without a tag, and the bouncer allows it in because that port allows incoming untagged packets, the switch will apply a tag to that packet (the 'default VLAN' on that port) so that the packet can then go through the process described above. If the packet then gets sent out on a port which uses the same default VLAN and doesn't require tags, that tag will be removed and the packet will leave the port without a tag. This is the default (out of the box) configuration of a switch in most cases: packets arrive without tags, they get 'tag 1' as they enter, and 'tag 1' is removed as they exit.
 

audiobahn

Member
Sep 29, 2021
38
11
8
IP Helper is for unicast forwarding the packets from a broadcast in a VLAN. It should not be necessary when the client and server are on the same VLAN.

You're saying it works when you don't have the LAG setup, and just one port, but when you enable the LAG, and you've checked that the firewall sees the LAG, the DHCP broadcast stops flowing?
I meant the IP helper option… it only comes up for single ports and not for the lag.

Having said that... I've been doing some experiments and it seems that VLAN 1 traffic works as expected but other VLANS don't. The way I've got things set up is that each VLAN has it's own subnet. 10.10.[VLAN].X.

I noticed the switch IP address is set up as 10.10.10.10 255.255.255.0. Should this be 10.10.10.10 255.255.0.0 instead for the switch to be able to operate within all VLAN IP ranges?

If so, how do I change that? This is what i get
Code:
SSH@roadrunner(config)#interface ve 1
SSH@roadrunner(config-vif-1)#interface ve 1
SSH@roadrunner(config-vif-1)# ip address 10.10.10.10/16
Error: duplicate ip address !
 

ManoftheSea

Member
Apr 18, 2023
41
16
8
I noticed the switch IP address is set up as 10.10.10.10 255.255.255.0. Should this be 10.10.10.10 255.255.0.0 instead for the switch to be able to operate within all VLAN IP ranges?
No. The VE has an address in the VLAN 1, but regardless of your netmask, it's not going to talk on the other VLANs. To do that, you need to have VEs in those VLANs to do the routing. But I don't think this is what you're trying to do... You want pfSense to do your routing, right? That's why you have the tagged traffic on the LAG. So you need to tell pfSense to serve DHCP over each of the tagged interfaces, or you need each of the VLANs to have a helper address of the pfSense box (10.10.10.1), a VE with an address on the VLAN and outside of 10.10.10.0/24, and pfSense needs to know how to serve addresses based on the gateway IP (the VE on the VLAN)

But really, I'd expect you to want to have routing be done by the ICX, in which case you don't need the tagged traffic to go to the pfSense box.
 

kpfleming

Active Member
Dec 28, 2021
438
223
43
Pelham NY USA
But really, I'd expect you to want to have routing be done by the ICX, in which case you don't need the tagged traffic to go to the pfSense box.
Unless you want to implement layer 3 access controls (routing policies) between those subnets (VLANs). In that case you have to choose between doing that in the ICX and letting it handle all layer 3 traffic between the subnets, or doing that in pfSense and restricting the ICX to handling only layer 2 traffic.
 

audiobahn

Member
Sep 29, 2021
38
11
8
No. The VE has an address in the VLAN 1, but regardless of your netmask, it's not going to talk on the other VLANs. To do that, you need to have VEs in those VLANs to do the routing. But I don't think this is what you're trying to do... You want pfSense to do your routing, right? That's why you have the tagged traffic on the LAG. So you need to tell pfSense to serve DHCP over each of the tagged interfaces, or you need each of the VLANs to have a helper address of the pfSense box (10.10.10.1), a VE with an address on the VLAN and outside of 10.10.10.0/24, and pfSense needs to know how to serve addresses based on the gateway IP (the VE on the VLAN)

But really, I'd expect you to want to have routing be done by the ICX, in which case you don't need the tagged traffic to go to the pfSense box.
I think we're getting close. Here's what I'm currently doing, which is what I'm trying to replicate... Once replicated I can improve routing, if the ICX can do it better.

The pfSense has a default LAN (10) and two VLANS, 20 & 40. DHCP is served by each VLAN for it's subnet on IPs & gateways 10.10.20.1 & 10.10.40.1. So an IOT device which signs on the IOT SSID, gets a 10.10.20.X IP and then is bound by the VLAN20 Firewall rules. Since the Switch sits on the LAN I pressume all "LAN" traffic doesn't get routed via pfSense but everything else must. Based on your previous answer do I just need to set helper addresses on each of the VLANs?
 

audiobahn

Member
Sep 29, 2021
38
11
8
Unless you want to implement layer 3 access controls (routing policies) between those subnets (VLANs). In that case you have to choose between doing that in the ICX and letting it handle all layer 3 traffic between the subnets, or doing that in pfSense and restricting the ICX to handling only layer 2 traffic.
I believe the ICX only doing L2 seems to reflect my current setup.
 

ManoftheSea

Member
Apr 18, 2023
41
16
8
Unless you want to implement layer 3 access controls (routing policies) between those subnets (VLANs). In that case you have to choose between doing that in the ICX and letting it handle all layer 3 traffic between the subnets, or doing that in pfSense and restricting the ICX to handling only layer 2 traffic.
And I do. Though I keep running into edge cases where it doesn't work, and I consider going back to having a Linux router...

Things the ICX6450 doesn't do: IPv6 Prefix Delegation, SSDP relay.

My user IPv6 and guest network ipv4 "firewalls", so far...
Code:
ipv6 access-list permissive
 permit tcp any any established
 permit icmp any any echo-reply
 permit icmp any any destination-unreachable
 permit icmp any any packet-too-big
 permit icmp any any time-exceeded
 permit icmp any any parameter-problem
 permit icmp any any echo-request
 permit icmp any any router-solicitation
 permit icmp any any router-advertisement
 permit icmp any any nd-ns
 permit icmp any any nd-na
 permit icmp any any 141
 permit icmp any any 142
 permit icmp fe80::/10 any 130
 permit icmp fe80::/10 any 131
 permit icmp fe80::/10 any 132
 permit icmp fe80::/10 any 143
 permit icmp any any 148
 permit icmp any any 149
 permit icmp any any 145
 permit icmp any any 146
 permit icmp any any 147
 permit udp fe80::/10 eq 547 fe80::/10 eq 546
 permit udp any host ff02::fb eq 5353
 permit udp any host ff02::f eq 1900
 permit tcp any any
 permit ahp any any
 permit esp any any
 permit sctp any any
 permit udp any any
Code:
access-list 104 permit udp host 0.0.0.0 eq bootpc host 255.255.255.255 eq bootps
access-list 104 permit udp host 172.20.20.1 eq bootps 172.20.20.0 0.0.0.255 eq bootpc
access-list 104 permit udp 172.20.20.0 0.0.0.255 host 192.168.200.5 eq dns
access-list 104 permit tcp 172.20.20.0 0.0.0.255 host 192.168.200.5 eq dns
access-list 104 permit icmp any 172.20.20.0 0.0.0.255
access-list 104 permit icmp any 192.168.0.0 0.0.255.255 echo-reply
access-list 104 deny ip any 192.168.0.0 0.0.255.255
access-list 104 deny ip any 10.0.0.0 0.255.255.255
access-list 104 deny icmp any 192.168.0.0 0.0.255.255
access-list 104 deny icmp any 10.0.0.0 0.255.255.255
access-list 104 permit ip any any
access-list 104 permit icmp any any
acess-list 104 permit igmp any any
acccess-list 104 permit udp any host 239.255.255.250 eq 1900
 

NablaSquaredG

Bringing 100G switches to homelabs
Aug 17, 2020
1,727
1,155
113
hey, most of this is covered here the header with package count and offsets is the standard FIT image type/standard Extracting Firmware - Fohdeesha Docs although I haven't had time to add UFI specific stuff
The main reason why I did this was because I wanted the newer bootloader binaries which aren't in the downloads anymore.

Would it make sense to update the bootloader versions in the guides?
 

Gerhen

Member
Nov 29, 2023
44
7
8
Hi all,

I'm trying to understand why the transit VLAN would need the trunk port designed as "untagged" for traffic to flow.

I'm in the process of setting up an ICX7230 as a router to handle inter-vlan and internal network routing with port 1/2/1 designated as the trunk port and connected to the firewall (which connects to internet).

I have 5 vlans (10, 20, 30, 40, 99) and a trunk vlan 255 specified.

If I set 1/2/1 as tagged across all vlans (10 through 255), I can't ping anything: neither between vlans nor the router upstream (note: i'm only testing for internal connectivity at this time)

If I set 1/2/1 as tagged for 10 through 99 and untagged for 255, everything works -> I can ping across vlans and the router upstream.

Can someone explain why this would be the case so as to better understand why?

I'm attaching both configurations as reference. For those looking to leverage my configuration, Config B is the one that works whereas Config A does not.
 

Attachments

Last edited:

ManoftheSea

Member
Apr 18, 2023
41
16
8
@Gerhen
First of all, in neither config does it look like you've assigned VEs to your VLANs. Within your VLAN configurations, you need to issue "router-interface ve N" statements.

I'm confused by your statement "router-on-a-stick" but also "switch handling inter-vlan". I understand inter-VLAN to mean doing the routing between VLANs, where "intra-VLAN" would be L2 switching.

Where are other devices plugged in? The router on a stick is in 1/2/1?
 
Last edited:

Gerhen

Member
Nov 29, 2023
44
7
8
@Gerhen
1. First of all, in neither config does it look like you've assigned VEs to your VLANs. Within your VLAN configurations, you need to issue "router-interface ve N" statements.

2. ...I'm confused by your statement "router-on-a-stick" but also "switch handling inter-vlan"....

3. Where are other devices plugged in? The router on a stick is in 1/2/1?
Itemizing my reply to help with readability:
1. I’m on the new 09.0.10 version of fastiron and they've removed the "router-interface ve N" statements. In this version, when you create VLAN XX, a corresponding "interface ve XX" is created where XX matches between VLAN and Interface.​
(with the 08x firmware, the matching between VLAN number to the Interface number was not a hardset rule, so they had "router-interface" there to point to the correct interface for that vlan)​
2. Regarding "router-on-a-stick", you're right! I misspoke, I've edited my prior post for clarity. I will have a "one-arm" trunk port between the firewall and the L3 router but the L3 router will handle all the internal routing (there's no L2 switch in the topology which would be the case otherwise).​
3. At this time only using two connections: 1/2/1 which connects to the firewall and 1/1/1 which is connected to one PC so that I can test that DHCP is working. I'm carefully provisioning the Brocade unit alongside my existing network in the hope that I can swap the new hardware in with minimal downtime.​
 
Last edited:

ManoftheSea

Member
Apr 18, 2023
41
16
8
1. Ah, I see. My fault, I didn't know the newer firmware did so. It's a sensible change, I'll keep it in mind for future discussion.
2. I've got a setup where my Internet Gateway Device (OpenWRT on an EspressoBIN) has a point to point connection. I don't send all the VLANs to it. If I did, I would need to tell it to expect the tagged traffic on sub-interfaces of the ethernet port. So, for your situation where only TAGGED 255 traffic works, I would want to look at the pfSense and see if it's expecting the traffic to be tagged.
3. Based on the connections, you said DHCP works when you tag VLAN 225? I see you have the helper-address set for ve/vlan 20 and that's untagged to the PC, which is all as I would expect. No DHCP snooping, so no DHCP trust assignment either.

It's not relevant to your current problem, but what's the reason behind sending all the VLANs to this device?
 

TeleFragger

Active Member
Oct 26, 2016
264
55
28
52
Brocade ICX6610 - ~$200 on ebay
  • the BEEF KING
  • 24/48 1gbE copper (PoE available)
  • 16x 10gbE (8x SFP+ in the front, 8x via 2 QSFP+ breakout ports on the rear)
  • 2x 40gbE (separate from the previously mentioned breakout ports)
  • Supports OpenFlow in hardware for SDN, including hybrid port mode
  • SupportS MACSEC on the SFP+ ports for 80gbps of real time L2 AES-128 encryption
  • Same OS features as ICX6450 but adds advanced protocols like BGP, VRFs, tunnels, everything
  • 80w power draw for the 24 port models with or without PoE
  • 110w power draw for the 48 port models with or without PoE
  • audible - about the same as an R710, little quieter than LB6M
  • 2x redundant hot-swap PSU's
  • Fans cannot be modified
  • Aggregate capacity: 528gbps / 396Mpps (wirespeed regardless of features enabled)

  • Datasheet
  • FAQ
  • Architecture Brief
  • (note: when the above PDF's say the QSFP ports can only be used for stacking, they're lying)
I am looking at one of these on ebay and I am just asking a few questions here to see if this gets me what I need.
I currently have a Brocade FastIron FCX648s and it works fine. I have an adapter in the back for 10gb (old cx4 connector they call it)

This thing is catching my eye for a few things...
1 - 10gbE - convert my machines from the old connectx-1 cx4 thick cable to sfp+
I have Reolink POE cameras and I see it says PoE Available.

So My questions:
  1. What is needed to do POE on these? so I can remove my 8 port POE switch and use this switch?
  2. do all brocade adapters from other switches work on each other model? so I can move my CX4 10gb adapter over?
  3. Found one on ebay (he has 8 left) and says SWITCH HAS BEEN RESET, NO SOFTWARE OR LICENSING WILL BE INCLUDED!
    1. My question is what the licenses is used for as i am just doing this as a home user that has a home virtual lab, etc. and not using any in switch features.
I think that is good to start with. thanks for any help.
 

bwahaha

Active Member
Jun 9, 2023
127
97
28
So My questions:
  1. What is needed to do POE on these? so I can remove my 8 port POE switch and use this switch?
  2. do all brocade adapters from other switches work on each other model? so I can move my CX4 10gb adapter over?
  3. Found one on ebay (he has 8 left) and says SWITCH HAS BEEN RESET, NO SOFTWARE OR LICENSING WILL BE INCLUDED!
    1. My question is what the licenses is used for as i am just doing this as a home user that has a home virtual lab, etc. and not using any in switch features.
1) make sure it's a poe model. There are some without.
2) sfp module? mine doesn't care about brand, so yes.
3) Follow the guides for the 6610, all licensing will be applied. You want the licensing for 10gbe.
 
  • Like
Reactions: TeleFragger

TeleFragger

Active Member
Oct 26, 2016
264
55
28
52
1) make sure it's a poe model. There are some without.
2) sfp module? mine doesn't care about brand, so yes.
3) Follow the guides for the 6610, all licensing will be applied. You want the licensing for 10gbe.
so i want the licensing... how much does that cost?
here is what i am looking at..
Brocade ICX6610-48P-E 48-port PoE+ Gigabit Ethernet Switch 8x 10GbE 1xPSU/Fan | eBay


Edited - think i found it.. they are not 10gb.. but 1gb sfp+ ports?
side note, i just noticed that my current switch has 4 sfp+ ports??? i just found a random cable from my hoarding (hah) and it fits in there..
so if i get an sfp+ card, I can just run with that or as you are saying licensing?
I know nothing about these switches, but that someone here recommended the one I have as I found it years ago with the needed port on the back to connect to my hp procurve 6400cl!
 

bwahaha

Active Member
Jun 9, 2023
127
97
28
so i want the licensing... how much does that cost?
here is what i am looking at..
Brocade ICX6610-48P-E 48-port PoE+ Gigabit Ethernet Switch 8x 10GbE 1xPSU/Fan | eBay

side note, i just noticed that my current switch has 4 sfp+ ports??? i just found a random cable from my hoarding (hah) and it fits in there..
so if i get an sfp+ card, I can just run with that or as you are saying licensing?
I know nothing about these switches, but that someone here recommended the one I have as I found it years ago with the needed port on the back to connect to my hp procurve 6400cl!
Follow the guide, licensing is applied for "free"; you pay nothing. Good seller, imo. I got mine from them.

I won't comment on licensing for the fcx, since I don't know.


Bit more info in the link to the 648s.

First there's the "stacking" FCX models: These are the FCX624S and the FCX648S (the models with "S" at the end). These are by far the most popular and easiest to find for pennies. These have two special CX4 16gbps ports on the rear to stack with each other. These "stacking" models only take the 2-port 10gbE XFP card. Search ebay for "ES4625M" to find these cards around $40. This is the easiest option if you want a good lab switch with 2x 10gbE ports for around $70 total.
 
  • Like
Reactions: TeleFragger