Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[audiobahn]

Hi everyone, I'm trying to replace my existing managed but dumb switch with an ICX to set up a LAG to extend the bandwidth of my firewall from 1GbE to 4GbE.

However, when I connect the firewall LAN interface (or confirmed working LAG ports) on the switch, devices don't seem to be picking up IP addresses from the pfSense firewall (10.10.10.1).

Needless to say I'm new to advanced switch configs. Am I missing something really obvious here? Below is my switch config.

!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.95hT213
!
stack unit 1
  module 1 icx7150-c12-poe-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-2-sfp-plus-port-20g-module
!
lag pfSense dynamic id 1
 lacp-timeout long
 ports ethe 1/1/9 to 1/1/12
!
!
vlan 1 name DEFAULT-VLAN by port
 router-interface ve 1
!
vlan 20 name IOT_VLAN20 by port
 tagged lag 1
!
vlan 30 name VPN_VLAN30 by port
 tagged lag 1
!
vlan 40 name CLRNET_VLAN40 by port
 tagged lag 1
!
aaa authentication web-server default local
aaa authentication login default local
enable aaa console
hostname roadrunner
ip dns server-address 1.1.1.1
no ip forward-protocol udp
no ip load-sharing
no ip rarp
ip route 0.0.0.0/0 10.10.10.1
ip router-id 10.10.10.10
no ip source-route
!
logging host 10.10.10.11  udp-port 6514
no telnet server
username super password .....
!
snmp-server community ..... ro
snmp-server community ..... ro
!
clock summer-time
!
ntp
 disable serve
 server 10.10.10.1
!
web-management https
!
manager registrar
!
manager port-list 987
!
interface ethernet 1/3/1
 speed-duplex 1000-full
!
interface ve 1
 ip address 10.10.10.10 255.255.255.0
!
!
ip ssh  idle-time 0
!
end

 

[ManoftheSea]

I'm not particularly competent at reading these things, but it looks like you don't have your LAG within vlan 1. So the ICX router doesn't know where 10.10.10.1 is from VE 1.
I believe you want to add "deploy" to your LAG configuration.
I believe you want to add either "tagged lag 1" or "untagged lag 1" (or "un/tagged eth 1/1/9 to 1/1/12") to your vlan 1.
Alternately, you want the ICX to only do L2 and you're trying to do all the routing on the pfSense, in which case you need to add those ports to vlans 20, 30, 40.

 

[audiobahn]

I'm not particularly competent at reading these things, but it looks like you don't have your LAG within vlan 1. So the ICX router doesn't know where 10.10.10.1 is from VE 1.
I believe you want to add "deploy" to your LAG configuration.
I believe you want to add either "tagged lag 1" or "untagged lag 1" (or "un/tagged eth 1/1/9 to 1/1/12") to your vlan 1.
Alternately, you want the ICX to only do L2 and you're trying to do all the routing on the pfSense, in which case you need to add those ports to vlans 20, 30, 40.

Hi thanks. I was under the impression all ports were by default untagged vlan 1. This seems to be the case when I tried it including lg1. No luck unfortunately.

 

[bwahaha]

Hi thanks. I was under the impression all ports were by default untagged vlan 1. This seems to be the case when I tried it including lg1. No luck unfortunately.

on my 6610, I have to enable dual-mode to allow both tagged and untagged ports.

 

[fohdeesha]

@fohdeesha

FWIW, I started digging around in the UFI file format from Ruckus, specifically because I wanted to extract the packages

The first 84 bytes of a Ruckus Firmware image are some kind of header. From this header you can reconstruct another table, which has a number of packages with a specific size each and an offset where this table is located.
For SPR08095mufi.bin for example, this is

size_per_package: 42
num_packages: 13
pkg_table_start_offset: 512
pkg_table_size: size_per_package * num_packages = 546

Although it doesn't make sense to read size_per_package from the header, as it's hardcoded as 42 in the rest of the Ruckus code...

size_per_package is the size of each entry in the package table in bytes.

Each entry has:
2 bytes of whatever - maybe a version? (acc 2)
32 bytes for the package file name, should be zero terminated, so 31 characters + '0' byte (acc 34)
4 bytes for the offset of the package in the file after the table - this is a BIG ENDIAN coded uint32 (don't ask me why it's big endian) (acc 38)
4 bytes for the filesize of the package - - this is a BIG ENDIAN coded uint32 (don't ask me why it's big endian) (acc 42)

For SPR08095mufi.bin, this gives:

size_per_package: 42
num_packages: 13
pkg_table_start_offset: 512
pkg_table_size: 546


index = 0
entry_offset_in_pkg_table = 0
pkg_file_offset_after_table = 0
total_file_offset = 1058
fileName = SPR08095m.bin
fileSize = 33554432

index = 1
entry_offset_in_pkg_table = 42
pkg_file_offset_after_table = 33554432
total_file_offset = 33555490
fileName = SPR08095m.sig
fileSize = 256

index = 2
entry_offset_in_pkg_table = 84
pkg_file_offset_after_table = 33554688
total_file_offset = 33555746
fileName = spz10126.bin
fileSize = 786944

index = 3
entry_offset_in_pkg_table = 126
pkg_file_offset_after_table = 34341632
total_file_offset = 34342690
fileName = mnz10126.bin
fileSize = 786944

index = 4
entry_offset_in_pkg_table = 168
pkg_file_offset_after_table = 35128576
total_file_offset = 35129634
fileName = Python-2.7.13.tar.gz
fileSize = 7329468

index = 5
entry_offset_in_pkg_table = 210
pkg_file_offset_after_table = 42458044
total_file_offset = 42459102
fileName = httpPkg.tar.gz
fileSize = 4944796

index = 6
entry_offset_in_pkg_table = 252
pkg_file_offset_after_table = 47402840
total_file_offset = 47403898
fileName = dhcp-4.2.5-P1.tar.gz
fileSize = 739706

index = 7
entry_offset_in_pkg_table = 294
pkg_file_offset_after_table = 48142546
total_file_offset = 48143604
fileName = perf-4.4.tar.gz
fileSize = 503799

index = 8
entry_offset_in_pkg_table = 336
pkg_file_offset_after_table = 48646345
total_file_offset = 48647403
fileName = protobuf-c-1.3.1.tar.gz
fileSize = 174130

index = 9
entry_offset_in_pkg_table = 378
pkg_file_offset_after_table = 48820475
total_file_offset = 48821533
fileName = python2-protobuf-3.7.0-1-aarch6
fileSize = 551692

index = 10
entry_offset_in_pkg_table = 420
pkg_file_offset_after_table = 49372167
total_file_offset = 49373225
fileName = python2-six-1.12.0-1-any.pkg.ta
fileSize = 19732

index = 11
entry_offset_in_pkg_table = 462
pkg_file_offset_after_table = 49391899
total_file_offset = 49392957
fileName = FileMgrPkg.tar.gz
fileSize = 3782445

index = 12
entry_offset_in_pkg_table = 504
pkg_file_offset_after_table = 53174344
total_file_offset = 53175402
fileName = logmgrPkg.tar.gz
fileSize = 7612128

I can share the code which directly extracts the files if you want. Allows you to extract current, most up to date bootloaders and stuff like that.

Just tested the extractor - Successfully extracts spz10118.bin from SPR08090mcufi.bin (tested by comparing MD5 hash of extracted file and known good file from archive)

hey, most of this is covered here the header with package count and offsets is the standard FIT image type/standard Extracting Firmware - Fohdeesha Docs although I haven't had time to add UFI specific stuff

 

[NablaSquaredG]

hey, most of this is covered here the header with package count and offsets is the standard FIT image type/standard Extracting Firmware - Fohdeesha Docs although I haven't had time to add UFI specific stuff

No, this can't be standard FIT, because the header I parse is located BEFORE the FIT (D0 0D FE ED), in the region which you strip away (hence the necessity to strip the start of the image)

The UFI format is a mixture of FIT + proprietary extensions, this is why there's a special binary called "process_packages" which is responsible for installing the packages, for SPR0809mcufi.bin it's

dhcp-4.2.5-P1.tar.gz
httpPkg.tar.gz
protobuf-c-1.3.1.tar.gz
python2-protobuf-3.7.0-1-aarch6
FileMgrPkg.tar.gz
logmgrPkg.tar.gz
perf-4.4.tar.gz
Python-2.7.13.tar.gz
python2-six-1.12.0-1-any.pkg.ta

 

[audiobahn]

on my 6610, I have to enable dual-mode to allow both tagged and untagged ports.

Dual mode is no longer a thing. It seems they removed it since FastIron 80.0.80. I think the VLANs setup is correct and the problem lies more with DHCP somehow. I found something about IP helper but that only seems to be available for Ports, not LAGs.

 

[Gerhen]

Hi all,
Trying to understand the difference between tagged and untagged ports and when to you which when dealing with VLANs.

I'm using untagged at the moment, but I'd like to understand why choose one vs the other, can someone provide a couple of real-world examples to help me understand how to apply this in practice?

 

[audiobahn]

Hi all,
Trying to understand the difference between tagged and untagged ports and when to you which when dealing with VLANs.

I'm using untagged at the moment, but I'd like to understand why choose one vs the other, can someone provide a couple of real-world examples to help me understand how to apply this in practice?

Best explanation I've had so far was...

Imagine each port has a little bouncer at it's door. When a data packet arrives at the port door it's either tagged (has an access badge for a specific VLAN) or un-tagged (does not have any access badges).

If it comes tagged, as long as that access badge is allowed through the door, the bouncer will let it through, otherwise it won't.
If it's un-tagged, it's given a badge of the un-tagged designation of the port and then let it through.

Each port could have multiple "tagged" assignments but only one "un-tagged" assignment (even if it's just the "default" 1).

Example:
An un-tagged data packet from a PC turns up to Port 1 which has an un-tagged VLAN40 designation.
Since it has no badge it's given a VLAN40 badge and let in the club (switch).
It tries to go out through Port 2 which is un-tagged VLAN1 & tagged VLAN 20 & 30. The bouncer doesn't let it through.
It then tries to go out through Port 3 which is un-tagged VLAN1 & tagged VLAN 30 & 40. The bouncer let's it through since it has a valid VLAN40 badge (given to it from Port 1).

I hope this makes sense. Others might be able to explain it better.

 

[ManoftheSea]

Dual mode is no longer a thing. It seems they removed it since FastIron 80.0.80. I think the VLANs setup is correct and the problem lies more with DHCP somehow. I found something about IP helper but that only seems to be available for Ports, not LAGs.

IP Helper is for unicast forwarding the packets from a broadcast in a VLAN. It should not be necessary when the client and server are on the same VLAN.

You're saying it works when you don't have the LAG setup, and just one port, but when you enable the LAG, and you've checked that the firewall sees the LAG, the DHCP broadcast stops flowing?

 

[kpfleming]

I hope this makes sense. Others might be able to explain it better.

In addition to that, think of VLANs as if they were literally separate physical networks connecting nodes together (a non-virtual LAN). If you had that, you'd have to have separate NICs/ports in every devices for each of the physical networks to connect into.

In order to avoid that, VLANs allow you to simulate having those separate physical networks, while sharing ports/NICs. To make that work, the packets have to have identifiers (badges, as above, or VLAN tags) to indicate which VLAN they belong to, so the devices can make decisions about how to handle them.

So to expand on the 'bouncer' metaphor above, which applies to both switches and hosts... on a switch, once the bouncer has a allowed a packet in, the switch has to decide which other ports (if any) that packet should be sent out on. As it sends the packet out on each of those ports, it either applies or removes a suitable badge so that the device on the other end of that link (which has its own bouncer) will be able to handle the packet when it arrives.

It's also important to understand that *inside the switch* all packets have tags (badges), always. If a packet arrives on a port without a tag, and the bouncer allows it in because that port allows incoming untagged packets, the switch will apply a tag to that packet (the 'default VLAN' on that port) so that the packet can then go through the process described above. If the packet then gets sent out on a port which uses the same default VLAN and doesn't require tags, that tag will be removed and the packet will leave the port without a tag. This is the default (out of the box) configuration of a switch in most cases: packets arrive without tags, they get 'tag 1' as they enter, and 'tag 1' is removed as they exit.

 

[audiobahn]

IP Helper is for unicast forwarding the packets from a broadcast in a VLAN. It should not be necessary when the client and server are on the same VLAN.

You're saying it works when you don't have the LAG setup, and just one port, but when you enable the LAG, and you've checked that the firewall sees the LAG, the DHCP broadcast stops flowing?

I meant the IP helper option… it only comes up for single ports and not for the lag.

Having said that... I've been doing some experiments and it seems that VLAN 1 traffic works as expected but other VLANS don't. The way I've got things set up is that each VLAN has it's own subnet. 10.10.[VLAN].X.

I noticed the switch IP address is set up as 10.10.10.10 255.255.255.0. Should this be 10.10.10.10 255.255.0.0 instead for the switch to be able to operate within all VLAN IP ranges?

If so, how do I change that? This is what i get

SSH@roadrunner(config)#interface ve 1
SSH@roadrunner(config-vif-1)#interface ve 1
SSH@roadrunner(config-vif-1)# ip address 10.10.10.10/16
Error: duplicate ip address !

 

[ManoftheSea]

I noticed the switch IP address is set up as 10.10.10.10 255.255.255.0. Should this be 10.10.10.10 255.255.0.0 instead for the switch to be able to operate within all VLAN IP ranges?

No. The VE has an address in the VLAN 1, but regardless of your netmask, it's not going to talk on the other VLANs. To do that, you need to have VEs in those VLANs to do the routing. But I don't think this is what you're trying to do... You want pfSense to do your routing, right? That's why you have the tagged traffic on the LAG. So you need to tell pfSense to serve DHCP over each of the tagged interfaces, or you need each of the VLANs to have a helper address of the pfSense box (10.10.10.1), a VE with an address on the VLAN and outside of 10.10.10.0/24, and pfSense needs to know how to serve addresses based on the gateway IP (the VE on the VLAN)

But really, I'd expect you to want to have routing be done by the ICX, in which case you don't need the tagged traffic to go to the pfSense box.

 

[kpfleming]

But really, I'd expect you to want to have routing be done by the ICX, in which case you don't need the tagged traffic to go to the pfSense box.

Unless you want to implement layer 3 access controls (routing policies) between those subnets (VLANs). In that case you have to choose between doing that in the ICX and letting it handle all layer 3 traffic between the subnets, or doing that in pfSense and restricting the ICX to handling only layer 2 traffic.

 

[audiobahn]

No. The VE has an address in the VLAN 1, but regardless of your netmask, it's not going to talk on the other VLANs. To do that, you need to have VEs in those VLANs to do the routing. But I don't think this is what you're trying to do... You want pfSense to do your routing, right? That's why you have the tagged traffic on the LAG. So you need to tell pfSense to serve DHCP over each of the tagged interfaces, or you need each of the VLANs to have a helper address of the pfSense box (10.10.10.1), a VE with an address on the VLAN and outside of 10.10.10.0/24, and pfSense needs to know how to serve addresses based on the gateway IP (the VE on the VLAN)

But really, I'd expect you to want to have routing be done by the ICX, in which case you don't need the tagged traffic to go to the pfSense box.

I think we're getting close. Here's what I'm currently doing, which is what I'm trying to replicate... Once replicated I can improve routing, if the ICX can do it better.

The pfSense has a default LAN (10) and two VLANS, 20 & 40. DHCP is served by each VLAN for it's subnet on IPs & gateways 10.10.20.1 & 10.10.40.1. So an IOT device which signs on the IOT SSID, gets a 10.10.20.X IP and then is bound by the VLAN20 Firewall rules. Since the Switch sits on the LAN I pressume all "LAN" traffic doesn't get routed via pfSense but everything else must. Based on your previous answer do I just need to set helper addresses on each of the VLANs?

 

[audiobahn]

Unless you want to implement layer 3 access controls (routing policies) between those subnets (VLANs). In that case you have to choose between doing that in the ICX and letting it handle all layer 3 traffic between the subnets, or doing that in pfSense and restricting the ICX to handling only layer 2 traffic.

I believe the ICX only doing L2 seems to reflect my current setup.

 

[audiobahn]

Good news is that I found a guide on exactly what I'm after... Thanks for your help everyone. Link for future reference below:

pfSense router-on-a-stick VLAN configuration with a Brocade ICX7000 series switch

nguvu.org

 

[ManoftheSea]

Unless you want to implement layer 3 access controls (routing policies) between those subnets (VLANs). In that case you have to choose between doing that in the ICX and letting it handle all layer 3 traffic between the subnets, or doing that in pfSense and restricting the ICX to handling only layer 2 traffic.

And I do. Though I keep running into edge cases where it doesn't work, and I consider going back to having a Linux router...

Things the ICX6450 doesn't do: IPv6 Prefix Delegation, SSDP relay.

My user IPv6 and guest network ipv4 "firewalls", so far...

ipv6 access-list permissive
 permit tcp any any established
 permit icmp any any echo-reply
 permit icmp any any destination-unreachable
 permit icmp any any packet-too-big
 permit icmp any any time-exceeded
 permit icmp any any parameter-problem
 permit icmp any any echo-request
 permit icmp any any router-solicitation
 permit icmp any any router-advertisement
 permit icmp any any nd-ns
 permit icmp any any nd-na
 permit icmp any any 141
 permit icmp any any 142
 permit icmp fe80::/10 any 130
 permit icmp fe80::/10 any 131
 permit icmp fe80::/10 any 132
 permit icmp fe80::/10 any 143
 permit icmp any any 148
 permit icmp any any 149
 permit icmp any any 145
 permit icmp any any 146
 permit icmp any any 147
 permit udp fe80::/10 eq 547 fe80::/10 eq 546
 permit udp any host ff02::fb eq 5353
 permit udp any host ff02::f eq 1900
 permit tcp any any
 permit ahp any any
 permit esp any any
 permit sctp any any
 permit udp any any

access-list 104 permit udp host 0.0.0.0 eq bootpc host 255.255.255.255 eq bootps
access-list 104 permit udp host 172.20.20.1 eq bootps 172.20.20.0 0.0.0.255 eq bootpc
access-list 104 permit udp 172.20.20.0 0.0.0.255 host 192.168.200.5 eq dns
access-list 104 permit tcp 172.20.20.0 0.0.0.255 host 192.168.200.5 eq dns
access-list 104 permit icmp any 172.20.20.0 0.0.0.255
access-list 104 permit icmp any 192.168.0.0 0.0.255.255 echo-reply
access-list 104 deny ip any 192.168.0.0 0.0.255.255
access-list 104 deny ip any 10.0.0.0 0.255.255.255
access-list 104 deny icmp any 192.168.0.0 0.0.255.255
access-list 104 deny icmp any 10.0.0.0 0.255.255.255
access-list 104 permit ip any any
access-list 104 permit icmp any any
acess-list 104 permit igmp any any
acccess-list 104 permit udp any host 239.255.255.250 eq 1900

 

[NablaSquaredG]

hey, most of this is covered here the header with package count and offsets is the standard FIT image type/standard Extracting Firmware - Fohdeesha Docs although I haven't had time to add UFI specific stuff

The main reason why I did this was because I wanted the newer bootloader binaries which aren't in the downloads anymore.

Would it make sense to update the bootloader versions in the guides?

 

[Gerhen]

Hi all,

I'm trying to understand why the transit VLAN would need the trunk port designed as "untagged" for traffic to flow.

I'm in the process of setting up an ICX7230 as a router to handle inter-vlan and internal network routing with port 1/2/1 designated as the trunk port and connected to the firewall (which connects to internet).

I have 5 vlans (10, 20, 30, 40, 99) and a trunk vlan 255 specified.

If I set 1/2/1 as tagged across all vlans (10 through 255), I can't ping anything: neither between vlans nor the router upstream (note: i'm only testing for internal connectivity at this time)

If I set 1/2/1 as tagged for 10 through 99 and untagged for 255, everything works -> I can ping across vlans and the router upstream.

Can someone explain why this would be the case so as to better understand why?

I'm attaching both configurations as reference. For those looking to leverage my configuration, Config B is the one that works whereas Config A does not.