Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[dswartz]

Interesting behavior when adding two LAG interfaces to a new VLAN. I can't add the individual ports already in the LAG, so I look around a bit more, and see that I can add lg4 and lg5 (the 2 LAGs). The odd part: after the VLAN is created, not only do the 2 LAGs show up, but the 4 ports comprising them do also. This seems by intent, as: 1) I can't add ports already in a LAG to a VLAN - they don't show as available, and 2) if I do 'modify vlan' and remove individual port(s), the LAG is removed as well. Kinda strange, but ok...

 

[fohdeesha]

Interesting behavior when adding two LAG interfaces to a new VLAN. I can't add the individual ports already in the LAG, so I look around a bit more, and see that I can add lg4 and lg5 (the 2 LAGs). The odd part: after the VLAN is created, not only do the 2 LAGs show up, but the 4 ports comprising them do also. This seems by intent, as: 1) I can't add ports already in a LAG to a VLAN - they don't show as available, and 2) if I do 'modify vlan' and remove individual port(s), the LAG is removed as well. Kinda strange, but ok...

That's how LAGs work in versions above 8030 (icx7xxx series). When you make a LAG, it's treated like its own interface so you don't have to worry about the individual port members anymore. If you want it in a vlan, you just add the LAG interface and it takes care of the rest

 

[infoMatt]

That's how LAGs work in versions above 8030 (icx7xxx series). When you make a LAG, it's treated like its own interface so you don't have to worry about the individual port members anymore. If you want it in a vlan, you just add the LAG interface and it takes care of the rest

Yea but on the 6xxx lineup you can edit only the "master" or primary interface of the group (can't remember now on top of my head the exact nomenclature of the primary interface, but still...), the other interfaces are "read only".

In either case, it's plainly wrong adding VLANs on member interfaces, how can it be that on a LAG only one interface can accept packets of a VLAN and others can't?

 

[dswartz]

Yea but on the 6xxx lineup you can edit only the "master" or primary interface of the group (can't remember now on top of my head the exact nomenclature of the primary interface, but still...), the other interfaces are "read only".

In either case, it's plainly wrong adding VLANs on member interfaces, how can it be that on a LAG only one interface can accept packets of a VLAN and others can't?

It seemed kind of non-intuitive to me. It can't make sense to have N-1 members of a LAG in the VLAN, no? Maybe this is supposed to be self-documenting? e.g. showing you which individual ports are involved?

 

[fphillipeck]

Why don't you tell us what the problem is and we can help you. It's really not that hard: create the vlan and tag or untag the ports. You could even do it from the web gui if CLI is really that hard.

The problem is I have pfSense on Proxmox. Configured for a transit vlan, since i guess pfsense cant do dhcp on non-direct connected vlans? so im gonna need the switch to do it... gateway configured on pfsense at 192.168.3.1 and a static route for 192.168.3.0/24. So on the switch, its an FCX648S fodeesha flashed, two vlans configured, vlan 3 is transit, vlan 10 is whatever. The switch is configured with a virtual router interface at 192.168.3.2, default route of 0.0.0.0/0 192.168.3.1. show mac even shows the interfaces tagged in their proper vlans, but i cant get them to ping eachother

i suspect the issue is with pfsense virtualized on proxmox somewhere, maybe pfsense doesnt like the physical nics in that machine?

hope this makes sense, im trying to be quick since ive already put hours into this thing and tbh the thing just isnt worth any more of my time and ive yet to see a successful L3 routed ping from it. Ive had this thing for like i said, 6 months, ive put weekends into playing around trying to get it to work for me.

 

[fohdeesha]

The problem is I have pfSense on Proxmox. Configured for a transit vlan, since i guess pfsense cant do dhcp on non-direct connected vlans? so im gonna need the switch to do it... gateway configured on pfsense at 192.168.3.1 and a static route for 192.168.3.0/24. So on the switch, its an FCX648S fodeesha flashed, two vlans configured, vlan 3 is transit, vlan 10 is whatever. The switch is configured with a virtual router interface at 192.168.3.2, default route of 0.0.0.0/0 192.168.3.1. show mac even shows the interfaces tagged in their proper vlans, but i cant get them to ping eachother

i suspect the issue is with pfsense virtualized on proxmox somewhere, maybe pfsense doesnt like the physical nics in that machine?

hope this makes sense, im trying to be quick since ive already put hours into this thing and tbh the thing just isnt worth any more of my time and ive yet to see a successful L3 routed ping from it. Ive had this thing for like i said, 6 months, ive put weekends into playing around trying to get it to work for me.

if the "transit subnet" interface you made on pfsense is new, it won't have any rules on it under firewall > rules, so it won't even allow icmp (ping). you need to add rules to allow traffic. You need to also make sure you have routes on both pfsense and the icx, so pfsense knows how to get back to the subnets you've set up on the fcx on other VE's

It also sounds like you may be trunking two different VE's/vlans back to pfsense, I wouldn't do that when running l3 on both vlans because then pfsense may be seeing the same MAC address in two different VLANs (the VE interfaces will have the same MAC typically)

if what you have is:
pfsense 192.168.3.1 <-----------> FCX 192.168.3.2, don't put anything else in that vlan. Like you said, install a default route on the FCX pointing to pfsense: "ip route 0.0.0.0/0 192.168.3.1". However just as important, you need to create a static route on pfsense for any subnets you'll have on the FCX. for instance if you have vlan20 / VE 20 on the FCX created and the VE has an ip/subnet of 172.16.0.1/24, you would tell pfsense that subnet is accessible via the FCX's transit IP.

to do so in pfsense you add a new gateway, with the IP of 192.168.3.2, then add a route telling it that 172.16.0.0/24 is reachable via that gateway. In pfsense, on the youransit subnet/lan interface, do NOT fill out or select anything for " IPv4 Upstream gateway "

here are some pictures of my config that should help: pfsense l3 config

if you still can't get it I can teamviewer in and fix it for you sometime

 

[3nodeproblem]

Does anyone have any experience or idea of compatability of with 10gtek/sfpcables ASF-10G-T for the ICX series? Thinking of putting these in a 6450.

10G-T SFP+ Copper module

10Gtek ASF-10G-T RJ45 copper module, in stock, 100% compatible with Cisco, Juniper, HP Aruba, Intel, Dell, etc

www.sfpcables.com

UPDATE: E-mailed support and they said it would be fine so taking a shot.

 

[dswartz]

It seemed kind of non-intuitive to me. It can't make sense to have N-1 members of a LAG in the VLAN, no? Maybe this is supposed to be self-documenting? e.g. showing you which individual ports are involved?

Now that I think more on this, I think it does makes sense. Actual data will come in/go out via the LAG, but LACP packets are sent/received on the individual ports?

 

[fphillipeck]

I got it working, I appreciate the help. I was on the pfsense forums and truking vlans seemed the way to go but I had nothing but issues...
so after reading your reply i split em out and finally got it working lol
Appreciate the help

 

[losx]

Have a question that hopefully someone can answer though it is related more to unifi gear. I have a 6450 that I have watched the videos on and gone through and set up 3 of my vlan's so far with poe power etc and I have it working BUT I have my UDMP doing the routing through the SFP+ port. I tried to leverage DHCP on the switch at first and after reading even more figured out why I was having issues. Seems some of my devices would not take the DHCP address.

Before going through and changing everything I read that pfSense does not handle DHCP duties unless the subnet is listed. Does anyone know if the Unifi USG or UDMP act the same way? I only ask because I would hate to restart setting the switch up as this is already my 3rd attempt and everything else is working well.

If this is the wrong place to ask I can also check on the Unifi forums.

 

[ArmedAviator]

I've dealt with exactly your issues a few tries now. I finally bit the bullet, set up a VM running BIND/named + ISC dhcpd and it's working so much more streamlined than pfSense - it just takes a few more keystrokes to configure - and watch out for typos.

I have never used the Unifi stuff outside of the Unifi controller software and a single WAP. I'm not sure what the L3 performance of the UDM Pro is but I suspect it is less than that of the ICX6450. I am going to blanket recommend using the ICX6450 as your L3 VLAN router and send any other traffic to the UDM Pro for internet access. You know your network needs more than me, though.

I finally am getting my network cleaned up after doing some research and trial+error. LLDP, FDP, OSPF, Loopback interfaces for management, a proper (non-pfSense) DNS+DHCP server (soon to be 2 for redundancy), a /30 trunk to my OPNSense WAN firewall for NAT duty and native IPv6 filtering with /64 subnetting, and a Lancache server running custom domains for the many installs of Void and Ubuntu Linux I have o the network - oh and Steam, XBox, and Windows updates - it's very fun downloading OS packages or games up to almost 10Gbps from a local cache.

 

[gregsachs]

Have a question that hopefully someone can answer though it is related more to unifi gear. I have a 6450 that I have watched the videos on and gone through and set up 3 of my vlan's so far with poe power etc and I have it working BUT I have my UDMP doing the routing through the SFP+ port. I tried to leverage DHCP on the switch at first and after reading even more figured out why I was having issues. Seems some of my devices would not take the DHCP address.

Before going through and changing everything I read that pfSense does not handle DHCP duties unless the subnet is listed. Does anyone know if the Unifi USG or UDMP act the same way? I only ask because I would hate to restart setting the switch up as this is already my 3rd attempt and everything else is working well.

If this is the wrong place to ask I can also check on the Unifi forums.

I am using a USG-3, I know there are some differences between USG and UDM...
I have a trunk connection between 6450 and the USG, with it doing DHCP for all vlans. This works flawlessly for me. I am limited to GBE on that link, but it is easier to manage the intra-vlan firewall on the USG than the ICX, hence ignoring the L3 on the ICX.

 

[EngChiSTH]

So, the eBay seller was good, got my refund, and since they were quick about things, I went ahead and ordered the last one....and same problems. This time, it is only the POE led for port 21 that is stuck on, but same, 24 of those "BCM_ERR: while 'bcm_port_mdix_set' is Feature unavailable" errors on boot and plugging anything into ports 1 thru 24 does not activate a link, but plugging in ports 25 thru 48 will bring up a link.

This, after he said they tested all the ports.

I loaded SPR08080f firmware and factory set-default, just to be sure and that didn't change anything.

Am I doing something wrong, or is this just another bad switch that failed the same exact way? I figured I'd check here before I send yet another one back. I am now hesitant to get another 7250.

ICX7250-48P

put a name here as PSA for people know what sellers to watch for/potentially avoid

 

[epicurean]

Been reading the last few post and it makes me wonder if its so bad to have pfsense handle the DHCP and firewall duties. Is it a "bad" practice? I have an ICX6450 and with help from forumers here, able to configure vlans together with pfsense with no issues so far.

 

[losx]

Its not bad but then it's a layer 3 capable switch acting as a layer 2 switch. Since I have my desktop with a 10GB card and server I would prefer to transfer across the switch at full wire speed without going back to the router. Also I have IPS enabled on the router which will slow down traffic even more. Just not sure if the UDMP will do DHCP. If no one else knows I may try it out when I get time.. My guess is I screw it up a few times LOL.

Anyone have the config to show the transit gateway? Just a quick copy of the show run to give me a quick jump start?

 

[neb50]

The Edgerouter POE can do the DHCP allocation for the separate VLAN's/network segments that are defined on the switch without having to define them on the router.

Set the router IP it as the target for the ip helper-address for the VLAN interface ve
Define the different DHCP address ranges in the Edgerouter
Add a route from the Edgerouter to the IP range(s) for the VLAN(s) back to the switch IP that the Edgerouter is connected to

I think this is what you are looking for and I have been using this config setup without any issues with the Edgerouter handling the DHCP and DNS for the whole network and the switch is handling all of the VLAN routing.

 

[dswartz]

Curious behavior from the 7250-24. The command to display the ARP table only ever seems to show entries in the default VLAN. I have two others, and nothing shows for them, although entries from VLAN 2 and 3 are present in the mac address table. Is this a bug? A feature? If the latter, it sure seems odd...

 

[fohdeesha]

Curious behavior from the 7250-24. The command to display the ARP table only ever seems to show entries in the default VLAN. I have two others, and nothing shows for them, although entries from VLAN 2 and 3 are present in the mac address table. Is this a bug? A feature? If the latter, it sure seems odd...

arp is a layer 3 concept, the switch will not see or have any ARP entries for VLANs where it does not have an IP interface in said vlan (no VE). for layer 2 vlans where it's not doing IP routing, all it needs to know is MAC addresses for MAC to port mapping (layer 2 switching) which is why you see MAC entries for those vlans

 

[dswartz]

arp is a layer 3 concept, the switch will not see or have any ARP entries for VLANs where it does not have an IP interface in said vlan (no VE). for layer 2 vlans where it's not doing IP routing, all it needs to know is MAC addresses for MAC to port mapping (layer 2 switching) which is why you see MAC entries for those vlans

Makes sense, thanks.

 

[LodeRunner]

Just got a 7450 for cheap, seeing this on boot:

Spoiler

OS>set_board_level: gi_board_type = 101SOC unit 6 attached to PCI device BCM56548_A0
Initializing the parallel detect
SOC unit 7 attached to PCI device BCM56548_A0
SchanTimeOut:soc_schan_op operation timed out
  HDR[CPU=0 COS=0 EBIT=0 ECODE=0 L=4 ACC_TYPE=0 DST=7 OP=11=READ_REG_CMD]
  DW[ 0]=0x2c700200 DW[ 1]=0x02024d00
Failed to release semaphore bcm_config_lock (handle -1253745880), count 1, itr 0, error 1

stack: 004042c0 00ba641c 014652ec 011f2918 00ba9bbc 01ce2d07
Failed to release semaphore PORT_TAB (handle -1248019008), count 1, itr 0, error 1

stack: 004042c0 00ba641c 014652ec 011f29d4 00ba9bbc 01ce2d07
Failed to release semaphore bcm_link_LOCK (handle -1260167496), count 1, itr 0, error 1

stack: 004042c0 00ba641c 014652ec 0117c3cc 011f29dc 00ba9bbc 01ce2d07

 Error in TXERR WAR cpu0SchanTimeOut:soc_schan_op operation timed out
  HDR[CPU=0 COS=0 EBIT=0 ECODE=0 L=4 ACC_TYPE=0 DST=7 OP=11=READ_REG_CMD]
  DW[ 0]=0x2c700200 DW[ 1]=0x02024d00
Failed to release semaphore bcm_config_lock (handle -1253745880), count 1, itr 0, error 1

stack: 004042c0 00ba641c 014652ec 011f2918 00ba9bbc 01ce2d07
Failed to release semaphore PORT_TAB (handle -1248019008), count 1, itr 0, error 1

stack: 004042c0 00ba641c 014652ec 011f29d4 00ba9bbc 01ce2d07
Failed to release semaphore bcm_link_LOCK (handle -1260167496), count 1, itr 0, error 1

stack: 004042c0 00ba641c 014652ec 0117c3cc 011f29dc 00ba9bbc 01ce2d07

 Error in TXERR WAR cpu0Initializing the parallel detect

Not finding anything on Google about that, anything to be worried about?