Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

dswartz

Active Member
Jul 14, 2011
444
37
28
So, the eBay seller was good, got my refund, and since they were quick about things, I went ahead and ordered the last one....and same problems. This time, it is only the POE led for port 21 that is stuck on, but same, 24 of those "BCM_ERR: while 'bcm_port_mdix_set' is Feature unavailable" errors on boot and plugging anything into ports 1 thru 24 does not activate a link, but plugging in ports 25 thru 48 will bring up a link.

This, after he said they tested all the ports.

I loaded SPR08080f firmware and factory set-default, just to be sure and that didn't change anything.

Am I doing something wrong, or is this just another bad switch that failed the same exact way? I figured I'd check here before I send yet another one back. I am now hesitant to get another 7250.

ICX7250-48P
Hmmm, I wonder if you are getting 'ebayed'? I bought 2 7250-24 units on ebay, but shelled out extra for two literal "never opened" units, and they were perfect. Also, though, mine are 24 port units, so... Hard to believe there is an issue with the 48 port units like this though.
 
Last edited:

RandellH

New Member
Oct 18, 2020
3
0
1
The return and replacement were cross shipped, so I know I didn't get the "old" one sent back to me. It was 2 different switches with the same issue. That's why I'm wondering "is it me", or "is there a known issue"?
 

ArmedAviator

Member
May 16, 2020
47
21
8
Ohio
Does the ICX6450 support OSPF v3 (IPv6)? I thought it'd have identical routing features to the ICX6610 (fully licensed thanks to @fohdeesha ), but the 6450 doesn't show the ipv6 router global config option.
 

dswartz

Active Member
Jul 14, 2011
444
37
28
Interesting behavior when adding two LAG interfaces to a new VLAN. I can't add the individual ports already in the LAG, so I look around a bit more, and see that I can add lg4 and lg5 (the 2 LAGs). The odd part: after the VLAN is created, not only do the 2 LAGs show up, but the 4 ports comprising them do also. This seems by intent, as: 1) I can't add ports already in a LAG to a VLAN - they don't show as available, and 2) if I do 'modify vlan' and remove individual port(s), the LAG is removed as well. Kinda strange, but ok...
 

Attachments

Last edited:

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
Interesting behavior when adding two LAG interfaces to a new VLAN. I can't add the individual ports already in the LAG, so I look around a bit more, and see that I can add lg4 and lg5 (the 2 LAGs). The odd part: after the VLAN is created, not only do the 2 LAGs show up, but the 4 ports comprising them do also. This seems by intent, as: 1) I can't add ports already in a LAG to a VLAN - they don't show as available, and 2) if I do 'modify vlan' and remove individual port(s), the LAG is removed as well. Kinda strange, but ok...
That's how LAGs work in versions above 8030 (icx7xxx series). When you make a LAG, it's treated like its own interface so you don't have to worry about the individual port members anymore. If you want it in a vlan, you just add the LAG interface and it takes care of the rest
 

infoMatt

Active Member
Apr 16, 2019
181
76
28
That's how LAGs work in versions above 8030 (icx7xxx series). When you make a LAG, it's treated like its own interface so you don't have to worry about the individual port members anymore. If you want it in a vlan, you just add the LAG interface and it takes care of the rest
Yea but on the 6xxx lineup you can edit only the "master" or primary interface of the group (can't remember now on top of my head the exact nomenclature of the primary interface, but still...), the other interfaces are "read only". ;)

In either case, it's plainly wrong adding VLANs on member interfaces, how can it be that on a LAG only one interface can accept packets of a VLAN and others can't?
 

dswartz

Active Member
Jul 14, 2011
444
37
28
Yea but on the 6xxx lineup you can edit only the "master" or primary interface of the group (can't remember now on top of my head the exact nomenclature of the primary interface, but still...), the other interfaces are "read only". ;)

In either case, it's plainly wrong adding VLANs on member interfaces, how can it be that on a LAG only one interface can accept packets of a VLAN and others can't?
It seemed kind of non-intuitive to me. It can't make sense to have N-1 members of a LAG in the VLAN, no? Maybe this is supposed to be self-documenting? e.g. showing you which individual ports are involved?
 

fphillipeck

New Member
May 29, 2020
3
1
3
Why don't you tell us what the problem is and we can help you. It's really not that hard: create the vlan and tag or untag the ports. You could even do it from the web gui if CLI is really that hard.
The problem is I have pfSense on Proxmox. Configured for a transit vlan, since i guess pfsense cant do dhcp on non-direct connected vlans? so im gonna need the switch to do it... gateway configured on pfsense at 192.168.3.1 and a static route for 192.168.3.0/24. So on the switch, its an FCX648S fodeesha flashed, two vlans configured, vlan 3 is transit, vlan 10 is whatever. The switch is configured with a virtual router interface at 192.168.3.2, default route of 0.0.0.0/0 192.168.3.1. show mac even shows the interfaces tagged in their proper vlans, but i cant get them to ping eachother

i suspect the issue is with pfsense virtualized on proxmox somewhere, maybe pfsense doesnt like the physical nics in that machine?

hope this makes sense, im trying to be quick since ive already put hours into this thing and tbh the thing just isnt worth any more of my time and ive yet to see a successful L3 routed ping from it. Ive had this thing for like i said, 6 months, ive put weekends into playing around trying to get it to work for me.
 

fohdeesha

Kaini Industries
Nov 20, 2016
1,994
1,813
113
29
fohdeesha.com
The problem is I have pfSense on Proxmox. Configured for a transit vlan, since i guess pfsense cant do dhcp on non-direct connected vlans? so im gonna need the switch to do it... gateway configured on pfsense at 192.168.3.1 and a static route for 192.168.3.0/24. So on the switch, its an FCX648S fodeesha flashed, two vlans configured, vlan 3 is transit, vlan 10 is whatever. The switch is configured with a virtual router interface at 192.168.3.2, default route of 0.0.0.0/0 192.168.3.1. show mac even shows the interfaces tagged in their proper vlans, but i cant get them to ping eachother

i suspect the issue is with pfsense virtualized on proxmox somewhere, maybe pfsense doesnt like the physical nics in that machine?

hope this makes sense, im trying to be quick since ive already put hours into this thing and tbh the thing just isnt worth any more of my time and ive yet to see a successful L3 routed ping from it. Ive had this thing for like i said, 6 months, ive put weekends into playing around trying to get it to work for me.
if the "transit subnet" interface you made on pfsense is new, it won't have any rules on it under firewall > rules, so it won't even allow icmp (ping). you need to add rules to allow traffic. You need to also make sure you have routes on both pfsense and the icx, so pfsense knows how to get back to the subnets you've set up on the fcx on other VE's

It also sounds like you may be trunking two different VE's/vlans back to pfsense, I wouldn't do that when running l3 on both vlans because then pfsense may be seeing the same MAC address in two different VLANs (the VE interfaces will have the same MAC typically)

if what you have is:
pfsense 192.168.3.1 <-----------> FCX 192.168.3.2, don't put anything else in that vlan. Like you said, install a default route on the FCX pointing to pfsense: "ip route 0.0.0.0/0 192.168.3.1". However just as important, you need to create a static route on pfsense for any subnets you'll have on the FCX. for instance if you have vlan20 / VE 20 on the FCX created and the VE has an ip/subnet of 172.16.0.1/24, you would tell pfsense that subnet is accessible via the FCX's transit IP.

to do so in pfsense you add a new gateway, with the IP of 192.168.3.2, then add a route telling it that 172.16.0.0/24 is reachable via that gateway. In pfsense, on the youransit subnet/lan interface, do NOT fill out or select anything for " IPv4 Upstream gateway "

here are some pictures of my config that should help: pfsense l3 config

if you still can't get it I can teamviewer in and fix it for you sometime
 
  • Like
Reactions: klui

3nodeproblem

Member
Jun 28, 2020
42
11
8
Does anyone have any experience or idea of compatability of with 10gtek/sfpcables ASF-10G-T for the ICX series? Thinking of putting these in a 6450.


UPDATE: E-mailed support and they said it would be fine so taking a shot.
 
Last edited:

dswartz

Active Member
Jul 14, 2011
444
37
28
It seemed kind of non-intuitive to me. It can't make sense to have N-1 members of a LAG in the VLAN, no? Maybe this is supposed to be self-documenting? e.g. showing you which individual ports are involved?
Now that I think more on this, I think it does makes sense. Actual data will come in/go out via the LAG, but LACP packets are sent/received on the individual ports?
 

fphillipeck

New Member
May 29, 2020
3
1
3
I got it working, I appreciate the help. I was on the pfsense forums and truking vlans seemed the way to go but I had nothing but issues...
so after reading your reply i split em out and finally got it working lol
Appreciate the help :)
 
  • Like
Reactions: fohdeesha

losx

New Member
Oct 16, 2020
15
2
3
Have a question that hopefully someone can answer though it is related more to unifi gear. I have a 6450 that I have watched the videos on and gone through and set up 3 of my vlan's so far with poe power etc and I have it working BUT I have my UDMP doing the routing through the SFP+ port. I tried to leverage DHCP on the switch at first and after reading even more figured out why I was having issues. Seems some of my devices would not take the DHCP address.

Before going through and changing everything I read that pfSense does not handle DHCP duties unless the subnet is listed. Does anyone know if the Unifi USG or UDMP act the same way? I only ask because I would hate to restart setting the switch up as this is already my 3rd attempt and everything else is working well.

If this is the wrong place to ask I can also check on the Unifi forums.
 

ArmedAviator

Member
May 16, 2020
47
21
8
Ohio
I've dealt with exactly your issues a few tries now. I finally bit the bullet, set up a VM running BIND/named + ISC dhcpd and it's working so much more streamlined than pfSense - it just takes a few more keystrokes to configure - and watch out for typos.

I have never used the Unifi stuff outside of the Unifi controller software and a single WAP. I'm not sure what the L3 performance of the UDM Pro is but I suspect it is less than that of the ICX6450. I am going to blanket recommend using the ICX6450 as your L3 VLAN router and send any other traffic to the UDM Pro for internet access. You know your network needs more than me, though.

I finally am getting my network cleaned up after doing some research and trial+error. LLDP, FDP, OSPF, Loopback interfaces for management, a proper (non-pfSense) DNS+DHCP server (soon to be 2 for redundancy), a /30 trunk to my OPNSense WAN firewall for NAT duty and native IPv6 filtering with /64 subnetting, and a Lancache server running custom domains for the many installs of Void and Ubuntu Linux I have o the network - oh and Steam, XBox, and Windows updates - it's very fun downloading OS packages or games up to almost 10Gbps from a local cache.
 
  • Like
Reactions: fohdeesha

gregsachs

Active Member
Aug 14, 2018
320
88
28
Have a question that hopefully someone can answer though it is related more to unifi gear. I have a 6450 that I have watched the videos on and gone through and set up 3 of my vlan's so far with poe power etc and I have it working BUT I have my UDMP doing the routing through the SFP+ port. I tried to leverage DHCP on the switch at first and after reading even more figured out why I was having issues. Seems some of my devices would not take the DHCP address.

Before going through and changing everything I read that pfSense does not handle DHCP duties unless the subnet is listed. Does anyone know if the Unifi USG or UDMP act the same way? I only ask because I would hate to restart setting the switch up as this is already my 3rd attempt and everything else is working well.

If this is the wrong place to ask I can also check on the Unifi forums.
I am using a USG-3, I know there are some differences between USG and UDM...
I have a trunk connection between 6450 and the USG, with it doing DHCP for all vlans. This works flawlessly for me. I am limited to GBE on that link, but it is easier to manage the intra-vlan firewall on the USG than the ICX, hence ignoring the L3 on the ICX.
 

EngChiSTH

Member
Jun 27, 2018
50
18
8
Chicago
So, the eBay seller was good, got my refund, and since they were quick about things, I went ahead and ordered the last one....and same problems. This time, it is only the POE led for port 21 that is stuck on, but same, 24 of those "BCM_ERR: while 'bcm_port_mdix_set' is Feature unavailable" errors on boot and plugging anything into ports 1 thru 24 does not activate a link, but plugging in ports 25 thru 48 will bring up a link.

This, after he said they tested all the ports.

I loaded SPR08080f firmware and factory set-default, just to be sure and that didn't change anything.

Am I doing something wrong, or is this just another bad switch that failed the same exact way? I figured I'd check here before I send yet another one back. I am now hesitant to get another 7250.

ICX7250-48P
put a name here as PSA for people know what sellers to watch for/potentially avoid
 
  • Like
Reactions: fohdeesha

epicurean

Active Member
Sep 29, 2014
676
42
28
Been reading the last few post and it makes me wonder if its so bad to have pfsense handle the DHCP and firewall duties. Is it a "bad" practice? I have an ICX6450 and with help from forumers here, able to configure vlans together with pfsense with no issues so far.