Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

humm3r1

New Member
Nov 14, 2020
3
0
1
Hello,

Thank you everyone for the wealth of very useful and helpful information and links!

I wanted to seek some advice re: multicast setup.

with a Cisco 3750E stack, I enabled PIM Sparse and pim multicast routing, and had a Cisco WLC and a VM running avahi set up so that with IGMP snooping on the WLC and my SG300 switch connected to the 3750E, I could see chromecast and roku from VLAN 41 (IPTV network) into VLAN 31/32/33 where my clients reside.

I'm getting unexpected results at the moment when setting things up. I've tried setting ip pim sparse on the ve interfaces (31/32/33/41), as well as setting up an RP on VLAN 900 (with ip pim sparse).

What I find so interesting is I seem to be able to find the Roku's on wired, but not on wireless. I can find the Nest/chromecast on my phone wirelessly, but not on wired. Basically, wired = roku but no chromecast, while wireless = chromecast, but no roku. BOTH roku's and the Nest are connected to the same wifi network on VLAN 41, and I am testing from VLAN 32. I find it odd I can see 2/3 devices on wired and not the Nest, and only 1/3 on wireless.


Basic Topology:
Web -> PfSense -> ICX 6610 -> WLC -> AP -> Clients (Wireless)
Web -> PfSense -> ICX 6610 -> SG300 -> Clients (Wired, may also go directly into 6610 and skip SG300 depending on physical location, since SG300 is an office switch to extend ports)

Am I missing anything special here?

I put my config into Pastebin here for sake of easiness. I'd appreciate any guidance you can provide! I'm sure I am just misunderstanding or misconfiguring something here.


Thanks in advance!



EDIT: This config works fine, spotify on desktop sees all clients, but the roku's on iPhone hide in the airplay menu which I never thought of checking previously.
 
Last edited:

Roelf Zomerman

Active Member
Jan 10, 2019
147
27
28
blog.azureinfra.com
and its in!.. now need to silence this thing with same mod that was in this thread already..

UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
Serial #: BXK3816L033
License: ICX6610_PREM_ROUTER_SOFT_PACKAGE (LID: dzmINGLnFII)
P-ENGINE 0: type E02B, rev 01
P-ENGINE 1: type E02B, rev 01

:)

where would I see if there is a POE module as well? enabling inline power on a port doesnt do much
 
Last edited:

bdavid89

New Member
Oct 9, 2020
7
1
3
Hi,

I have a ICX6610-48-HPOE and I'm stuck at the very first steps and I can't figure it out what (basic) mistake(s) I made.

I followed fohdeesha's guide to update the bootloader and the FW and factory reset (no problems here) but after the OS loads and I set up the basic network (disabling dhcp, creating default vlan, setting a static ip, plugging the cable to the front, set ip for PC) and try to update the PoE FW it give me the error message: "TFTP session timed out"

I'm using the same TFTP app (tftpd64) which I used to update the bl/FW so I tried to ping the switch from the PC, no response, tried to ping the PC from the switch and no response.
The strange part: I opened the tftpd64's log and it looks like the switch connected but no data transfer begins. "TIMEOUT waiting for ACK block #1" (this line repeats 4 times and 4 transfer window opens all of them stuck at 0 bytes)

What I tried:

I started Wireshark and tried to ping the switch but the only thing I see is that the ping packet was sent. But if I ping the PC from the switch I see the incoming packet and the reply to it but switch still says "no reply from remote host" (same thing happens with the TFTP request they gets answered but the switch "ignores" them)

- Disabled firewall
- I connected a second PC to the switch and they can ping each other but none of them could ping the switch.
- Tried different ports (couple at the front and a QSPF port with a splitter cable)
- Tried different network card (Realtek USB ethernet and Intel X710)
- The secondary flash had on older OS, nothing changed (08.0.30sT7f3 FCXR08030s.bin)
- Upgraded to the latest? OS (08.0.30uT7f3 FCXR08030u.bin)
- Set an IP address in different subnet for the management port (and then connected the PC to it) but no difference

Configuration
Version
port1

I hope you can help me, because I don't really know what I did wrong. I hoped I can learn some advanced networking with this switch but I'm already lost at the first steps. :D
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
I connected a second PC to the switch and they can ping each other but none of them could ping the switch
Which of the two addresses have you tried? You've defined 192.168.1.1/24 on Mgmt and 192.168.2.1/24 on VE1. I suspect there's a routing problem, and I am not sure if the switch can route between management and VEs...

I am not sure, but you might be able to specify source address for the tftp transfer command... try hitting '?' and see if you can find a suitable parameter.


if I ping the PC from the switch I see the incoming packet and the reply to it
Pay close attention at the MAC address of the reply packets... is it the switch one or your default gateway one? If the PC doesn't know the route to 192.168.1.0/24 it will send the reply to the default gateway, which will discard the packet as it is not part of a known flow.
 

bdavid89

New Member
Oct 9, 2020
7
1
3
Pay close attention at the MAC address of the reply packets... is it the switch one or your default gateway one? If the PC doesn't know the route to 192.168.1.0/24 it will send the reply to the default gateway, which will discard the packet as it is not part of a known flow.
This could be the problem since the PCs that I used had 2 network adapters one connected to the switch the other one is to another network which is on different subnet (10.x.x.x) and had a default gateway (to the internet) and on the adapter which is connected to the switch I didn't configure gateway (I thought it's not needed because of the direct connection) but I will disconnect it to be sure and try that way.

I tried to used the management port because at least I knew that port worked while I updated the FW and I didn't wanted to route anything from/to there.

(ps: Sorry for these noob posts, regardless of I work as a sysadmin I don't have too much experience with these kind of switches. At my workplace our most advanced switches are a couple of Mellanox SX1012 and the "hardest" thing what I configured was to set up MLAG between them and they still working but after this I don't think anyone would believe me :D )
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Sorry for these noob posts, regardless of I work as a sysadmin I don't have too much experience with these kind of switches.
No need to apologize... everyone has started from zero at some point in their life. ;)

And every sysadmin faces those days where you can't really find the solution for hours, where maybe a colleague says "have you tried doing this or that?"... "Damn it!". It's the life.

This could be the problem since the PCs that I used had 2 network adapters one connected to the switch the other one is to another network
It shouldn't be, as a locally connected network has the priority over any route configured, but everything depends on the source address that the switch is using for the connection. I've personally never assigned an address to mgmt and VE at the same time, I don't know for sure which one it is going to use, but that to me is the major stopping block.

Maybe I should put some numbers in there. Suppose the PC has a NIC with address 192.168.2.10/24 and 10.x.x.x/yy, if the switch sends the packets with source 192.168.1.1 the reply goes to the default gateway, accordingly to the routing table.

Take a look at the packets, and assign the PC address accordingly, and all should go all right. ;)
 
Last edited:

bdavid89

New Member
Oct 9, 2020
7
1
3
I removed the management port IP, saved it and reload (just in case)

Now the IPs:
switch: 192.168.2.2/24
pc nic1: 192.168.2.7/24
pc nic2: 10.1.150.2/16 gw 10.1.0.1

And the switch is only connected to that pc nothing else

Still not working but now I can't do too much with the other NIC which is connected to the internet because I use it for remote desktop
(the switch is currently at my workplace because I wanted to try some transceivers/DACs which we have in stock so I know what can I buy. so now I remotely try to make things better... or worse :D )

I ran Wireshark again looking the two NICs but the ping request packet only appeared on the correct NIC.
Maybe it's irrelevant info but Wireshark shows a warning message at the request packet "Source MAC must not be a group address" (when pinging from the switch)

Packet capture

On Monday I will "get physical" with it and disconnect everything which is not needed.

"have you tried doing this or that?"... "Damn it!"
That happened too many times... :D
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
It might seem that POE+ is not available on my 6610-48p - how to check if that module is inserted/working?
it said right in your info dump earlier:

Code:
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
If you didn't have a PoE model, the inline power commands would return command not found. You can run "show inline power detail" to see fw version etc
 

Roelf Zomerman

Active Member
Jan 10, 2019
147
27
28
blog.azureinfra.com
it said right in your info dump earlier:

Code:
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
If you didn't have a PoE model, the inline power commands would return command not found. You can run "show inline power detail" to see fw version etc
so thats what I thought as well..

but
sh inline power detail

returns nothing

6610#inline power install-firmware stack-unit 1 tftp 172.16.5.90 fcx_poeplus_02.1.0.b004.fw
6610#Flash Memory Write (8192 bytes per dot) .........
tftp download successful file name = poe-fw
Sending PoE Firmware to Unit 1.
Stack 1 slot 1: PoE is not initialized.
 

Roelf Zomerman

Active Member
Jan 10, 2019
147
27
28
blog.azureinfra.com
can you pull the power supplies out and ensure they are PoE models? should be RPS16 (should say 1000w on it somewhere)
I got 2 P-Supplies:
P1 = Emerson AA26800L-E, PN 23-000014401 REV A : output 12V 250Watts RPS15-E
P2 = Emerson AA26800L-E, PN 23-000014401 REV B : output 12V 250Watts RPS15-E

I don't see 1000Watts anywhere..

only connected P2 - left from the back (as this is Rev B and removed P1)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
I got 2 P-Supplies:
P1 = Emerson AA26800L-E, PN 23-000014401 REV A : output 12V 250Watts
P2 = Emerson AA26800L-E, PN 23-000014401 REV B : output 12V 250Watts

I don't see 1000Watts anywhere..
someone sold you a PoE switch with the non-PoE power supplies. That's why you couldn't update the PoE firmware, the PoE board isn't getting any power. you need to swap those for the POE supplies (rps16, matching the airflow direction of the fan tray)
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
I ran Wireshark again looking the two NICs but the ping request packet only appeared on the correct NIC.
Maybe it's irrelevant info but Wireshark shows a warning message at the request packet "Source MAC must not be a group address" (when pinging from the switch)
Hmmm... very very strange... Why is the switch responding with mac address "01:92:00:00:00:00"? Hmmmm....

Now I am hitting the table with my forehead too :D Maybe @fohdeesha has seen it before.
 

bmh.01

New Member
Mar 12, 2018
6
1
3
36
Had my first real problem with my ICX6610 last night.

After setting up 2 ipv6 access-lists with about 40 ACLs each and assigning them to their respective ves, the switch rebooted itself after about 3 minutes. When it came back online, it did it again (same config). Then it did it again. No errors in the console, just a sudden reset.

I was able to login and remove the ACLs quick enough on the 4th reboot and it's been fine since.

Has anyone else noticed any reliability issues with IPv6 stuff? It seems to be that the 8.0.30 train is fairly immature with regards to IPv6.
I'd agree with the immaturity with v6 from my experience, so far I've run into:

1. /126 subnets don't work properly. A 6610 will only work as when assigned the second address, anything else results in 'bad scope' errors. They do work correctly on a 6450 though these use a different architecture.
2. IPv6 ND issues when vmotioning VMs between hosts where it take ~15 minutes for the neighbour entry to expire and the VM to become accessible over v6 on the destination host. Need to look into this more to see if it's mitigatable as there seems to be a guest OS aspect to this.
 

rootwyrm

Member
Mar 25, 2017
74
93
18
www.rootwyrm.com
Again: it is not IPv6 immaturity, it is outright lack of support because it predates final standards.
None of what you are trying to do was even standardized when this equipment was actively maintained. Anything that 'works' only works incidentally, not deliberately. (And /126 is a weird subnet regardless. I'd be surprised if it worked anywhere except JunOS.)

If you want to do IPv6 shenanigans, something running 08.0.80+ or later firmware is an absolute must.
 

bmh.01

New Member
Mar 12, 2018
6
1
3
36
It's immaturity, probably driven by a lack of development focus/effort/priority.

Plenty of other mainstream vendors supported these concepts on older platforms that reached EoL before the 6610 did.

/126 is pretty standard for P2P links when you don't want to burn /64s (and avoid neighbor cache exhaustion attacks) and the platform doesn't support RFC6164 (which dates from 2011).

The ICX 6xxx switches are well featured for the price but things like this are downsides.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,727
3,075
113
33
fohdeesha.com
It's immaturity, probably driven by a lack of development focus/effort/priority.

Plenty of other mainstream vendors supported these concepts on older platforms that reached EoL before the 6610 did.

/126 is pretty standard for P2P links when you don't want to burn /64s (and avoid neighbor cache exhaustion attacks) and the platform doesn't support RFC6164 (which dates from 2011).

The ICX 6xxx switches are well featured for the price but things like this are downsides.
RFC6164 was added in the icx7xxx series so indeed the 6 series doesn't support /127 links. I've used /126 links quite a bit now on the 6 series platforms however with zero issue, however they are all indeed using the second address, not the first. Are you the guy I discussed this with on reddit?

Sure, they have some special no touch zones, but I think saying "if you want to use IPv6, you HAVE to use a 7 series" is a really vague overstatement - I'm using 6 series in production in a few places for a few years with zero issue - no issue with ND expiry either, at least on xen hosts, live migrates between physical hosts, I certainly don't see any IPv6 downtime, or if there is it's far too small to be measured. Taking IPv6 tables in via BGP in a couple locations as well and performing a good amount of filtering, no issue. They certainly have some blind spots, but if your use case doesn't involve them (which is a LOT of use cases), they work perfectly fine for v6

Also, if any of you are paying Ruckus customers with a 6 series under support, your SE will gladly take these bug reports and forward them to the engineering team for fixing, I know last year at least 1 resolved issue in the release notes was something we reported beforehand
 

bmh.01

New Member
Mar 12, 2018
6
1
3
36
Yeh I'm the guy from reddit.

I agree with you for the money they're a decent switch, they just have some foibles like anything else. I'd probably rather have a Nexus 3064 but they're 3-4x the price and will come with another set of drawbacks to work around. It's good to hear that they're still interested in actively supporting them.

I need to do some more digging into the ND issue as it seems to work with later Linux kernel guests apart from the expected few seconds while the cache entry expires. This is with a VMware distributed switch, tuning the reachable timeout doesn't seem to effect it either although I need to check if that's for the switches advertisements or the ones it receives and caches.
 
  • Like
Reactions: fohdeesha