Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

fohdeesha

Kaini Industries
Nov 20, 2016
2,921
3,457
113
34
fohdeesha.com
Protip:
If you upgrade/migrate from the "switch mode" firmware to the "router mode" firmware in @fohdeesha 's guides, you will break your management interface and have to console into the switch ;)
reason #12324 why it's easier to just start on the layer3 FW in the first place, even if you don't need l3 features at the moment :p zero downside

For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
 
  • Like
Reactions: nickf1227

aaroneaton

New Member
Jan 15, 2021
12
0
1
www.rfehosting.com
reason #12324 why it's easier to just start on the layer3 FW in the first place, even if you don't need l3 features at the moment :p zero downside

For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
Ill give it a try. Thanks
 

ZFSZealot

New Member
Aug 16, 2021
29
7
3
For the two people above having breakout link issues, once the other end is plugged in and the server is booted up etc, can you reboot all the switches? On some chassis I've seen some weird stuff where the breakout stack ports only want to link up if the stuff is all plugged in and ready on switch boot
Regarding mine, the following comes out of the serial console on startup. A clue? I'm using 1/2/2, 1/2/3 in a static LAG, and same for 1/2/7, 1/2/8. No physical connections on 1/2/1, 1/2/4, 1/2/5, 1/2/6, 1/2/9, 1/2/10 - yet. Do all four lanes in the QSFP+ on these breakout stacking ports have to be physically connected to something?

Code:
Parsing Config Data ...
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7
EDIT: And all four of 1/2/2, 1/2/3, 1/2/7 and 1/2/8 have links up after the reboot despite the errors I pasted above. The errors must be something to do with stacking code unifying those broken out ports for stacking or something?

If this switch doesn't like servers at the end of the breakout QSFP+ ports going up and down without itself being reloaded too this may not meet my use case - I like to keep one ESXi up most of the time and only spin up the others if I need them - power use and all... Going to have to experiment and will report.
 

ZFSZealot

New Member
Aug 16, 2021
29
7
3
I'm assuming

M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7


is a safe bet that the QSFP to SFP+ DAC breakout I got is no bueno? Any possible way of troubleshooting this further?

EDIT: Upon booting looks like I'm seeing activity on the links and they do show up as up in the switch. Initially when I plugged them in the switch was on and I got nothing, so I reloaded. I guess might be a case of it not liking the modules being hotplugged?
The conversation starting here sounds almost exactly like what I'm experiencing - including adding tagged vlans to them causing the links to go down and never come back up until the switch is reloaded. I think the conclusion was that the switch was bad, but could the fact that I'm seeing the same thing show this is a bug instead? I do have another 6610 to try if it's necessary but they came from the same place so who knows if trying it will show anything conclusive. I'm running the latest 08.0.30u firmware. I do have some different breakouts coming in the mail supposedly today too - another thing to try.
 
Last edited:

chickenparm555

New Member
Sep 22, 2021
3
0
1
Hi all - two questions about the ICX7xxx series:

1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?

2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?

I found this, but it ONLY mentions the 7150:

And this, but it doesn't mention the 7150:

So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.



Thank you!!
 

LodeRunner

Active Member
Apr 27, 2019
557
237
43
Hi all - two questions about the ICX7xxx series:

1. Can the SFP+ "uplink" ports be used just like any other port on the switch? Or are there limitations on what they can/cannot be used for. (I know only certain ports can be uplinks, but can uplinks be "normal" ports)?

2. Are you able to stack any model ICX7xxx with any other model ICX7xxx? For example a ICX7150-C12P at my desk and a "bigger beef" ICX7450 in a sever rack?

I found this, but it ONLY mentions the 7150:

And this, but it doesn't mention the 7150:

So I'm concerned I can't mix the entry-level switches with the higher-end ones in a single stack.



Thank you!!
Uplink ports can be use as normal switchports.

No cross-model stacking. Must be in same family. So 71xx, 72xx, 74xx, etc.

The only exception is if you are using them as 802.1br SPX extenders with a 76, 77, or 78 series as the CB, but that's apparently gone away in v9 of the firmware, possibly indicating EoL/discontinuation of 802.1br support.
 
  • Like
Reactions: fohdeesha

chickenparm555

New Member
Sep 22, 2021
3
0
1
Uplink ports can be use as normal switchports.

No cross-model stacking. Must be in same family. So 71xx, 72xx, 74xx, etc.

The only exception is if you are using them as 802.1br SPX extenders with a 76, 77, or 78 series as the CB, but that's apparently gone away in v9 of the firmware, possibly indicating EoL/discontinuation of 802.1br support.
Awesome, thank you!
 

ZFSZealot

New Member
Aug 16, 2021
29
7
3
View attachment 19652

Yeah, the spring loaded heatsink retention pin had broken due to age/heat/thermal stress. Luckily the metal spring landed on top of the heatsink and didn't short anything. And yes, it had been running like this for multiple days. :)
I knew I had seen a post about this. kiteboarder, since you have an instance of this failure where the heatsink isn't bonded to the chip underneath, if IIRC, fohdeesha was looking for the part number off of the top of that chip, ages ago - top of page 51 in this thread. Page 301 has my discussion about it, with a response from rootwyrm about how to fix it when the heatsink becomes bonded to the chip.
 

EngChiSTH

Active Member
Jun 27, 2018
122
46
28
Chicago
Uplink ports can be use as normal switchports.

No cross-model stacking. Must be in same family. So 71xx, 72xx, 74xx, etc.

The only exception is if you are using them as 802.1br SPX extenders with a 76, 77, or 78 series as the CB, but that's apparently gone away in v9 of the firmware, possibly indicating EoL/discontinuation of 802.1br support.
Thank you - is there such thing as 'stacking guide' ? I have 6450-24 (currently running) and 6450-48 (racked but off) and realize I now need more than 4 SFP+ devices connected. is this as simple as running fiber cable between ports of 6450-24 and 6450-48 or should i instead look at something like 7250 ?
 

aindfan

New Member
Sep 25, 2021
10
4
3
tl;dr: Tried setting up an untagged vlan with router interface, client can't reach switch, and definitely can't reach upstream firewall. Routing table suggests that everything should be fine... I think?

Hi everyone, thanks for the wealth of knowledge in this thread (especially to fohdeesha for the detailed documentation!). I'm running into what I think is a basic problem with a new (to me) ICX7250-48P, and I hope someone may have some advice.

I'd like the ICX7250 to be the "core" router for my home network (all inter-vlan routing happening on the switch), with traffic to the internet going out an OpnSense firewall. My complete running config is here; here are what I think are the relevant parts:

First, define vlan 10 and corresponding router interface ve 10 (with IP 192.168.10.1/24), and plug a computer into port eth1/1/1:
Code:
vlan 10 by port
 untagged ethe 1/1/1
 router-interface ve 10
 spanning-tree 802-1w
!
interface ve 10
 ip address 192.168.10.1 255.255.255.0
Next, define vlan 253 and corresponding router interface ve 253 (with IP 192.168.253.1/24), and plug the LAN port of the OpnSense box into port eth 1/1/48:
Code:
vlan 253 by port
 untagged ethe 1/1/48
 router-interface ve 253
 spanning-tree 802-1w
!
interface ve 253
 ip address 192.168.253.1 255.255.255.0
Set the default route to towards the OpnSense box, whose LAN interface has a static IP of 192.168.253.10:
Code:
ip dns server-address 192.168.253.10
ip route 0.0.0.0/0 192.168.253.10
And finally, because a search suggested that OpnSense doesn't like being a DHCP server for subnets that aren't directly attached, use the DHCP server on the router:
Code:
ip dhcp-server enable
!
ip dhcp-server pool vlan10_corenet_pool
 excluded-address 192.168.10.1 192.168.10.19
 excluded-address 192.168.10.250 192.168.10.254
 lease 1 0 0
 network 192.168.10.0 255.255.255.0
 option  3 ip 192.168.10.1
 option  6 ip 192.168.253.10
 option  15 ascii corenet.home.my-domain-redacted.com
 deploy
As far as I can tell (from reading documentation and watching a few YouTube videos), this should work. And I have evidence that I'm on the right track: the VE's are up when I connect a PC to eth1/1/1 and the OpnSense box to eth1/1/48:
Code:
sw1#sh ip int
Interface           IP-Address      OK?  Method    Status             Protocol   VRF
Ve 1                192.168.1.1     YES  manual    down               down       default-vrf
Ve 253              192.168.253.1   YES  manual    up                 up         default-vrf
Ve 10               192.168.10.1    YES  manual    up                 up         default-vrf
The routing table looks as I'd expect it to:
Code:
sw1#sh ip route
Total number of IP routes: 3
        Destination        Gateway         Port          Cost          Type Uptime
1       0.0.0.0/0          192.168.253.10  ve 253        1/1           S    46m43s
2       192.168.10.0/24    DIRECT          ve 10         0/0           D    4m57s
3       192.168.253.0/24   DIRECT          ve 253        0/0           D    46m44s
I also know that I can reach the internet from the switch (running traceroute 8.8.8.8 on the serial console works as expected).

However, the PC connected to eth1/1/1 is not having a good time. Wireshark confirms that my PC is sending out a DHCP Discover, and the switch assigns an IP:
Code:
sw1#sh ip dhcp-server binding
Bindings from all pools:
        IP Address    Client-ID/        Lease expiration Type
                      Hardware address

     192.168.10.20    2cf0.5d7f.cc03   000d:23h:59m:41s   Automatic
The log shows that we never heard back from the PC (`No ARP-PING reply from client 192.168.10.20`). Okay, that's fine, I can assign that as a static IP on the interface on my PC. I'm still not able to ping the the router interface IP (192.168.10.1) even though it's set as the default gateway and my PC's routing table confirms that the default route is correct.

Does anyone happen to have any hints about getting past this?
 

nickf1227

Active Member
Sep 23, 2015
197
129
43
34
Can you do
show interface 1/1/1

Then
Show mac-address eth 1/1/1

Then
Show arp int eth 1/1/1

Then
show 802.1w

On your PC open a command prompt and do
Arp -a

If you setup another port untagged in vlan 10 can, and you give another device and IP, can your PC and that device see each other? Ping and Do arp -a again to confirm

Is your PC connected to another network? Is there an IP address space overlap on the other network?

Have you tried a differant cable?


This is probably not your problem, but Why do you have spanning tree running on the L3 link? Why is it a /24?
 
Last edited:

aindfan

New Member
Sep 25, 2021
10
4
3
Thanks!

Can you do
show interface 1/1/1
Code:
sw1#sh int eth 1/1/1
GigabitEthernet1/1/1 is up, line protocol is up
  Port up for 6 second(s)
  Hardware is GigabitEthernet, address is 78a6.e11b.0594 (bia 78a6.e11b.0594)
  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDIX
  EEE Feature Disabled
  Untagged member of L2 VLAN 10, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 2224 bits/sec, 3 packets/sec, 0.00% utilization
  300 second output rate: 1200 bits/sec, 1 packets/sec, 0.00% utilization
  7378 packets input, 1114915 bytes, 0 no buffer
  Received 3290 broadcasts, 4088 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  4612 packets output, 423719 bytes, 0 underruns
  Transmitted 237 broadcasts, 3247 multicasts, 1128 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4                   0                   0
         5                   0                   0
         6                   0                   0
         7                4279                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   4                   0
         2                   2                   0
         3                 327                   0
Then
Show mac-address eth 1/1/1
Code:
sw1#sh mac-address eth 1/1/1
Total active entries from port 1/1/1 = 1
MAC-Address     Port                 Type         VLAN
2cf0.5d7f.cc03  1/1/1                Dynamic      10
Then
Show arp int eth 1/1/1
Yeah, no luck here:
Code:
sw1#sh arp ethernet 1/1/1
No.   IP Address       MAC Address    Type     Age Port               Status
Then
show 802.1w
That was a heck of a lot of output. Just to simplify, I've removed spanning-tree from my VLANs; nothing has changed.

On your PC open a command prompt and do
Arp -a
Coinciding with the output above, no luck here. Just the default static entries that Windows provides. Wireshark shows that whenever I have a static IP assigned and I try to ping the gateway IP, the PC keeps sending ARP broadcasts with "Who has 192.168.10.1? Tell 192.168.10.20" (I set 192.168.10.20 as the static IP on the interface).

If you setup another port untagged in vlan 10 can, and you give another device and IP, can your PC and that device see each other? Ping and Do arp -a again to confirm
I should be able to give that a try in the next few days, but based on the arp output I'm not holding my breath that it will work.

Is your PC connected to another network? Is there an IP address space overlap on the other network?
Yes, my PC has a wifi interface with a 192.168.0.x/24 address. That's working fine.

This is probably not your problem, but Why do you have spanning tree running on the L3 link? Why is it a /24?
Honestly, I had seen that as something to enable in a guide or video somewhere and had made a note to follow up on what it actually meant later. As I mentioned above, I removed the spanning-tree config statements from the vlans and nothing changed.

Thanks again!
 

itronin

Well-Known Member
Nov 24, 2018
1,353
896
113
Denver, Colorado
tl;dr:

Does anyone happen to have any hints about getting past this?
Maybe I missed it and I'm tired and probably should not reply but here's some quick thoughts.
I did not see you mentioning the configuration you put on the opnsense box to support your not directly connected vlan.
you may be missing the route back on your opnsense box. switch is able to tr out to the iNet because its sourcing off the .253.1 int which is directly connected to opnsense so opnsense knows how to send back to the switch.

did you put a route on the opnsense box pointing back to for your vlan 10 subnet

e.g. 192.168.10.0/24 via 192.168.253.1
 
  • Like
Reactions: aindfan

aindfan

New Member
Sep 25, 2021
10
4
3
Maybe I missed it and I'm tired and probably should not reply but here's some quick thoughts.
You didn't miss it, and that's a great point. It's not directly related to the problem I'm working through with 192.168.10.1/24 not being able to talk to the switch, but it's something I was going to need to do at some point, so I appreciate the reminder. I just added the static route now.
 

nickf1227

Active Member
Sep 23, 2015
197
129
43
34
Thanks!



Code:
sw1#sh int eth 1/1/1
GigabitEthernet1/1/1 is up, line protocol is up
  Port up for 6 second(s)
  Hardware is GigabitEthernet, address is 78a6.e11b.0594 (bia 78a6.e11b.0594)
  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDIX
  EEE Feature Disabled
  Untagged member of L2 VLAN 10, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 2224 bits/sec, 3 packets/sec, 0.00% utilization
  300 second output rate: 1200 bits/sec, 1 packets/sec, 0.00% utilization
  7378 packets input, 1114915 bytes, 0 no buffer
  Received 3290 broadcasts, 4088 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  4612 packets output, 423719 bytes, 0 underruns
  Transmitted 237 broadcasts, 3247 multicasts, 1128 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4                   0                   0
         5                   0                   0
         6                   0                   0
         7                4279                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   4                   0
         2                   2                   0
         3                 327                   0


Code:
sw1#sh mac-address eth 1/1/1
Total active entries from port 1/1/1 = 1
MAC-Address     Port                 Type         VLAN
2cf0.5d7f.cc03  1/1/1                Dynamic      10


Yeah, no luck here:
Code:
sw1#sh arp ethernet 1/1/1
No.   IP Address       MAC Address    Type     Age Port               Status


That was a heck of a lot of output. Just to simplify, I've removed spanning-tree from my VLANs; nothing has changed.



Coinciding with the output above, no luck here. Just the default static entries that Windows provides. Wireshark shows that whenever I have a static IP assigned and I try to ping the gateway IP, the PC keeps sending ARP broadcasts with "Who has 192.168.10.1? Tell 192.168.10.20" (I set 192.168.10.20 as the static IP on the interface).



I should be able to give that a try in the next few days, but based on the arp output I'm not holding my breath that it will work.



Yes, my PC has a wifi interface with a 192.168.0.x/24 address. That's working fine.



Honestly, I had seen that as something to enable in a guide or video somewhere and had made a note to follow up on what it actually meant later. As I mentioned above, I removed the spanning-tree config statements from the vlans and nothing changed.

Thanks again!
I'm not seeing a problem with your config.

The problem, most likely, is a dual-horizon problem. Disable your wifi and I bet it'll work ;)

Your device isn't showing in the ARP table because you have a static IP right now.

Remember, you can only have one default gateway
 
Last edited:
  • Like
Reactions: aindfan

aindfan

New Member
Sep 25, 2021
10
4
3
Thanks again!

The problem, most likely, is a dual-horizon problem. Disable your wifi and I bet it'll work ;)
No luck, unfortunately. I disabled wifi and then connected to the switch, same symptoms. The wired NIC just gives itself a 169.254 IP even though the switch has a DHCP lease for it.

Remember, you can only have one default gateway
Right, at one point I even added a static route to my PC via the wired interface, and even that didn't get me anywhere.