Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Can you do
show interface 1/1/1

Then
Show mac-address eth 1/1/1

Then
Show arp int eth 1/1/1

Then
show 802.1w

On your PC open a command prompt and do
Arp -a

If you setup another port untagged in vlan 10 can, and you give another device and IP, can your PC and that device see each other? Ping and Do arp -a again to confirm

Is your PC connected to another network? Is there an IP address space overlap on the other network?

Have you tried a differant cable?


This is probably not your problem, but Why do you have spanning tree running on the L3 link? Why is it a /24?
 
Last edited:

aindfan

New Member
Sep 25, 2021
10
4
3
Thanks!

Can you do
show interface 1/1/1
Code:
sw1#sh int eth 1/1/1
GigabitEthernet1/1/1 is up, line protocol is up
  Port up for 6 second(s)
  Hardware is GigabitEthernet, address is 78a6.e11b.0594 (bia 78a6.e11b.0594)
  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDIX
  EEE Feature Disabled
  Untagged member of L2 VLAN 10, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 2224 bits/sec, 3 packets/sec, 0.00% utilization
  300 second output rate: 1200 bits/sec, 1 packets/sec, 0.00% utilization
  7378 packets input, 1114915 bytes, 0 no buffer
  Received 3290 broadcasts, 4088 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  4612 packets output, 423719 bytes, 0 underruns
  Transmitted 237 broadcasts, 3247 multicasts, 1128 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4                   0                   0
         5                   0                   0
         6                   0                   0
         7                4279                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   4                   0
         2                   2                   0
         3                 327                   0
Then
Show mac-address eth 1/1/1
Code:
sw1#sh mac-address eth 1/1/1
Total active entries from port 1/1/1 = 1
MAC-Address     Port                 Type         VLAN
2cf0.5d7f.cc03  1/1/1                Dynamic      10
Then
Show arp int eth 1/1/1
Yeah, no luck here:
Code:
sw1#sh arp ethernet 1/1/1
No.   IP Address       MAC Address    Type     Age Port               Status
Then
show 802.1w
That was a heck of a lot of output. Just to simplify, I've removed spanning-tree from my VLANs; nothing has changed.

On your PC open a command prompt and do
Arp -a
Coinciding with the output above, no luck here. Just the default static entries that Windows provides. Wireshark shows that whenever I have a static IP assigned and I try to ping the gateway IP, the PC keeps sending ARP broadcasts with "Who has 192.168.10.1? Tell 192.168.10.20" (I set 192.168.10.20 as the static IP on the interface).

If you setup another port untagged in vlan 10 can, and you give another device and IP, can your PC and that device see each other? Ping and Do arp -a again to confirm
I should be able to give that a try in the next few days, but based on the arp output I'm not holding my breath that it will work.

Is your PC connected to another network? Is there an IP address space overlap on the other network?
Yes, my PC has a wifi interface with a 192.168.0.x/24 address. That's working fine.

This is probably not your problem, but Why do you have spanning tree running on the L3 link? Why is it a /24?
Honestly, I had seen that as something to enable in a guide or video somewhere and had made a note to follow up on what it actually meant later. As I mentioned above, I removed the spanning-tree config statements from the vlans and nothing changed.

Thanks again!
 

itronin

Well-Known Member
Nov 24, 2018
1,234
794
113
Denver, Colorado
tl;dr:

Does anyone happen to have any hints about getting past this?
Maybe I missed it and I'm tired and probably should not reply but here's some quick thoughts.
I did not see you mentioning the configuration you put on the opnsense box to support your not directly connected vlan.
you may be missing the route back on your opnsense box. switch is able to tr out to the iNet because its sourcing off the .253.1 int which is directly connected to opnsense so opnsense knows how to send back to the switch.

did you put a route on the opnsense box pointing back to for your vlan 10 subnet

e.g. 192.168.10.0/24 via 192.168.253.1
 
  • Like
Reactions: aindfan

aindfan

New Member
Sep 25, 2021
10
4
3
Maybe I missed it and I'm tired and probably should not reply but here's some quick thoughts.
You didn't miss it, and that's a great point. It's not directly related to the problem I'm working through with 192.168.10.1/24 not being able to talk to the switch, but it's something I was going to need to do at some point, so I appreciate the reminder. I just added the static route now.
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
Thanks!



Code:
sw1#sh int eth 1/1/1
GigabitEthernet1/1/1 is up, line protocol is up
  Port up for 6 second(s)
  Hardware is GigabitEthernet, address is 78a6.e11b.0594 (bia 78a6.e11b.0594)
  Configured speed auto, actual 1Gbit, configured duplex fdx, actual fdx
  Configured mdi mode AUTO, actual MDIX
  EEE Feature Disabled
  Untagged member of L2 VLAN 10, port state is FORWARDING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  MACsec is Disabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  VLAN-Mapping is disabled
  Not member of any active trunks
  Not member of any configured trunks
  No port name
  IPG MII 96 bits-time, IPG GMII 96 bits-time
  MTU 1500 bytes, encapsulation ethernet
  MMU Mode is Store-and-forward
  300 second input rate: 2224 bits/sec, 3 packets/sec, 0.00% utilization
  300 second output rate: 1200 bits/sec, 1 packets/sec, 0.00% utilization
  7378 packets input, 1114915 bytes, 0 no buffer
  Received 3290 broadcasts, 4088 multicasts, 0 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  4612 packets output, 423719 bytes, 0 underruns
  Transmitted 237 broadcasts, 3247 multicasts, 1128 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled
  Protected: No
  MAC Port Security: Disabled

UC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   0                   0
         2                   0                   0
         3                   0                   0
         4                   0                   0
         5                   0                   0
         6                   0                   0
         7                4279                   0


MC Egress queues:
Queue counters    Queued packets    Dropped Packets
         0                   0                   0
         1                   4                   0
         2                   2                   0
         3                 327                   0


Code:
sw1#sh mac-address eth 1/1/1
Total active entries from port 1/1/1 = 1
MAC-Address     Port                 Type         VLAN
2cf0.5d7f.cc03  1/1/1                Dynamic      10


Yeah, no luck here:
Code:
sw1#sh arp ethernet 1/1/1
No.   IP Address       MAC Address    Type     Age Port               Status


That was a heck of a lot of output. Just to simplify, I've removed spanning-tree from my VLANs; nothing has changed.



Coinciding with the output above, no luck here. Just the default static entries that Windows provides. Wireshark shows that whenever I have a static IP assigned and I try to ping the gateway IP, the PC keeps sending ARP broadcasts with "Who has 192.168.10.1? Tell 192.168.10.20" (I set 192.168.10.20 as the static IP on the interface).



I should be able to give that a try in the next few days, but based on the arp output I'm not holding my breath that it will work.



Yes, my PC has a wifi interface with a 192.168.0.x/24 address. That's working fine.



Honestly, I had seen that as something to enable in a guide or video somewhere and had made a note to follow up on what it actually meant later. As I mentioned above, I removed the spanning-tree config statements from the vlans and nothing changed.

Thanks again!
I'm not seeing a problem with your config.

The problem, most likely, is a dual-horizon problem. Disable your wifi and I bet it'll work ;)

Your device isn't showing in the ARP table because you have a static IP right now.

Remember, you can only have one default gateway
 
Last edited:
  • Like
Reactions: aindfan

aindfan

New Member
Sep 25, 2021
10
4
3
Thanks again!

The problem, most likely, is a dual-horizon problem. Disable your wifi and I bet it'll work ;)
No luck, unfortunately. I disabled wifi and then connected to the switch, same symptoms. The wired NIC just gives itself a 169.254 IP even though the switch has a DHCP lease for it.

Remember, you can only have one default gateway
Right, at one point I even added a static route to my PC via the wired interface, and even that didn't get me anywhere.
 

nickf1227

Active Member
Sep 23, 2015
197
128
43
33
If there is a route between OPN sense to the 192.168.10.0/24 and the ICX and a route to the OPNSense box, can OPNSense ping 192.168.10.1?
 
  • Like
Reactions: aindfan

aindfan

New Member
Sep 25, 2021
10
4
3
If there is a route between OPN sense to the 192.168.10.0/24 and the ICX and a route to the OPNSense box, can OPNSense ping 192.168.10.1?
Yes, the OPNSense box can ping 192.168.10.1 (and even telnet in and manage the switch at that IP when I temporarily enabled the telnet server). As I'd expect, when I disconnected the PC, I got a routing loop (ping says TTL exceeded, and traceroute confirms) when I tried to traceroute from the OPNSense box to 192.168.10.1 (192.168.253.10 -> 192.168.253.1 -> back out the switch default route to 192.168.253.10 -> etc.). Reconnecting the PC made 192.168.10.1/24 reappear in the switch's routing table and made the routing loop go away (ping worked normally again).

A few more notes:
  • `show lldp neighbors` shows my PC connected (with the correct MAC on the correct port)
  • `show ip dhcp-server statistics` shows an equal number of received DHCP-DISCOVER and sent DHCP-OFFER packets. Both numbers increment every time I disconnect and reconnect my PC. No other DHCP packet types have more than 0 sent/received.
  • I get the same behavior when I connect my PC to a port with no untagged vlan associated (thus using the default vlan 1). When I set a static IP on my PC of 192.168.1.10/24 (default gateway 192.168.1.1), all I see in wireshark is my PC broadcasting out ARP packets looking for 192.168.1.1 (please tell 192.168.1.10).
Thanks again! I might reach out to the ebay seller with a link to what I've done so far in case this sounds like an RMA...
 
  • Wow
Reactions: itronin

itronin

Well-Known Member
Nov 24, 2018
1,234
794
113
Denver, Colorado
Yes, the OPNSense box can ping 192.168.10.1 (and even telnet in and manage the switch at that IP when I temporarily enabled the telnet server). As I'd expect, when I disconnected the PC, I got a routing loop (ping says TTL exceeded, and traceroute confirms) when I tried to traceroute from the OPNSense box to 192.168.10.1 (192.168.253.10 -> 192.168.253.1 -> back out the switch default route to 192.168.253.10 -> etc.). Reconnecting the PC made 192.168.10.1/24 reappear in the switch's routing table and made the routing loop go away (ping worked normally again).

A few more notes:
  • `show lldp neighbors` shows my PC connected (with the correct MAC on the correct port)
  • `show ip dhcp-server statistics` shows an equal number of received DHCP-DISCOVER and sent DHCP-OFFER packets. Both numbers increment every time I disconnect and reconnect my PC. No other DHCP packet types have more than 0 sent/received.
  • I get the same behavior when I connect my PC to a port with no untagged vlan associated (thus using the default vlan 1). When I set a static IP on my PC of 192.168.1.10/24 (default gateway 192.168.1.1), all I see in wireshark is my PC broadcasting out ARP packets looking for 192.168.1.1 (please tell 192.168.1.10).
Thanks again! I might reach out to the ebay seller with a link to what I've done so far in case this sounds like an RMA...
do you have a standalone NIC (could even by 10/100Mb) that you can test in this PC or another PC? Sounds like some sort of MAC LUT arp issue - which could be hardware too.

You probably tried this but if you didn't, add another port to VLAN 10 move your PC there and make sure the issue follows. If it were me I'd try a block away from port 1, like 9 or better yet 17 just in case your issue is on a block of ports from a single chip in the switch.
 

aindfan

New Member
Sep 25, 2021
10
4
3
do you have a standalone NIC (could even by 10/100Mb) that you can test in this PC or another PC? Sounds like some sort of MAC LUT arp issue - which could be hardware too.
Megafacepalm o'clock: good call, it works perfectly on another machine. That inspired me to consider the hardware more carefully, leading me to download the latest copy of the RealTek 2.5G NIC drivers (from RealTek, not my motherboard manufacturer).

And guess what? It works now! Thanks so much for sticking around for this troubleshooting journey and making sure that I covered all of my bases.

You probably tried this but if you didn't, add another port to VLAN 10 move your PC there and make sure the issue follows. If it were me I'd try a block away from port 1, like 9 or better yet 17 just in case your issue is on a block of ports from a single chip in the switch.
Another great thought, I did try that. Now that I updated the drivers, I'm having the same (successful) experience on any of the ports that I try.

Phew. At least I learned something there. Thanks again!
 

aindfan

New Member
Sep 25, 2021
10
4
3
p.s. And just to confirm that everything is extra super working now, I set up my Engenius AP to use tagged VLANs for different SSIDs, and I confirmed that a wifi client connecting to each SSID gets a DHCP IP from the correct pool from the switch. It's probably time to save a backup of all of these configs before I start adding access lists and IPv6 and any other fun things that will break a currently working setup...
 

itronin

Well-Known Member
Nov 24, 2018
1,234
794
113
Denver, Colorado
p.s. And just to confirm that everything is extra super working now, I set up my Engenius AP to use tagged VLANs for different SSIDs, and I confirmed that a wifi client connecting to each SSID gets a DHCP IP from the correct pool from the switch. It's probably time to save a backup of all of these configs before I start adding access lists and IPv6 and any other fun things that will break a currently working setup...
just something to keep in mind:

many folks in this thread (incl. @fohdeesha) discourage using the switch's DHCP server cause its borked in some ways. If you run into issues around DHCP - you may want to consider the possibility the DHCP server is not happy before the client... You're mostly super simple right now but that may not be the case as your journey continues. My advice, take the leap now since IP address management is foundational as your explorations get more advanced.

for me, I run a pair of Centos 7 vm's with ISC-Bind/ISC-DHCP and the stack forwards the requests.
 

aindfan

New Member
Sep 25, 2021
10
4
3
discourage using the switch's DHCP server cause its borked in some ways
Thanks! I'd seen some posts about this from ~2018 and wasn't sure if it was still the case. At the moment the only hardware that I have for this is a Fitlet2 that's running OpnSense, and I'd like to avoid running a hypervisor on my internet-facing firewall "appliance" (mostly for simplicity and stability, I have no doubt it could be done sufficiently securely). I'll probably pick up a small server sometime soon and run DHCP there, but the switch should be okay for now.
 
  • Like
Reactions: itronin

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Regarding mine, the following comes out of the serial console on startup. A clue? I'm using 1/2/2, 1/2/3 in a static LAG, and same for 1/2/7, 1/2/8. No physical connections on 1/2/1, 1/2/4, 1/2/5, 1/2/6, 1/2/9, 1/2/10 - yet. Do all four lanes in the QSFP+ on these breakout stacking ports have to be physically connected to something?

Code:
Parsing Config Data ...
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 2, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 2, port 1/2/2 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/2
------------------------------------------------------------------
M:9 L:0 - chow_qsfp_read, qsfp 3, error in seting up mux
------------------------------------------------------------------
M:9 L:0 - link_40G_4x10G_get_media: qsfp 3, port 1/2/7 error in reading qsfp
chow_40G_4x10G_get_media: error in reading qsfp 1/2/7
EDIT: And all four of 1/2/2, 1/2/3, 1/2/7 and 1/2/8 have links up after the reboot despite the errors I pasted above. The errors must be something to do with stacking code unifying those broken out ports for stacking or something?

If this switch doesn't like servers at the end of the breakout QSFP+ ports going up and down without itself being reloaded too this may not meet my use case - I like to keep one ESXi up most of the time and only spin up the others if I need them - power use and all... Going to have to experiment and will report.
Yes, I believe the chow mux errors occur when the units boot up with the breakout ports connected, it attempts the basic 4x10gb stack mux thinking a 6610 is on the other side, and of course can't because it's actually servers. As for the switch not liking the links going up and down - that certainly shouldn't be the case. I know I have some 6610's here I've used with breakout cables that didn't need to have stuff plugged in during boot for them to work. However I have seen that fix some people's link issues where they have a stubborn switch or breakout cable. Not sure what the root cause is
 
  • Like
Reactions: ZFSZealot

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Thanks! I'd seen some posts about this from ~2018 and wasn't sure if it was still the case. At the moment the only hardware that I have for this is a Fitlet2 that's running OpnSense, and I'd like to avoid running a hypervisor on my internet-facing firewall "appliance" (mostly for simplicity and stability, I have no doubt it could be done sufficiently securely). I'll probably pick up a small server sometime soon and run DHCP there, but the switch should be okay for now.
for what it's worth all the DHCP server issues we had were in the 8030 firmware (icx6xxx series), I've heard (and seen in a lot of release notes) that the DHCP server functionality has been greatly improved in 8080, 8090, and 8095, and I believe you said you have a 7250 so it can run these. worth a try. To potentially save yourself a lot of troubleshooting time, just remember in the back of your head if some random IOT devices/rokus/chromecasts suddenly don't get a DHCP lease, it's not your wifi, it's the switch's DHCP serv
 
  • Like
Reactions: aindfan

aindfan

New Member
Sep 25, 2021
10
4
3
I believe you said you have a 7250 so it can run these. worth a try. To potentially save yourself a lot of troubleshooting time, just remember in the back of your head if some random IOT devices/rokus/chromecasts suddenly don't get a DHCP lease, it's not your wifi, it's the switch's DHCP serv
Thanks! Yes, I'm running my 7250 on 8095d (from your new guide - thank you for that!). Do you happen to remember if the DHCP symptom was that the switch knew about the lease but the client never configured its IP, or did the switch not have a lease for the client at all?
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
Thanks! Yes, I'm running my 7250 on 8095d (from your new guide - thank you for that!). Do you happen to remember if the DHCP symptom was that the switch knew about the lease but the client never configured its IP, or did the switch not have a lease for the client at all?
honestly don't remember, if you search the thread I might have given those details when I was originally reporting it
 
  • Like
Reactions: aindfan and JoshDi