Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

brob

New Member
Feb 3, 2021
2
0
1
Hi

I am having troubles getting the computers on my VLAN's to be able to access the internet through my 6450. I can ping google and my firewall from the CLI on the 6450, but am unable to ping from any computer on my network (google or even my firewall).

I have been trying all day to get this to work. looked thru this forum and can not seem to find anything that says why I am unable to get this to work. I have included the ip route 0.0.0.0/0 192.168.1.1. below is the basic topology


ISP---- DSL modem--------- Watchguard firewall(192.168.1.1)-----------(192.168.1.250, 1/1/48)ICX 6450(1/1/4, VLAN5)--------(192.168.5.4)computer


Below is my config. and I also put the show IP route at the end. I have my firewall pluged into 1/1/48 and my computer into 1/1/4 with a IP of 192.168.5.4


from my computer I can ping 192.168.5.1, 192.168.1.250, but am not able to ping 192.168.1.1. but yet the switch can thru its CLI. I dont understand.


Can anyone help me with this?

Thanks








Current configuration:
!
ver 08.0.30uT313
!
stack unit 1
module 1 icx6450-48p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
priority 128
stack-port 1/2/1 1/2/3
stack unit 2
module 1 icx6450-48p-poe-port-management-module
module 2 icx6450-sfp-plus-4port-40g-module
priority 128
stack-port 2/2/1 2/2/3
stack enable
!
global-stp
!
!
!
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 2 name RMD by port
untagged ethe 1/1/1 ethe 1/1/7 to 1/1/47 ethe 2/1/1 to 2/1/48
spanning-tree 802-1w
spanning-tree 802-1w priority 1
!
vlan 5 name management by port
untagged ethe 1/1/4
router-interface ve 5
spanning-tree 802-1w
spanning-tree 802-1w priority 1
!
vlan 9 by port
untagged ethe 1/1/48
router-interface ve 9
!
vlan 10 name Voice by port
untagged ethe 1/1/2
spanning-tree 802-1w
spanning-tree 802-1w priority 1
!
!
!
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
enable aaa console
ip dhcp-client disable
ip dns server-address 192.168.1.110 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
no telnet server
username root password .....
!
!
!
!
ntp
!
!
hitless-failover enable
!
!
!
interface ve 5
ip address 192.168.5.1 255.255.255.0
!
interface ve 9
ip address 192.168.1.250 255.255.255.0
!
!
!
!
!
!
!
!
!
end


1 0.0.0.0/0 192.168.1.1 ve 9 1/1 S 12m3s
2 192.168.1.0/24 DIRECT ve 9 0/0 D 12m5s
3 192.168.5.0/24 DIRECT ve 5 0/0 D 11m55s
 

kapone

Well-Known Member
May 23, 2015
954
535
93
Your firewall doesn't have a static route back to 192.168.5.x.

When you ping the firewall from the switch CLI, you're on the same subnet as the firewall, so no specific route needed on the firewall. When you ping it from that .5.x computer....:)

computer (.5.x) --> switch-->firewall-->firewall goes...how the hell do I send a response back to .5.4?? I don't know where it is!
 
  • Like
Reactions: fohdeesha

brob

New Member
Feb 3, 2021
2
0
1
Thank you very much Kapone. That was the ticket!

Been doing a lot of learning on networking recently..
 

ArmedAviator

Member
May 16, 2020
89
51
18
Ohio
There might also be an issue with NAT on your firewall. I know this is an issue with pfSense.OPNSense where you have to manually adjust NAT settings.
 

Nikotine

New Member
Mar 17, 2021
8
0
1
I don't have these specific passive copper direct attached cables. DACs are different from transceiver + fiber cables. You won't be able to get temperature, power readings, etc. from DACs.
It would be to connect the switch (6450) to a NAS over a short distance, so not sure if fiber is needed.
My understanding is that copper is fine for short distances.
The options are very confusing.

Looking for 57-0000075-01, I find these for €15:
Or the same for €238:

This doesn't make any sense to me...
 

m4r1k

Member
Nov 4, 2016
64
6
8
32
Sorry I missed that you had the running-config linked already.

Th pasted ping shows the packet delays when pinging the switch.
Hey there!

Apologies for the late answer, in such shape my lab didn't really work, and setting something custom up takes time.

tl;dr is weird without ANY change, this morning re-applying my config (same as the one on GitHub) the issue is essentially gone (well, I of course upgraded to the latest firmware but two days ago the situation was unable)

I also created a much simpler setup: two 10Gbps interfaces of two different R630 in the same vlan, ping between the two and no drop nor delay whatsoever.

Should I take that the switch is about to die?

In your latest post, you say it happens to any traffic going through the switch (i.e. edge device to edge device). If the latter is true, than there may be an issue. If pings between edge devices are not experiencing this issue, than it is likely normal. Traffic sent to the switch itself is shunted to the management CPU and given low priority. The pings you see are higher than I've seen thus far on my setup, however I'm not sure how loaded up your switch is with traffic.
See below the system logs
Code:
#show logging
Syslog logging: enabled ( 0 messages dropped, 0 flushes, 0 overruns)
    Buffer logging: level ACDMEINW, 58 messages logged
    level code: A=alert C=critical D=debugging M=emergency E=error
                I=informational N=notification W=warning

Static Log Buffer:
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 1  is up
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 2  is up

Dynamic Log Buffer (1000 lines):
Apr  5 13:58:51:I:Security: SSH login by un-authenticated SSH user from src IP 192.168.178.68 from src MAC 3023.03e2.2a39 to PRIVILEGED EXEC mode using RSA as Server Host Key.
Apr  5 13:58:50:I:Security: SSH login by un-authenticated SSH user from src IP 192.168.178.68 from src MAC 3023.03e2.2a39 to USER EXEC mode using RSA as Server Host Key.
Apr  5 13:57:44:I:NTP: System clock is synchronized to 82.161.139.11.
Apr  5 13:56:35:I:Security: Time is updated by NTP server "82.161.139.11" from  "01:00:00.000 GMT+01 Mon Jan 01 1900 " to "13:56:35.499 GMT+01 Mon Apr 05 2021 "
Apr  5 13:55:30:I:System: Interface ethernet 1/1/5, state up
Apr  5 13:55:27:I:System: Interface ethernet 1/1/5, state down
Apr  5 13:55:27:I:System: Interface ethernet 1/1/14, state up
Apr  5 13:55:27:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is force-up.
Apr  5 13:55:27:I:System: Logical link on dynamic lag interface ethernet 1/1/16 is up.
Apr  5 13:55:27:I:System: Interface ethernet 1/1/16, state up
Apr  5 13:55:26:I:System: Interface ethernet 1/1/13, state up
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is force-up.
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/1/15 is up.
Apr  5 13:55:26:I:System: Interface ethernet 1/1/15, state up
Apr  5 13:55:26:I:System: Interface ethernet 1/1/5, state up
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/3/8 is up.
Apr  5 13:55:26:I:System: Interface ethernet 1/3/8, state up
Apr  5 13:55:26:I:System: Logical link on dynamic lag interface ethernet 1/3/7 is up.
Apr  5 13:55:26:I:System: Interface ve 140, state up
Apr  5 13:55:26:I:System: Interface ve 130, state up
Apr  5 13:55:26:I:System: Interface ve 120, state up
Apr  5 13:55:26:I:System: Interface ve 110, state up
Apr  5 13:55:26:I:System: Interface ve 100, state up
Apr  5 13:55:25:I:Trunk: Group (1/3/7, 1/3/8) created by 802.3ad link-aggregation module.
Apr  5 13:55:25:I:System: dynamic lag 30, has new peer info (priority=65535,id=e443.4b44.5b2e,key=15) (N/A)
Apr  5 13:55:25:I:System: Interface ethernet 1/1/7, state up
Apr  5 13:55:25:I:System: Interface ethernet 1/1/4, state up
Apr  5 13:55:25:I:System: Interface ethernet 1/1/6, state up
Apr  5 13:55:25:I:System: Interface ethernet 1/1/2, state up
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 2  is up
Apr  5 13:55:24:I:System: Stack unit 1   Power supply 1  is up
Apr  5 13:55:24:I:System: Interface ethernet 1/1/1, state up
Apr  5 13:55:24:I:System: Interface ve 178, state up
Apr  5 13:55:23:I:System: Logical link on force-up dynamic lag interface ethernet 1/3/7 is back to LACP control.
Apr  5 13:55:23:I:System: Interface ve 178, state down
Apr  5 13:55:23:I:System: Interface ve 140, state down
Apr  5 13:55:23:I:System: Interface ve 130, state down
Apr  5 13:55:23:I:System: Interface ve 120, state down
Apr  5 13:55:23:I:System: Interface ve 110, state down
Apr  5 13:55:23:I:System: Interface ve 100, state down
Apr  5 13:55:23:I:System: Logical link on dynamic lag interface ethernet 1/3/7 is force-up.
Apr  5 13:55:23:I:System: Logical link on dynamic lag interface ethernet 1/3/7 is up.
Apr  5 13:55:23:I:System: Interface ethernet 1/3/7, state up
Apr  5 13:55:23:I:System: Interface ve 178, state up
Apr  5 13:55:23:I:System: Interface ve 140, state up
Apr  5 13:55:23:I:System: Interface ve 130, state up
Apr  5 13:55:23:I:System: Interface ve 120, state up
Apr  5 13:55:23:I:System: Interface ve 110, state up
Apr  5 13:55:23:I:System: Interface ve 100, state up
Apr  5 13:55:23:I:System: Warm start
Apr  5 13:55:00:I:System: Port init success Stack unit 1 Port 1/2/1 Lane 0 T 0 R 0 Type 0:  00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x0000 00x00000x000
Apr  5 13:52:28:I:DHCPC: protocol disabled by user
Apr  5 13:52:28:I:NTP: client association is mobilized for 129.250.35.250.
Apr  5 13:52:28:I:NTP: client association is mobilized for 82.161.139.11.
Apr  5 13:52:28:I:NTP: client association is mobilized for 213.109.127.82.
Apr  5 13:52:28:I:NTP: The system clock is not synchronized to any time source.
Apr  5 13:52:28:I:NTP: client association is mobilized for 95.211.160.148.
Apr  5 13:52:28:I:NTP: The system clock is not synchronized and does not have a reference configured.
 

DASHIP

New Member
May 4, 2016
11
0
1
51
Be careful making sure you check the model number before buying. The model numbering scheme is confusing. I just purchased a 6610-48-PE thinking it was a 6610-48P-E. The latter has PoE ports, while the former does not. If you are unsure, be sure to check the datasheets listed on the first page of this thread. Here is an excerpt for the 6610-48 as an example. Notice how close some of the model numbers are. With the noted inaccuracy of eBay postings, it is easy to get the wrong switch. Also, the models with PoE have a "P" at the end of the model name on the front of the switch, in white lettering: "ICX 6610-48P". If the "P" is not present on the front, it is not a PoE model.
 

Attachments

ArmedAviator

Member
May 16, 2020
89
51
18
Ohio
tl;dr is weird without ANY change, this morning re-applying my config (same as the one on GitHub) the issue is essentially gone (well, I of course upgraded to the latest firmware but two days ago the situation was unable)
I suspect the issue is and never was with the switch but either a bad connection/cable somewhere or a routing table on an edge device doing funky things. Be sure to check for packet errors on your end device(s).

I also created a much simpler setup: two 10Gbps interfaces of two different R630 in the same vlan, ping between the two and no drop nor delay whatsoever.
This is what really matters.


Should I take that the switch is about to die?
Certainly not. Traffic sent to the switch management interface (i.e. ICMP ping) is removed from the normal path through the ASICs and sent to the management CPU. It is also fairly low priority, so if the management CPU is doing something else, the ICMP pings/SNMP returns will end up with varied latency. Meanwhile, traffic passing through the switch normally from device to device will experience none of the latency that the management CPU is returning in ping results.
 

m4r1k

Member
Nov 4, 2016
64
6
8
32
I suspect the issue is and never was with the switch but either a bad connection/cable somewhere or a routing table on an edge device doing funky things. Be sure to check for packet errors on your end device(s).


This is what really matters.


Certainly not. Traffic sent to the switch management interface (i.e. ICMP ping) is removed from the normal path through the ASICs and sent to the management CPU. It is also fairly low priority, so if the management CPU is doing something else, the ICMP pings/SNMP returns will end up with varied latency. Meanwhile, traffic passing through the switch normally from device to device will experience none of the latency that the management CPU is returning in ping results.
I think we're speaking too early without an important variable: time. The problem is back :-(

To make sure there is nothing wrong, I re-initialized two of my baremetal nodes (an R630 and an R730) with a super simple setup (no lag, no lacp, no lldp, no loop-detection etc), a single vlan, and not config whatsoever but the result is packet drop and delay.

Tomorrow I'll run a DPDK app to check the device's stats and put some real load on the system and see the true stability beyond ping.

Last week when I opened the ICX I immediately noticed that the main CPU was wayyyy hotter than the reported 50C. Would that be something to look into?
 

dennisp

New Member
Apr 1, 2021
6
4
3
I'm 94 pages in and already ordered a pair of 6610 to replace the Cisco 3750 I use at home. Also I get to learn some new tech and hopefully save some power in the process. 100+ pages to go but I skipped ahead to say thanks to @fohdeesha for sharing all of this info.
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,087
1,949
113
30
fohdeesha.com
I think we're speaking too early without an important variable: time. The problem is back :-(

To make sure there is nothing wrong, I re-initialized two of my baremetal nodes (an R630 and an R730) with a super simple setup (no lag, no lacp, no lldp, no loop-detection etc), a single vlan, and not config whatsoever but the result is packet drop and delay.

Tomorrow I'll run a DPDK app to check the device's stats and put some real load on the system and see the true stability beyond ping.

Last week when I opened the ICX I immediately noticed that the main CPU was wayyyy hotter than the reported 50C. Would that be something to look into?
I can almost promise your switch is fine, it sounds like another device is flooding the switch with what end up being CPU bound packets, something like a broadcast storm etc. the fact the issue/latency went away when you unplugged all your hosts also points to this. When it happens and you can reproduce it, unplug one host/device at a time until the issue goes away, then you know which one it was. also run "show cpu" a few times to see what usage is at (although it's not super reliable in my experience)
 
  • Like
Reactions: dswartz

fohdeesha

Kaini Industries
Nov 20, 2016
2,087
1,949
113
30
fohdeesha.com
Is it possible to configure interface management 1 with its own routing table on a ICX 6610, playing with VRF but looks like that int management 1 wont work, no VRF Forwarding options?

I'm looking at possibly of having a backdoor in case of emergency, workaround is using a jumpbox in that mgmt subnet... but not clean.

ICX 6610
Primary FCXR08030u.bin
SW: Version 08.0.30uT7f3
the ICX7xxx series supports putting the separate management port in non-default VRFs (like a management VRF), but sadly the 6 series does not. I get around this usually by creating a management VRF like usual, designating it the management vrf, making it like vlan 1500 or something, and putting 1 ethernet port in it (regular ethernet port). that eth port is now a dedicated management port in its own isolated management VRF
 

dreamkass

New Member
Aug 14, 2012
20
2
3
the ICX7xxx series supports putting the separate management port in non-default VRFs (like a management VRF), but sadly the 6 series does not. I get around this usually by creating a management VRF like usual, designating it the management vrf, making it like vlan 1500 or something, and putting 1 ethernet port in it (regular ethernet port). that eth port is now a dedicated management port in its own isolated management VRF
Thanks @fohdeesha also found it reading the documentation but was 8.0.9x, maybe next upgrade ICX 7xxx or Arista
 

eduncan911

Active Member
Jul 27, 2015
139
66
28
66
eduncan911.com
So i'm starting to understand the power of 10G at my fingertips with these switches... :)

Question: Would it acceptable to create a bonded LAG pair of 10G links tagged with two VLANs, over leaving each port untagged for a specific VLAN?

I have two new servers I'm setting up and they each have dual-10G, along with various 1G ports. Running Proxmox, I'll have one 10G dedicated to server CLRNET traffic (library software for various Windows RDCs), and the other 10G dedicated to Ceph data sync on the backend.

Considering redundancy, possible failures (and the fact that I don't visit the school often), I'm now thinking of setting up a LAG group for these two 10G ports and tagging the group with the same two VLANs I was going to assign to each port untagged anyways.

The idea is to operate at 20Gbps and if one 10G link drops out for one of various reasons, the other 10G link will keep chugging along as I get an alert.

These machines won't be able to saturate a single 10G link by any means (though one could get up to 6G peak though). So I don't have to worry about one VLAN taking over the entire 20G bandwidth. Though, that would be interesting to know how to limit VLAN bandwidth on a LAG group.

Is this a good idea?
 

LodeRunner

Active Member
Apr 27, 2019
103
44
28
Pretty sure you do not modify VLANs on a per port basis in a LAG, you add the LAG to the VLANs.

Edit: or rather, when you tag/untag a LAG to a VLAN, it does the same to the ports. I'm pretty sure you'll get an error if you try to fuss with the individual ports.
 
  • Like
Reactions: eduncan911

richtj99

New Member
Jul 8, 2017
23
0
1
48
Hi,

So I am having a strange issue. I have a Sonicwall router going to a unifi switch, going to three 6450 switches.

Its all working "sort of". The Sonicwall does most of the DHCP on the other vlans except for vlan 168 which has a windows server doing DHCP.

My camera server (on Cisco switch that I want to remove) can't hit anything plugged the brocade.

I stumbled on something with stp and thought that might be related. I enabled stp on vlan 168 and that seemed to help as offline cameras on other switches (same vlan) showed up.

I think my Cisco's had stp by default and I have a number of related issues. Can I turn stp on globally per switch vs vlan?

At one point I was messing around and put a Sonicwall port with my tagged vlans and a similar port from the unifi switch to the same brocade, different port but same tagged vlans and my kids started crying about the network. Unplugging the feeds fixed it so I think that's the issue.
 

richtj99

New Member
Jul 8, 2017
23
0
1
48
So for me I think it was a few things:

1. Moving all ports off vlan 1
2. adding a ve interface on vlan 168
3. Turning on spanning-tree 802-1w on each vlan
4. setting untagged ports vs leaving some ports without any untagged vlan
 

infoMatt

Active Member
Apr 16, 2019
201
87
28
Pretty sure you do not modify VLANs on a per port basis in a LAG, you add the LAG to the VLANs.

Edit: or rather, when you tag/untag a LAG to a VLAN, it does the same to the ports. I'm pretty sure you'll get an error if you try to fuss with the individual ports.
IIRC the switch itself doesn't allow you to edit a single interface of a LAG. To apply VLANs or other settings to a LAG you have to apply those on the "master" or primary interface of the bond.
 

pubsub

New Member
Apr 7, 2021
2
0
1
Anyone know what the screw size is for the 6610 lid? I had a couple shear off on my while loosening them. I was able to unscrew the stub with some pliers but would like to replace them. They look to be M3's of some size (maybe 3-5mm length?). Not sure about the thread pitch.